For the ANTfrastructure agentic loop (shared/agentic-loop/), once this repo adopts it
(see Not adopted yet). The file was once zero bytes, which read as "the protocol exists
and the backlog is empty"; it was neither.
- [ ]actionable — the planner may pick it up- [b]blocked — skipped, and excluded from the pending count, so a backlog containing only blocked items still lets the planner run again- [x]completed — pruned on sight; the history lives in git
Entries are sorted by priority, and the executor takes the first - [ ] in file order:
P0 broken or blocking, P1 high value, P2 medium, P3 ideas, chores and blocked work.
The renderer's feature status per row is
crates/webgpu_renderer/docs/webgpu-renderer-roadmap.md;
its open work is here.
-
The NVIDIA packages have not run on a GPU they support (S, ★★★). The Linux and Windows packages are built and passed on the CPU, and hosted runners have no GPU. Done on an RTX 30xx or newer when
compare_detector_parity.pypasses against the CPU and the profile shows the provider; the hub'sdocs/gpu-artifact-receipts.mdhas the commands. DirectML on an NVIDIA card is unproven too. The owner runs this on a machine with an NVIDIA GPU. No release until this and the Linux AMD check are done. -
AMD's MIGraphX plugin has not run from this repo, on a card or in its image (S, ★★★). Hub CON231 step 2. Since 2026-11-12
migraphxregisterslibmigraphx-ep.soby path (crates/inference/src/ort_plugin.rs) and the AMD package carries the module and its texts. Proved on arm64 only: the search for the module, the option keys, the registration guard, and that a missing or unloadable module and a module without a device fail a named request and leaveautothe CPU. Seen in:latest-rocmon a runner without a card (candidate run 43345664993, 2026-11-13): the package builds, the census passes with the launcher's library path, the packaged module is found beside the packaged ONNX Runtime and registered, AMD's factory answersHIP failure 100: no ROCm-capable device is detected, the request by name is refused with that text andautoruns on the CPU; the tarball and the DEB pass the package check. The run stops at the AppImage: the hub'sverify-ort-appimage-payload.shasks an AMD AppImage for the wheel's exact bytes, and the image strips its CPU core (hub CON256 i; the fix needs an x86-64 host). Never run: a session on the module, and anything on a card. Next: when the hub's verifier is fixed, move the pin and dispatchlinux-x64-amd-gpu.ymlagain for the clean-host job; then, under native Ubuntu on the RX 9070 XT (under WSL even a plain HIP copy does not return, hub CON202), runscripts/linux/profile-gpu-packages.sh <dir> migraphx crates/inference/tests/fixtures/two_cats.jpg. The hub'sshared/python/verify_ort_profile.py, which that parity run uses, expectsMIGraphXsince hub 366b8259; the pin here is past it. Done whencompare_detector_parity.pypasses against the CPU for the tar, the deb and the AppImage and each run's profile has model operators on the providerMIGraphX; the hub'sdocs/gpu-artifact-receipts.mdhas the commands.
-
Diagnose the intermittent headless GPU-test SIGSEGV (M). A full default-parallel workspace run at
38b1c597crashed in theocclusiontest binary after 7/11 cases (SIGSEGV 11, exit 101). The same ELF then passed default and serial runs and five further default repeats; source CI on920ca1b7passed. The crash remains unexplained. Next: capture the GPU backend/driver and all-thread stack or core, distinguish test resource lifetime from a driver fault, then add a regression for the proven cause. Do not serialize all tests or call it fixed solely because CI is green. Attempts: 2026-11-03, 100 runs of theocclusionbinary without a crash (5 at a time, llvmpipe, 15 min bound). Since 2026-11-14 the Linux test step's JUnit record (scripts/linux/cargo-test-junit.py) names a binary that ended early, so the next crash in CI says which one. -
No browser has drawn a traced frame, the deferred path or the clouds (M, ★★★). Tint (headless Chrome on SwiftShader) compiles every traced kernel and the resolve (
crates/webgpu_renderer/docs/tracer.md§ Tests); the deferred and cloud WGSL have had only naga and the wasm32 check. Next: a headless Chrome run of the web demo that draws Forward, Deferred, clouds on, Ray traced and Path traced overcube_on_planeand reads each frame back: not the clear colour, no WebGPU error on the console. Guard: that run as a job on request, or its receipt indocs/testing.md. -
Detection quality has no regression set (M, ★★★). Audit 2026-11-12 (not double-checked):
crates/inference/tests/fixturesholds one photo, checked loosely (two cats, one at 0.8 or more). The rules incrates/observant/src/presence.rsandconfig.rs(2 hits, 60 s return, score 0.25) come from hand observations on the Pi, written indocs/observant.md. Next: record what the model returned run by run for the known hard cases (the drying rack, a cat behind the sofa and back, the feeding hour, a dark room) as JSON lines undercrates/observant/tests/scenarios, and replay them throughcounted,PresenceandVisits::record, asserting the visits and stays. Later a labelled set of 20 to 50 frames as a checksum-locked fetched file likemodels.lock, never tracked; the owner supplies own or free pictures. Guard: the replays in theteststep; the frame set underKATAGLYPHIS_REQUIRE_MODELwith a budget (cats found, false boxes at 0.25) for ORT and tract, so a new model or a retuned constant passes it before it reaches a camera.
-
A BC normal map on an adapter without BC becomes one white texel (S, ★★).
upload_material_texture(crates/webgpu_renderer/src/render/texture.rs) swaps any BCn payload it cannot upload for white, whatever the slot. White is neutral for base colour, metallic-roughness and occlusion, but a normal slot reads (1, 1, 1), a normal bent off the surface. Next: the flat (128, 128, 255) for a normal slot. Guard: a test of the fallback texel per slot. -
crates/guianswers a lost surface with a reconfigure only (S, ★★).on_failed_acquire(crates/gui/src/gui_wgpu/renderer.rs) mapsLostlikeOutdated; wgpu 30 says a lost surface must be created again beforeconfigure, so the camera GUI retries a configure that cannot help. The renderer already recreates it (recoveryincrates/webgpu_renderer/src/context.rs); reuse that rule. Guard: the mapping test inrenderer.rsexpects a recreate forLost. -
An OBJ opened in memory loses its alpha masks and every texture that is not PNG or JPEG (S, ★★).
obj_to_glb(crates/webgpu_renderer/src/app/obj_glb.rs) packs only PNG and JPEG and unhooks the slot of anything else with a warning, andparse_mtl(crates/webgpu_renderer/src/asset/obj_to_gltf.rs) reads nomap_d, so Sponza's vase plant and chain (map_d *_mask.jpg) draw as solid cards. The C++ engine cuts them (MaskCardDiscardsCutoutTexelsVisually). Next:map_dinto the base colour's alpha with MASK, and TGA/BMP decoded to PNG before packing. Guard: anobj_glbtest with a mask and a TGA texture. -
lavapipe refuses the full Colosseum's traced scene (M, ★★). lavapipe binds at most 128 MiB per storage buffer, and the full Colosseum's vertex buffer is larger, so
check(crates/webgpu_renderer/src/render/tracer/limits.rs) ends the build inTraceStatus::Failedwith the buffer named; desktop GPUs bind it and the browser takes Colosseum-web. Next, if CI or a software host should trace it: drop uv1 when no material samples it (24 bytes a vertex instead of 32) or split the vertices over two bindings. Guard: a limits test at the full Colosseum's counts. -
Skins and morph targets are traced at their first pose (M, ★★).
sync_animated_instances(crates/webgpu_renderer/src/render/forward/traced.rs) moves rigid instances only; a skinned or morphed mesh keeps the pose the build saw, so its traced frame and shadow disagree with raster once it moves. Next: refit that mesh's BVH from the skinned positions on a pose change. Guard:skinned_bar.gltfposed away from its rest, traced against raster asthe_traced_scene_follows_an_animated_nodedoes for a rigid node. -
The path tracer ignores punctual lights and a loaded HDR environment (M, ★★). NEE samples the sun alone and a bounce that misses sees the procedural gradient (
shaders/tracer/traced.slang,miss_radiance), soKHR_lights_punctualscenes and a bound.hdrlight the raster frame only. Next: punctual lights in NEE, then the environment as the miss term. Guard: a lone card lit by a point light, traced against raster within a tolerance. -
The deferred path has no anti-aliasing (M, ★★). The G-buffer is one sample, as in the C++ engine, so the Colosseum's sub-pixel crowd aliases where forward's 4x MSAA does not (
crates/webgpu_renderer/tests/deferred.rs; 6.4 levels apart on average,CHANGELOG.md). Next: FXAA after the lighting pass, off with forward. Guard: the deferred-against-forward edge measurements incrates/webgpu_renderer/tests/deferred.rsshrink. -
Clouds are dark and grainy in both renderers (M, ★★). The shared estimator (
shaders/common/volumetric_clouds.slang) has no ambient term, so a cloud is darker than the sky around it, and 8 steps through a slab seen at a grazing angle undersample the fine noise. Fixing it moves the C++ cloud goldens too, so change both renderers in one pin bump. Guard: a cloud's mean luminance against the sky behind it incrates/webgpu_renderer/tests/clouds.rsand the C++ golden. -
No performance number is tracked for the Pi (M, ★★). Audit 2026-11-12, which measured nothing: the only perf-host baseline in
perf/is for a Ryzen 9 5950X, and the bench gate in.github/workflows/reusable-linux.ymlfails only on a doubling. On a Pi 5 the model is about 99 % of a detection (261 ms against 1.6 ms and 1.8 ms for the two conversions,CHANGELOG.md), so frame copies are not worth chasing. The number that exists is the one/healthzserves; keeping a history of it is hub work (Home Assistant). Next: run the service one evening withKATAGLYPHIS_ORT_DEVICE=xnnpack, readstats.model_msand write the result intoCHANGELOG.md; on x86 XNNPACK is 3.7 times slower, on ARM nobody knows. A Cortex-A76 perf-host baseline only if it is still wanted: its setup builds the release CLI with fat LTO in the container and refuses a load above 2, a heavy job beside a live camera. On himbeere2 that link does not fitpi-ci's 5 GB at all: rustc was killed linkingkataglyphis_clion 2026-11-15, andCARGO_PROFILE_RELEASE_LTO=false CARGO_PROFILE_RELEASE_CODEGEN_UNITS=16built it; the setup needs that override, or a largerPI_CI_MEMORY, before a Pi baseline. Then first give the detector commands inquality-report.tomlObservANT's two threads without spinning, and record it on a quiet night as onesystemd-rununit. If one new gated bench is wanted: preprocessing at 1280x720, the app's default size. No allocation-count tests. -
Rust mutation testing is by hand (M, ★★). Audit 2026-11-12. The coverage half is done (2026-11-15): the
coveragestep leavessrc/frb_generated.rsout (1157 ofsrc's 1795 lines, never run by a Rust test), which tooksrcfrom 24 % to 67.6 %, and its floor from 19 % to 65 %. Thecargo mutantsscores indocs/testing.mdare from 2026-10-15; no workflow runs the tool, and the image does not carry it (hub CON238). If the owner accepts one more scheduled job, a weekly one runs the per-file commands ofdocs/testing.mdwith--jobs 1 -- --libon the pure modules and uploadsmutants.out. The fivecrates/inferencerecipes are inscripts/linux/tests/ort-license-mutations.jsonsince 2026-11-15 (consumer.oxidant-inference-*, each proved to bite). Guard: the raised floor, and akind = "count"artifact overmutants.out/missed.txtinquality-report.toml. The capture loop got six tests on 2026-11-15 (the real model, a late file, the stall, the re-probe, the engine's two end reasons): the first run after CON238 writescapture.rs's new count intodocs/testing.mdand addssrc/session_slot.rsto its file list. -
No packaged ObservANT has answered 502 to a panicking question yet (S, ★★). Since 2026-11-14 every ObservANT build uses its own profile (
[profile.observant], release withpanic = "unwind", owner decision of 2026-11-13): OxidANT's producer helper (790f8bb) and OmniAccelerANT's three package paths (8108bd6). Left: the proof in the field. Done when a packaged service answers 502 to a question whose answer panics and keeps streaming (crates/observant/src/ask.rs, the poisoned-lock recovery and the failed-task reply).
The top 5 of logs/code-quality/queue.md (bash scripts/linux/quality-report.sh), leaving out doc-dupes-hub.allow, which the owner keeps in every AGENTS.md; re-sync when it moves.
-
large-files.allow:3budget 2031 (allow_budget_total) -
rust-function-size.allow:15budget 281 (allow_budget_total) -
function-size.allow:6budget 171 (allow_budget_total) -
rust-function-size.allow:14budget 171 (allow_budget_total) -
function-size.allow:5budget 156 (allow_budget_total)
-
Decide whether the path tracer's bounce misses match the C++ sky (S). On purpose, a bounce that misses gets the renderer's sky gradient times the ambient strength and the environment intensity, where the C++ path tracer uses a fixed gradient at half intensity (roadmap, ray- and path-traced row). Owner decision; if it should match, the C++ side changes, since its raster has no ambient term to follow.
-
A new
set_instancescount waits for the next upload (S). A count change flagsTracer::instances_currentfalse and the traced scene keeps the old instances until a scene upload; the same count moves in place. Next: rebuild the top level on a count change. -
Bloom glows through clouds (S). The bright pass reads the HDR before the clouds, and the tonemap only scales it by the cloud's transmittance (
shaders/tonemap/tonemap.slang), so a hidden sun's halo shows at a cloud's edge. Next: bloom after the cloud composite, if the cost is acceptable. -
A one-pixel rim without clouds at silhouettes (S). The march stops at the resolved depth, the nearest of the four MSAA samples (
shaders/depth_resolve/depth_resolve.slang), so an edge pixel half sky gets no cloud. -
egui text renders heavier natively than in the browser (S). The native surface is sRGB (
surface_configincrates/webgpu_renderer/src/context.rs), so egui's coverage blends in linear light; the browser canvas is not, so it blends in sRGB. Next: paint egui into a non-sRGB view of the frame. -
egui's Style editor and inspection windows show tofu (S). The studio fonts (
crates/webgpu_renderer/src/ui/fonts.rs,FontDefinitions::empty()) carry no emoji, which those two debug windows (crates/webgpu_renderer/src/ui/editor/windows.rs) draw. Next: add egui's emoji face as the last fallback, inside the wasm budget. -
Debug builds trace slowly (S). The dev profile builds the renderer at opt-level 0 (
Cargo.tomlraises only dependencies), so building the full Colosseum's BVH in a debug viewer takes about half a minute. Owner decision:[profile.dev.package.kataglyphis_webgpu_renderer] opt-level = 1. -
The tracer uploads in one frame and keeps its CPU arrays (M). About 0.5 GB for the full Colosseum goes to the GPU in one frame (a hitch), and about 480 MB of CPU arrays stay for queries and top-level rebuilds. Next: spread the upload over frames; free the arrays a build no longer needs.
-
F5 leaves the post passes alone (S).
reload_shadersrebuilds forward, sky, deferred, clouds and both traced kernels; the tonemap (with the cloud composite), bloom, SSAO, histogram and depth-resolve WGSL keep the build's. Next: aTonemapPassreload in the same error scope first. -
WARP crashes when the renderer tests run in parallel on windows-11-arm (S).
docs/ci.md, the arm64 lane's run job:d3d10warp.dllfaults withSTATUS_ACCESS_VIOLATION; serially all pass. Re-dispatchwindows-arm64-cross.ymlwithparallel-renderer-testsafter a runner image update. -
A model run that hangs cannot be ended (S, ★). A run in flight for over 20 s shows in
/healthz, its boxes are dropped and a start in the app gives up after 10 s, but the run itself goes on: nothing callsRunOptions::terminate. Leave it until a GPU provider ships; on the CPU no hung run is on record. The four guards of 2026-11-13 are unit tests; addcapture.rsandsrc/session_slot.rsto thecargo mutantsfile list indocs/testing.mdat the next mutation run (75 minutes on a Pi 5, not run for this change). -
capture.rsmixes the camera with pure logic (M, ★). Audit 2026-11-12; its first half,to_jsonholdingdetectedwhile it took other locks, is done (2026-11-15, a test that deadlocked on the old order). Left: splitcrates/observant/src/capture.rsinto the part that needs a camera and the pure part (Visits,counted,Backoff, geometry), which helps the weakest mutation score; splitstatus.rs(31 fields) when the next feature adds one. Guard: the tests move unchanged, and the file names in thecargo mutantscommands ofdocs/testing.mdchange in the same commit. -
[b] CUDA still loads the old way; move it to the plugin row when the hub publishes its new NVIDIA image (S, ★★). Owner, 2026-11-12: only AMD now. The hub's contract (
docs/consumer-image-contract.md, The CUDA plugin EP) says a Rust consumer of the new:latest-nvidiaregisterslibonnxruntime_providers_cuda.soby path under the nameCUDAExecutionProvider: the module there has noGetProviderbridge, andort'sCUDA::default()appends in the in-tree form (SessionOptionsAppendExecutionProvider_CUDA_V2). Not run against that image. Blocked: the hub records its runtime wrappers and receipts as open. Then: one more row incrates/inference/src/ort_plugin.rs(the name, the variable that names its root, the module's file name), that row inproviders_for_devicein place ofCUDA::default(),cuda_probetaking the row's provider, andort'scudafeature dropped if nothing else needs it. The module lies beside the GPU ONNX Runtime library, in the image and in the NVIDIA package, which the search's second step finds; the hub names no root variable for it. -
[b] RVV
MatMulNBitssegfaults on a 4-bit model in the chain's ORT 1.31.0 (M, ★★). Crash:kataglyphis_text --test contract,signal: 11, SIGSEGV, run 41463196239 and locally on image7d4d4129. Under QEMU, intra-op workers fault inlibonnxruntime.so.1.31.0at vaddr0x150150c,flw fa5, 0x0(s10), in stripped code matchingSQ4BitGemmKernel_CompInt8_Impl, withsi_addr = RangeStartN * 18 * 4(for example0x1f80), soQuantBScaleis null. The reachable cause, from the v1.31.0 source:UseSharedPrePackedBuffersnarrowsshared_packed_b_finalizedunderMLAS_TARGET_ARM64whilePrePackuses!MLAS_TARGET_AMD64_IX86, so riscv64 setsscales_are_packed_. ORTmaincarries the same guard (checked 2026-11-02).RUST_MIN_STACK,QEMU_STACK_SIZEandvlen=256/512still crash;ORT_MLAS_RISCV_FORCE_SCALAR=1passes with the exact ids. Fix: a hub ORT patch or an upstream fix, which needs an image chain; then drop the scalar run fromriscv64-test. Until then a riscv64 process that loads a 4-bitaccuracy_level=4model on RVV crashes: the override covers this test, not a shipped riscv64 binary. Real silicon is untested, and the guard theory has no patched-build proof yet. -
[b] Accept DNNL on amd64 once
:latestships oneDNN (S). The chain ORT'slibonnxruntime_providers_dnnl.soneedslibdnnl.so.3, which the image lacks, soKATAGLYPHIS_ORT_DEVICE=dnnlisProviderUnavailableon amd64 too. Blocked on hub BACKLOG CON191 (an image chain). Thencpu_accelerators.rsandcheck-backend-parity.shrequire DNNL to run and match the CPU provider on amd64; nothing here changes. -
[b]
/askshows no text until the whole answer is there (L, ★★). Audit 2026-11-12:Assistant::runincrates/observant/src/ask.rsthrows the pieces away andweb.rshasBody = Full<Bytes>, so the Stream page waits up to 310 s behind one empty bubble. Blocked on the owner's decision: streaming the reply (the facts first, then the pieces, then a done line, with the JSON reply kept for clients that do not ask) changes the body type of the whole web module, the Dart client and their tests. It is not part of the timeout work (2026-11-15). -
[b] A push type-checks Rust for the host target only (L, ★★). Audit 2026-11-12: the
fmt-clippystep lints the host target, and no workflow or step callsscripts/linux/clippy-windows.sh. Code that fails on Windows alone is first compiled when a commit asks for the Windows lane; the recorded case isI420Layout, a struct nothing used on Windows (run 38496162066). Of 43 commits to the Windows lane's paths since 2026-10-16, 24 carried no Windows token; each was built by a later commit, after about a day in the median, so the cost is a late find. Blocked on the hub image: the script installs mingw and adds thex86_64-pc-windows-gnutarget as root, and the lane container runs as uid 1001. Once the image carries both: oneclippy-windowsstep forkataglyphis_observant,kataglyphis_textand the root package, with the added minutes measured (about 5 a package,docs/observant.md) before it runs on every push.
- llguidance accepts the hub tiny model's tokenizer (GenAI 0.17.0 in
:latest, index7d4d4129, amd64).GeneratorParams.set_guidance("json_schema", ...)on the tiny model decodesabcunder{"type":"object","properties":{"a":{"type":"string","maxLength":3}},"required":["a"],"additionalProperties":false,"x-guidance":{"whitespace_flexible":false}}to the ids[123, 34, 97, 34, 58, 34, 35, 36, 37, 34, 125, 256],{"a":"#$%"}then the end token 256. A schema that allows whitespace makes the successor rule loop on\t\nuntil the budget ends, so the schema test keeps the tight form. No SmolLM2-135M fallback is needed for schema tests; SmolLM2-135M runs the ModelANT contract (crates/text/tests/contract.rs). Sampling at temperature 1 over the full vocabulary gives seed 1 a different 24-token answer from seeds 2 to 4. The Windows chain builds GenAI with llguidance too since254b7ea(CHANGELOG.md).
The loop itself (config, runner wrappers, scripts/agentic-loop/) is not set up.
Copy-and-edit templates live in ANTfrastructure's shared/agentic-loop/templates/; a
consumer supplies this file, a config JSON, thin runner wrappers, and optionally
per-engine system prompts.