Skip to content

Pass structured trace evidence to candidate problem validation - #128

Closed
haggs wants to merge 2 commits into
mainfrom
dhaggerty/ase-1351-pass-structured-trace-evidence-to-nooa-for-candidate-problem
Closed

haggs wants to merge 2 commits into
mainfrom
dhaggerty/ase-1351-pass-structured-trace-evidence-to-nooa-for-candidate-problem

Conversation

@haggs

@haggs haggs commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor
  • Pass complete supporting Trace objects to the NOOA agent during candidate problem validation. Compass previously converted them into a truncated string with truncating_pformat, which NOOA truncated again for its initial preview. Supplying the objects keeps omitted evidence accessible through Python.
  • Update the agent instructions to inspect relevant spans and fields without modifying them, and print focused excerpts while checking the candidate's claim. This changes the previous instruction to read traces only as text.
  • Use the existing released NOOA dependency and its standard input preview. No dependency change is required.

I tested complete Compass runs on the same 532-trace dataset, using NOOA 0.0.10, the same model and five enabled signals, and the same ETHOS document. User sentiment was disabled and no codebase was supplied. The comparison uses baseline a78dc416 and runtime change 50fce1f6; the subsequent commit only updates documentation.

Measurement Baseline Structured objects
Complete run 14.6 min 34.5 min
Candidate problem validation 105 s 94 s
Compilation 7.2 min 22.8 min
Candidate problems 6 8
Final insights 1 3
Candidate problem validation input tokens, including cached tokens 273,992 1,296,100

These runs do not demonstrate an end-to-end latency improvement. Removing the eager formatting step eliminated about 30 seconds of baseline work, but NOOA still formats its previews, and candidate problem validation used substantially more input tokens. There was one complete run per revision, with the structured-object run first. Model responses and candidate selection differed, including in signal analysis before the changed code runs, so this pair cannot isolate the change's effect on total runtime. Both runs completed without suspension; an earlier suspended baseline was excluded.

I also compared four identical candidate problems produced by both runs. Three retained the same decision. For the fourth, the baseline accepted a claim of tool failure, while the structured-object agent inspected the actual responses and rejected it. I checked those responses and confirmed they reported success. This supports the evidence-access benefit, but does not establish an overall quality improvement or make the additional insights a quality metric.

I also tested a complete run with only CodeActConfig(prefill=None) changed in a temporary source snapshot. It finished in 62.4 minutes and produced two schema-valid insights without suspension. Candidate problem validation took 94.7 seconds for eight candidates, compared with 94.0 seconds for eight candidates with previews enabled. Its input tokens fell from 1,296,100 to 928,300, including cached tokens, while model calls increased from 38 to 59. Captured execution code shows the agents reading the candidate, ETHOS, and trace objects themselves. Signal analysis and compilation also took longer, so the total-runtime difference is not an isolated effect of the setting. This experiment did not show a latency benefit; prefill=None is not included in this PR.

Verified locally: both complete runs produced schema-valid outputs. The test suite passed with 941 tests and one skipped. A request-level regression verifies that evidence omitted from the initial preview reaches the next model request after the agent reads the original object; the same test fails against the previous implementation.

haggs added 2 commits November 4, 2026 10:22
Signed-off-by: Dan Haggerty <dhaggerty@nvidia.com>
Signed-off-by: Dan Haggerty <dhaggerty@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant