Repository navigation
Conversation
Signed-off-by: Dan Haggerty <dhaggerty@nvidia.com>
Signed-off-by: Dan Haggerty <dhaggerty@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Traceobjects to the NOOA agent during candidate problem validation. Compass previously converted them into a truncated string withtruncating_pformat, which NOOA truncated again for its initial preview. Supplying the objects keeps omitted evidence accessible through Python.I tested complete Compass runs on the same 532-trace dataset, using NOOA 0.0.10, the same model and five enabled signals, and the same ETHOS document. User sentiment was disabled and no codebase was supplied. The comparison uses baseline
a78dc416and runtime change50fce1f6; the subsequent commit only updates documentation.These runs do not demonstrate an end-to-end latency improvement. Removing the eager formatting step eliminated about 30 seconds of baseline work, but NOOA still formats its previews, and candidate problem validation used substantially more input tokens. There was one complete run per revision, with the structured-object run first. Model responses and candidate selection differed, including in signal analysis before the changed code runs, so this pair cannot isolate the change's effect on total runtime. Both runs completed without suspension; an earlier suspended baseline was excluded.
I also compared four identical candidate problems produced by both runs. Three retained the same decision. For the fourth, the baseline accepted a claim of tool failure, while the structured-object agent inspected the actual responses and rejected it. I checked those responses and confirmed they reported success. This supports the evidence-access benefit, but does not establish an overall quality improvement or make the additional insights a quality metric.
I also tested a complete run with only
CodeActConfig(prefill=None)changed in a temporary source snapshot. It finished in 62.4 minutes and produced two schema-valid insights without suspension. Candidate problem validation took 94.7 seconds for eight candidates, compared with 94.0 seconds for eight candidates with previews enabled. Its input tokens fell from 1,296,100 to 928,300, including cached tokens, while model calls increased from 38 to 59. Captured execution code shows the agents reading the candidate, ETHOS, and trace objects themselves. Signal analysis and compilation also took longer, so the total-runtime difference is not an isolated effect of the setting. This experiment did not show a latency benefit;prefill=Noneis not included in this PR.Verified locally: both complete runs produced schema-valid outputs. The test suite passed with 941 tests and one skipped. A request-level regression verifies that evidence omitted from the initial preview reaches the next model request after the agent reads the original object; the same test fails against the previous implementation.