Skip to content

Use merged NOOA type-hint caching for trace formatting - #124

Draft
haggs wants to merge 3 commits into
mainfrom
dhaggerty/ase-1340-cache-type-hint-resolution-during-validation-context
Draft

haggs wants to merge 3 commits into
mainfrom
dhaggerty/ase-1340-cache-type-hint-resolution-during-validation-context

Conversation

@haggs

@haggs haggs commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Problem

I found that candidate problem validation was spending most of its time preparing trace evidence before making an LLM call. _trace_context calls nooa.agentdoc.truncating_pformat once per candidate problem to turn its supporting traces into bounded text. Many spans share the same class, but visibility checks and structured inspection repeatedly resolved that class's annotations.

NOOA has now merged the annotation-cache fix. Successful resolutions are shared during a truncating_pformat or direct pformat call, including nested calls on the same thread. Later independent calls resolve annotations afresh, and instance visibility overrides remain live.

Integration

This PR updates the temporary uv source override to NOOA's merge commit, replacing the initial implementation with the completed fix, including cache-lifetime safeguards.

  • Adapt test fixtures to NOOA's response parts and LiteLLM's public completion entry point.
  • Include the pinned source's license and notices in the distribution. Since the license scanner omits Git dependencies, the disclosure tooling verifies the reviewed source revision and archive checksum.

Measured impact

I compared complete Compass CLI runs over the same synthetic dataset of 10,000 traces, with all six analysis signals enabled. The Compass source, exporter, and configuration were the same; the comparison changed only NOOA from the unmodified base to the initial cache implementation.

Measurement Before After Reduction
Supporting-trace formatting 45.8 min 2.9 min 94%
Candidate problem validation, including formatting and agent calls 50.5 min 7.6 min 85%
Total CLI runtime 130.8 min 81.7 min 38%

Both runs completed with valid insights. These measurements cover the initial cache implementation, before the upstream merge. There was one run per revision, with different candidates and LLM responses on a shared host, so the total-runtime reduction is an observed result. Formatting was timed directly using local instrumentation because hosted telemetry dropped spans.

Release gate

NOOA 0.0.10 on PyPI predates the fix. This remains a draft integration until a packaged release includes it; before merging, replace the Git override with that release and require its version. The current override applies only to uv-managed environments; the published dependency range is unchanged.

Verified locally: All 941 tests passed, with one skipped, using the merged NOOA dependency, including CLI integration tests. License contents matched in the wheel, source distribution, and a wheel rebuilt from the source distribution.

nicot added a commit that referenced this pull request Nov 4, 2026
## Why
[CVE-2026-114612](https://osv.dev/vulnerability/CVE-2026-114612) (CVSS
9.9): an internal user of a LiteLLM proxy can forge a proxy-admin token
and run commands through the MCP stdio endpoint. It affects 1.87.0 until
1.100.4 / 1.101.3 / 1.102.2 / 1.103.1 / 1.104.0-rc.2. Compass only uses
the LiteLLM SDK, not the proxy server, so this isn't exploitable through
Compass, but the locked 1.98.0 is in the affected range and scanners
flag it.

## Changes
- `pyproject.toml`: `litellm>=1.55,<2` → `litellm>=1.104.0,<2`. Every
final release from 1.104.0 on includes the fix; a floor of 1.100.4 would
still allow vulnerable 1.101.0–1.101.2.
- `uv.lock`: `uv lock --upgrade-package litellm` → litellm 1.108.0. It
requires `boto3>=1.43.117`, so boto3/botocore go to 1.43.124 and certifi
to 2026.10.23. `httpx[http2]` adds h2, hpack and hyperframe.
- `third_party/license_overrides.yaml`: OSV-Scanner reports no license
for the new boto3, botocore and hyperframe releases. Added overrides
matching their PyPI metadata: boto3 and botocore are `Apache-2.0`,
hyperframe is MIT (its classifier is `License :: OSI Approved :: MIT
License`).
- Regenerated `THIRD_PARTY_LICENSES.md`, `third_party/licenses.jsonl`
and `third_party/NOTICES.txt` with `make update-licenses` (OSV-Scanner
v2.3.3, the CI version).

## Testing
- `make check-licenses`, `tools/check_copyright_headers.py`, `ruff
check`, `ruff format --check`
- `pytest`: 941 passed

Note: #124 also changes `uv.lock` and the license disclosures, so
whichever PR merges second will need `uv lock` and `make
update-licenses` rerun.
@haggs haggs changed the title Use render-scoped type-hint caching from NOOA Use merged NOOA type-hint caching for trace formatting Nov 5, 2026
@haggs
haggs force-pushed the dhaggerty/ase-1340-cache-type-hint-resolution-during-validation-context branch from e0c2e7e to cc52640 Compare November 5, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant