Skip to content

fix(rescore): require a matching cache fingerprint, not just match is not None - #368

Open
AmirF194 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
AmirF194:fix/363-rescore-cache-fingerprint
Open

AmirF194 wants to merge 1 commit into
TIGER-AI-Lab:mainfrom
AmirF194:fix/363-rescore-cache-fingerprint

Conversation

@AmirF194

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes #363.

rescore_one's cache-hit check only tested whether match was set, never whether the cached verdict came from the judge, config, rubric, instruction and intercepted evidence being requested now. Rescoring under a new --judge-model replayed the old judge's verdicts with zero live calls, and the batch rollup unconditionally stamped roll["judge_model"] with the newly requested judge even though nothing was actually reran. The pending-selection loop in main() had the same gap, so a stale run could be excluded from rescoring entirely.

Fix

  • _cache_fingerprint() builds a non-secret identity from the judge model, its config (api_key/api_keys excluded), the rubric, the instruction and the intercepted evidence, stored as cache_fingerprint alongside each verdict.
  • _cache_is_stale() is the single place that decides whether a cached verdict answers the current request. Both rescore_one and the pending-selection loop call it, so they can't disagree.
  • A verdict with no cache_fingerprint (everything written before this change) is treated as unknown provenance and recomputed, not trusted just because match is set.
  • aggregate_batch() now tracks which judge produced each verdict. The rollup sets judge_model_mixed: true and prints a warning if a batch still mixes judges (e.g. after --limit), instead of silently reporting one.
  • --force is unchanged.

Two notes on scope: the fingerprint doesn't version the judge prompt text itself, only the inputs rescore_one receives, so an in-place prompt edit wouldn't invalidate old caches. And "rubric revision" from the issue is already handled structurally, since strict/lenient write separate files and never share a cache entry; the fingerprint includes rubric anyway for defense in depth.

Corpus

  • v2
  • v1
  • both
  • not applicable

Host-side evaluator change; no task data involved.

Test plan

  • tests/test_rescore_cache_provenance.py, new: the issue's own repro (cache from an old judge is not reused), a legacy cache with no fingerprint, an instruction change, a judge-config change, api keys excluded from the fingerprint, a matching cache still reused, and aggregate_batch flagging mixed judge provenance.
  • tests/test_rescore_retarget.py, updated: its "cached scored verdict is not retried" case now carries a matching fingerprint, since an unqualified cache is no longer trusted.
  • Full suite in a clean python:3.11-slim container via uv run --frozen pytest: 293 passed, 2 skipped, and every new/changed test fails without this fix. ruff check, ruff format --check, pyright, uv build + twine check, the three console-script --help checks, and the Markdown link checker (triggered by the CHANGELOG edit): all clean.

I have not exercised a real judge-provider call here; the tests drive rescore_one through fake judge functions, not live API traffic.

Related issues

Fixes #363.

… not None

rescore_one's cache-hit check only tested whether a cached verdict had a
match value, never whether it came from the judge, config, rubric,
instruction and intercepted evidence being requested now. Rescoring under
a new --judge-model replayed the old judge's verdicts with zero live
calls, and the batch rollup stamped the newly requested judge onto a
summary it never produced. The pending-selection loop in main() had the
same gap, so a stale run could be skipped from rescoring entirely.

_cache_fingerprint() builds a non-secret identity of judge model, its
non-secret config, rubric, instruction and intercepted evidence, stored
as cache_fingerprint alongside each verdict. _cache_is_stale() is the one
place that decides whether a cached verdict answers the current request,
shared by rescore_one and the pending-selection loop. A verdict with no
recorded fingerprint is treated as unknown provenance and recomputed.
aggregate_batch() now tracks which judge produced each verdict; the
rollup flags judge_model_mixed instead of silently reporting one judge
when a batch still mixes them.

Fixes TIGER-AI-Lab#363
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[P1] rescore: stale judge caches are silently attributed to a newly requested judge model

1 participant