Conversation
… not None rescore_one's cache-hit check only tested whether a cached verdict had a match value, never whether it came from the judge, config, rubric, instruction and intercepted evidence being requested now. Rescoring under a new --judge-model replayed the old judge's verdicts with zero live calls, and the batch rollup stamped the newly requested judge onto a summary it never produced. The pending-selection loop in main() had the same gap, so a stale run could be skipped from rescoring entirely. _cache_fingerprint() builds a non-secret identity of judge model, its non-secret config, rubric, instruction and intercepted evidence, stored as cache_fingerprint alongside each verdict. _cache_is_stale() is the one place that decides whether a cached verdict answers the current request, shared by rescore_one and the pending-selection loop. A verdict with no recorded fingerprint is treated as unknown provenance and recomputed. aggregate_batch() now tracks which judge produced each verdict; the rollup flags judge_model_mixed instead of silently reporting one judge when a batch still mixes them. Fixes TIGER-AI-Lab#363
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Fixes #363.
rescore_one's cache-hit check only tested whethermatchwas set, never whether the cached verdict came from the judge, config, rubric, instruction and intercepted evidence being requested now. Rescoring under a new--judge-modelreplayed the old judge's verdicts with zero live calls, and the batch rollup unconditionally stampedroll["judge_model"]with the newly requested judge even though nothing was actually reran. The pending-selection loop inmain()had the same gap, so a stale run could be excluded from rescoring entirely.Fix
_cache_fingerprint()builds a non-secret identity from the judge model, its config (api_key/api_keysexcluded), the rubric, the instruction and the intercepted evidence, stored ascache_fingerprintalongside each verdict._cache_is_stale()is the single place that decides whether a cached verdict answers the current request. Bothrescore_oneand the pending-selection loop call it, so they can't disagree.cache_fingerprint(everything written before this change) is treated as unknown provenance and recomputed, not trusted just becausematchis set.aggregate_batch()now tracks which judge produced each verdict. The rollup setsjudge_model_mixed: trueand prints a warning if a batch still mixes judges (e.g. after--limit), instead of silently reporting one.--forceis unchanged.Two notes on scope: the fingerprint doesn't version the judge prompt text itself, only the inputs
rescore_onereceives, so an in-place prompt edit wouldn't invalidate old caches. And "rubric revision" from the issue is already handled structurally, sincestrict/lenientwrite separate files and never share a cache entry; the fingerprint includesrubricanyway for defense in depth.Corpus
Host-side evaluator change; no task data involved.
Test plan
tests/test_rescore_cache_provenance.py, new: the issue's own repro (cache from an old judge is not reused), a legacy cache with no fingerprint, an instruction change, a judge-config change, api keys excluded from the fingerprint, a matching cache still reused, andaggregate_batchflagging mixed judge provenance.tests/test_rescore_retarget.py, updated: its "cached scored verdict is not retried" case now carries a matching fingerprint, since an unqualified cache is no longer trusted.python:3.11-slimcontainer viauv run --frozen pytest: 293 passed, 2 skipped, and every new/changed test fails without this fix.ruff check,ruff format --check,pyright,uv build+twine check, the three console-script--helpchecks, and the Markdown link checker (triggered by the CHANGELOG edit): all clean.I have not exercised a real judge-provider call here; the tests drive
rescore_onethrough fake judge functions, not live API traffic.Related issues
Fixes #363.