PUBLIC-READY
ETB-15 (name-pinned identity: the cache verifies a mutable name, not the artifact behind it)
What happens
The cache directory is <cache_dir>/<model_name>/<sha256(model_config dump)[:16]> (src/lighteval/utils/cache_management.py, SampleCache.__init__ and get_model_hash). For hub models the config carries revision: str = "main" by default, which is a mutable branch name, and for local models the config carries a path. In both cases the key pins the NAME, not the bytes.
The code already resolves the true commit for the results log: TransformersModel.__init__ sets self.model_sha = config.get_model_sha() (src/lighteval/models/transformers/transformers_model.py, around line 216), and the same resolution exists on the endpoint side. That resolved sha is simply never fed to the cache.
The documentation promises the opposite. docs/source/caching.mdx says: "A new cache is automatically created when: ... Model weights change (different revision, checkpoint, etc.)". With the default revision="main", weights changing under main leaves the config, the hash, and the cache directory identical, and run 2 is served run 1's responses while the run's own logging attributes them to the new weights.
Reproduced at pin 932e1f2. The PoC simulates the weight change deterministically (the stand-in model reads its behavior from a weights file; the two runs use byte-different weights files with an otherwise identical model config, which is exactly the state a moved branch or an overwritten checkpoint directory produces). The real hub is not contacted.
Repro
PoC: poc_cache_model_revision_name.py (two separate processes sharing one cache dir; real @cached decorator and real SampleCache; no network, no inference). Driver: REPRO-E1-F3.sh runs it twice and diffs stdout.
PYTHONPATH=<deps>:<lighteval src> HF_HUB_OFFLINE=1 /usr/bin/python3 poc_cache_model_revision_name.py
Key output:
== RUN2-weights-v2: weights_sha_prefix=WEIGHTS_b9599b06 model.greedy_until call count = 0
== RUN2-weights-v2: {"question": "Q0", "served_text": "WEIGHTS_b5c0e3eb::ANSWER_TO::Q0", "from_current_weights": false}
== verdict: VULNERABLE (3 stale of 3)
Impact
Continuous-evaluation setups that rerun a model alias on a schedule (nightly runs of main, a fine-tuning job overwriting a checkpoint directory, an endpoint redeployed behind the same name) get the previous weights' cached responses with zero calls, while the results log the new model sha next to them. Scores then describe a mixture of two model versions with nothing in the output saying so.
Suggested fix
Feed the resolved identity into the cache key. get_model_sha() already exists: include the resolved commit sha (and for local paths, a digest of the weights files or their mtimes) in the string that get_model_hash hashes, or fail loudly when the resolved sha differs from the one recorded in an existing cache directory. At minimum, update caching.mdx so it stops promising weight-change invalidation, and log cache hit counts per run.
PUBLIC-READY
ETB-15 (name-pinned identity: the cache verifies a mutable name, not the artifact behind it)
What happens
The cache directory is
<cache_dir>/<model_name>/<sha256(model_config dump)[:16]>(src/lighteval/utils/cache_management.py,SampleCache.__init__andget_model_hash). For hub models the config carriesrevision: str = "main"by default, which is a mutable branch name, and for local models the config carries a path. In both cases the key pins the NAME, not the bytes.The code already resolves the true commit for the results log:
TransformersModel.__init__setsself.model_sha = config.get_model_sha()(src/lighteval/models/transformers/transformers_model.py, around line 216), and the same resolution exists on the endpoint side. That resolved sha is simply never fed to the cache.The documentation promises the opposite. docs/source/caching.mdx says: "A new cache is automatically created when: ... Model weights change (different revision, checkpoint, etc.)". With the default
revision="main", weights changing undermainleaves the config, the hash, and the cache directory identical, and run 2 is served run 1's responses while the run's own logging attributes them to the new weights.Reproduced at pin 932e1f2. The PoC simulates the weight change deterministically (the stand-in model reads its behavior from a weights file; the two runs use byte-different weights files with an otherwise identical model config, which is exactly the state a moved branch or an overwritten checkpoint directory produces). The real hub is not contacted.
Repro
PoC:
poc_cache_model_revision_name.py(two separate processes sharing one cache dir; real@cacheddecorator and realSampleCache; no network, no inference). Driver:REPRO-E1-F3.shruns it twice and diffs stdout.Key output:
Impact
Continuous-evaluation setups that rerun a model alias on a schedule (nightly runs of
main, a fine-tuning job overwriting a checkpoint directory, an endpoint redeployed behind the same name) get the previous weights' cached responses with zero calls, while the results log the new model sha next to them. Scores then describe a mixture of two model versions with nothing in the output saying so.Suggested fix
Feed the resolved identity into the cache key.
get_model_sha()already exists: include the resolved commit sha (and for local paths, a digest of the weights files or their mtimes) in the string thatget_model_hashhashes, or fail loudly when the resolved sha differs from the one recorded in an existing cache directory. At minimum, update caching.mdx so it stops promising weight-change invalidation, and log cache hit counts per run.