Skip to content

fix: isolate few-shot sampler RNG and pool from shared and global state (#1307, #1309) - #1379

Open
Abelo9996 wants to merge 1 commit into
huggingface:mainfrom
Abelo9996:fix-fewshot-sampler-seed-reproducibility
Open

fix: isolate few-shot sampler RNG and pool from shared and global state (#1307, #1309)#1379
Abelo9996 wants to merge 1 commit into
huggingface:mainfrom
Abelo9996:fix-fewshot-sampler-seed-reproducibility

Conversation

@Abelo9996

Copy link
Copy Markdown

What

Two fixes in FewShotSampler (src/lighteval/tasks/prompt_manager.py) that leak state and break variance-seed reproducibility.

Sequential (#1307)

_init_fewshot_sampling_sequential rotated the list returned by task.fewshot_docs() in place. That list is the task's memoized _fewshot_docs, handed back by reference, so the rotation mutated shared state and each cached seed aliased the same over-rotated list. With few_shot_iterations > 1, offsets accumulate and every seed after the first selects the wrong examples. Fix: copy the pool with list(...) before rotating, the same way _init_fewshot_sampling_random already does.

Balanced (#1309)

_init_fewshot_sampling_balanced had two issues:

  1. It seeded and used the global random module, so selection depended on and perturbed global RNG state. Fixed with a local random.Random(variance_seed).
  2. The label loop guarded with if not next_label, which also fires on a present but falsy label (int 0, empty string), cutting the balanced selection short. Fixed to stop only on next_label is None, which is the real "cycle exhausted" signal.

Behaviour

random.Random(seed) produces the same sequence as random.seed(seed) on clean state, so selections are unchanged in the common case. The change fixes the cases where shared or global state leaked between seeds, and where a falsy label truncated the balanced sample.

Tests

  • New tests/unit/prompt/test_fewshot_sampler.py: sequential does not mutate the shared pool, sequential seeds are independent, a falsy label does not truncate the balanced sample, and balanced leaves global RNG state untouched.
  • Updated test_fewshot_sampler[sequential] in tests/unit/prompt/test_prompt_manager.py: its previous assertion passed only because the in-place rotation mutated the shared pool the assertion also read from; it now checks the intended rotation ([20:40]).
  • tests/unit/prompt/ is green; ruff check and ruff format --check pass on the changed files.

Closes #1307
Closes #1309

The sequential and balanced few-shot samplers leaked state that corrupts variance-seed reproducibility.

Sequential (huggingface#1307): _init_fewshot_sampling_sequential rotated the list returned by task.fewshot_docs() in
place. That list is the task's memoized pool, returned by reference, so the rotation mutated shared state
and every cached seed aliased the same over-rotated list, so each seed after the first selected the wrong
examples. Copy the pool with list(...) before rotating, mirroring the random path.

Balanced (huggingface#1309): _init_fewshot_sampling_balanced seeded and used the global random module, and its label
loop guarded with "if not next_label", which also fires on a present but falsy label (int 0, empty string),
truncating the selection. Use a local random.Random(variance_seed) and stop only when next_label is None.

random.Random(seed) yields the same sequence as random.seed(seed) on clean state, so selections are
unchanged in the common case; the change fixes the leaky and falsy-label cases.

Adds unit tests for both. Updates test_fewshot_sampler[sequential], whose previous assertion passed only
because the in-place rotation mutated the shared pool that the assertion also read from.

Closes huggingface#1307
Closes huggingface#1309
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant