Skip to content

Fix few-shot sampler isolation - #1334

Closed
atirna wants to merge 1 commit into
huggingface:mainfrom
atirna:fix/fewshot-sampler-isolation
Closed

Fix few-shot sampler isolation#1334
atirna wants to merge 1 commit into
huggingface:mainfrom
atirna:fix/fewshot-sampler-isolation

Conversation

@atirna

@atirna atirna commented Aug 16, 2026

Copy link
Copy Markdown

Description

I fixed the few-shot sampler so sequential variance seeds rotate independent copies of the task pool. I also fixed balanced sampling to retain falsy labels and use a local seeded RNG instead of mutating process-global random state.

The existing sequential assertion now compares the sampled offset with the unchanged pool. Previously, its expected value was read from the pool after sampling had mutated it.

Fixes #1307
Fixes #1309

Why

Sequential variance runs should choose offsets from the original few-shot pool for each seed, without changing later sampling or other consumers of the task pool. Balanced sampling should preserve valid empty labels and not affect unrelated random consumers.

Verification

  • before, on current main: the candidate sampler regression tests produced 4 failures and 2 passes
  • after: uv run --no-sync pytest tests/unit/prompt/test_prompt_manager.py -q (6 passed)
  • uv run --no-sync ruff format --check src/lighteval/tasks/prompt_manager.py tests/unit/prompt/test_prompt_manager.py
  • uv run --no-sync ruff check src/lighteval/tasks/prompt_manager.py tests/unit/prompt/test_prompt_manager.py

@atirna

atirna commented Sep 5, 2026

Copy link
Copy Markdown
Author

closing this one out for now, its been sitting a while and i dont want to leave a stale branch on your queue. happy to reopen or rebase it if its still useful to you.

@atirna atirna closed this Sep 5, 2026
@atirna
atirna deleted the fix/fewshot-sampler-isolation branch September 5, 2026 10:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant