Skip to content

fix: honor task-level num_samples and warn when it has no effect - #1361

Open
Abelo9996 wants to merge 1 commit into
huggingface:mainfrom
Abelo9996:fix-num-samples-config
Open

fix: honor task-level num_samples and warn when it has no effect#1361
Abelo9996 wants to merge 1 commit into
huggingface:mainfrom
Abelo9996:fix-num-samples-config

Conversation

@Abelo9996

Copy link
Copy Markdown

What

Honor the task-level num_samples config field, and warn when it cannot take effect.

Why

LightevalTaskConfig exposes a num_samples field (documented as "Number of samples to generate per input"), but LightevalTask.__init__ reset self.num_samples to [1] and only extended it from sampling metrics. A task-level num_samples was therefore silently dropped, and generation stayed at a single sample unless a GENERATIVE_SAMPLING metric such as pass@k or maj@k was attached. That is the behaviour reported in #618.

Change

  • Seed self.num_samples from config.num_samples (accepting a list or a bare int, keeping 1 present for base generative scoring). The request builder already takes the max, so the largest requested value wins.
  • Warn when num_samples > 1 is requested but no attached metric consumes multiple samples, so a misconfiguration is loud rather than silently generating unused samples.
  • Add unit tests for the default, list, bare-int, and warning cases (no model or network needed).

Open question

Per my comment on #618, this defaults the no-consuming-metric case to a warning. If you would prefer a hard error, that is a one-line change and I am happy to switch.

Scope

This covers the LightevalTaskConfig(num_samples=...) path. The metric_options={...: {"num_samples": N}} path already sets the count in Pipeline._update_num_samples; I left it untouched here and can extend the same warning there in a follow-up if useful.

Refs #618

LightevalTaskConfig exposes a num_samples field, but LightevalTask.__init__
reset self.num_samples to [1] and only extended it from sampling metrics, so a
task-level num_samples was silently dropped and generation stayed at one sample.

Seed self.num_samples from config.num_samples (accepting a list or a bare int)
so the documented field is respected, and warn when num_samples > 1 is requested
but no attached metric consumes multiple samples, instead of silently generating
extra samples that never reach a score.

Refs huggingface#618
@Abelo9996
Abelo9996 marked this pull request as ready for review September 10, 2026 18:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant