fix: honor task-level num_samples and warn when it has no effect - #1361
Open
Abelo9996 wants to merge 1 commit into
Open
fix: honor task-level num_samples and warn when it has no effect#1361Abelo9996 wants to merge 1 commit into
Abelo9996 wants to merge 1 commit into
Conversation
LightevalTaskConfig exposes a num_samples field, but LightevalTask.__init__ reset self.num_samples to [1] and only extended it from sampling metrics, so a task-level num_samples was silently dropped and generation stayed at one sample. Seed self.num_samples from config.num_samples (accepting a list or a bare int) so the documented field is respected, and warn when num_samples > 1 is requested but no attached metric consumes multiple samples, instead of silently generating extra samples that never reach a score. Refs huggingface#618
Abelo9996
marked this pull request as ready for review
September 10, 2026 18:04
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Honor the task-level
num_samplesconfig field, and warn when it cannot take effect.Why
LightevalTaskConfigexposes anum_samplesfield (documented as "Number of samples to generate per input"), butLightevalTask.__init__resetself.num_samplesto[1]and only extended it from sampling metrics. A task-levelnum_sampleswas therefore silently dropped, and generation stayed at a single sample unless aGENERATIVE_SAMPLINGmetric such aspass@kormaj@kwas attached. That is the behaviour reported in #618.Change
self.num_samplesfromconfig.num_samples(accepting a list or a bare int, keeping1present for base generative scoring). The request builder already takes the max, so the largest requested value wins.num_samples > 1is requested but no attached metric consumes multiple samples, so a misconfiguration is loud rather than silently generating unused samples.Open question
Per my comment on #618, this defaults the no-consuming-metric case to a warning. If you would prefer a hard error, that is a one-line change and I am happy to switch.
Scope
This covers the
LightevalTaskConfig(num_samples=...)path. Themetric_options={...: {"num_samples": N}}path already sets the count inPipeline._update_num_samples; I left it untouched here and can extend the same warning there in a follow-up if useful.Refs #618