SamplingMetric string `normalize` argument always raises: getmembers() list treated as a dict

[Abelo9996]

Describe the bug

SamplingMetric.__init__ (src/lighteval/metrics/metrics_sample.py, around lines 1121-1128) resolves a string normalize argument like this:

allowed_normalizations = inspect.getmembers(
    lighteval.metrics.normalizations, inspect.isfunction
)  # -> {name: fn}
if normalize in allowed_normalizations:
    self.normalize = allowed_normalizations[normalize]
else:
    raise ValueError(f"Unknown normalization function: {normalize}")

inspect.getmembers(...) returns a list of (name, function) tuples, not a dict (the trailing comment is inaccurate). As a result:

• normalize in allowed_normalizations compares a string against tuples and is always False, even for a valid function name.
• The lookup allowed_normalizations[normalize] would raise TypeError: list indices must be integers or slices, not str, but it is never reached because the if is always False.

So the documented normalize: Callable | str | None argument is broken for the str form on every sampling metric (AvgAtN, MajAtN, PassAtK, GPassAtK): passing a valid normalizer name raises ValueError: Unknown normalization function: ....

To reproduce

Pure standard library, no model call:

import inspect
import lighteval.metrics.normalizations as norm

allowed = inspect.getmembers(norm, inspect.isfunction)
print(type(allowed).__name__)         # list
print("helm_normalizer" in allowed)   # False, although helm_normalizer is a valid normalizer
allowed["helm_normalizer"]            # TypeError: list indices must be integers or slices, not str

Or through the public API:

from lighteval.metrics.metrics_sample import PassAtK

PassAtK(normalize="helm_normalizer")  # ValueError: Unknown normalization function: helm_normalizer

Expected behavior

A valid normalizer name resolves to the corresponding function; only an unknown name raises ValueError.

Suggested fix

Build a dict from the members:

allowed_normalizations = dict(
    inspect.getmembers(lighteval.metrics.normalizations, inspect.isfunction)
)

normalize in allowed_normalizations then checks names, and allowed_normalizations[normalize] returns the function. Happy to open a PR with this plus a unit test.

Version

main (commit 932e1f2).