Normalized multiple-choice probability underflows for long sequence log-probabilities

[YusefSyed]

Bug

NormalizedMultiChoiceProbability.compute() exponentiates sequence
log-probabilities before normalizing them. When every choice has a sufficiently
negative but finite log-probability, each value underflows to zero and the
metric returns 0.0 instead of the gold choice's relative probability.

This is easy to reach for long multiple-choice continuations because the metric
receives sequence-level log-likelihoods, not token probabilities.

Reproduction

No model or API call is required:

from lighteval.metrics.dynamic_metrics import NormalizedMultiChoiceProbMetric
from lighteval.models.model_output import ModelResponse
from lighteval.tasks.requests import Doc

doc = Doc(query="q", choices=["A", "B"], gold_index=0, task_name="test")
result = NormalizedMultiChoiceProbMetric().compute_sample(
    doc=doc,
    model_response=ModelResponse(logprobs=[-1000.0, -1001.0]),
)
print(result["normalized_mc_prob"])

On current main at
932e1f2f4c5af3e926534f12b2a84a3ae18d6d3f, this prints:

0.0

and emits RuntimeWarning: invalid value encountered in divide.

The stable expected value is:

1 / (1 + exp(-1)) == 0.7310585786300049

Root cause

The current path is:

normalized_probs = np.exp(normalized_log_probs)
normalized_probs = safe_divide(
    normalized_probs[gold_ixs],
    np.sum(normalized_probs),
)

For [-1000, -1001], both exponentials are 0.0, so the denominator is also
zero. The fallback in safe_divide then silently converts the metric to zero.

Proposed fix

For finite values, subtract the largest normalized log-probability before
exponentiating. Subtracting one shared constant preserves every probability
ratio while preventing underflow:

log_probs = np.asarray(normalized_log_probs, dtype=np.float64)
if log_probs.size > 0:
    max_log_prob = np.max(log_probs)
    if np.isfinite(max_log_prob):
        log_probs = log_probs - max_log_prob

A focused regression can use [-1000.0, -1001.0] and compare with the
independent logistic expression above. Existing finite normal-range behavior,
multi-gold aggregation, and character/token/PMI normalization remain unchanged.

The all--inf, NaN, +inf, and empty-choice cases are intentionally outside
this report; the finite shift leaves their pre-existing behavior unchanged.