Describe the bug
src/lighteval/tasks/tasks/dyck_language.py scores with inspect_ai's exact():
from inspect_ai.scorer import exact
...
scorer=exact(),
In every inspect-ai release from 0.3.27 (Sep 2024) through 0.3.252, exact() compared sets of words: order and token multiplicity were discarded before comparison (upstream UKGovernmentBEIS/inspect_ai#4618, fixed by #4619, first released in 0.3.253). Dyck targets are space-separated sequences of closing brackets, and order is the property being tested, so under inspect-ai <= 0.3.252 a wrongly-ordered closing sequence with the right bracket types scores 1.0.
To Reproduce
On inspect-ai 0.3.252:
from inspect_ai.scorer._classification import max_exact_score
print(max_exact_score("} ] )", [") ] }"])) # wrong bracket order
print(max_exact_score("hello world", ["world hello"])) # word order
print(max_exact_score("hello world", ["hello world"])) # control: identical
print(max_exact_score("bananas", ["hello world"])) # control: unrelated
Same script on the 0.3.253 wheel: the first two print 0.0, controls unchanged.
Expected behavior
A wrongly-ordered closing sequence scores 0. Requiring inspect-ai >= 0.3.253 for these tasks (or otherwise ensuring the fixed scorer is used) would do it. Happy to open a PR for the pin bump. Results produced with older pins can only be inflated, never deflated, because the old comparison marks more answers correct.
Version info
lighteval main; inspect-ai 0.3.252 vs 0.3.253; Python 3.13.5, Windows 11.
(Claude helped write this up; I ran the repro myself, on both wheels.)
Describe the bug
src/lighteval/tasks/tasks/dyck_language.pyscores with inspect_ai'sexact():In every inspect-ai release from 0.3.27 (Sep 2024) through 0.3.252,
exact()compared sets of words: order and token multiplicity were discarded before comparison (upstream UKGovernmentBEIS/inspect_ai#4618, fixed by #4619, first released in 0.3.253). Dyck targets are space-separated sequences of closing brackets, and order is the property being tested, so under inspect-ai <= 0.3.252 a wrongly-ordered closing sequence with the right bracket types scores 1.0.To Reproduce
On inspect-ai 0.3.252:
Same script on the 0.3.253 wheel: the first two print 0.0, controls unchanged.
Expected behavior
A wrongly-ordered closing sequence scores 0. Requiring
inspect-ai >= 0.3.253for these tasks (or otherwise ensuring the fixed scorer is used) would do it. Happy to open a PR for the pin bump. Results produced with older pins can only be inflated, never deflated, because the old comparison marks more answers correct.Version info
lighteval
main; inspect-ai 0.3.252 vs 0.3.253; Python 3.13.5, Windows 11.(Claude helped write this up; I ran the repro myself, on both wheels.)