Skip to content

[BUG] dyck_language scores with inspect_ai's exact(), which ignored word order in every release up to 0.3.252 #1329

Description

@LippInc

Describe the bug

src/lighteval/tasks/tasks/dyck_language.py scores with inspect_ai's exact():

from inspect_ai.scorer import exact
...
scorer=exact(),

In every inspect-ai release from 0.3.27 (Sep 2024) through 0.3.252, exact() compared sets of words: order and token multiplicity were discarded before comparison (upstream UKGovernmentBEIS/inspect_ai#4618, fixed by #4619, first released in 0.3.253). Dyck targets are space-separated sequences of closing brackets, and order is the property being tested, so under inspect-ai <= 0.3.252 a wrongly-ordered closing sequence with the right bracket types scores 1.0.

To Reproduce

On inspect-ai 0.3.252:

from inspect_ai.scorer._classification import max_exact_score

print(max_exact_score("} ] )", [") ] }"]))              # wrong bracket order
print(max_exact_score("hello world", ["world hello"]))  # word order
print(max_exact_score("hello world", ["hello world"]))  # control: identical
print(max_exact_score("bananas", ["hello world"]))      # control: unrelated
1.0
1.0
1.0
0.0

Same script on the 0.3.253 wheel: the first two print 0.0, controls unchanged.

Expected behavior

A wrongly-ordered closing sequence scores 0. Requiring inspect-ai >= 0.3.253 for these tasks (or otherwise ensuring the fixed scorer is used) would do it. Happy to open a PR for the pin bump. Results produced with older pins can only be inflated, never deflated, because the old comparison marks more answers correct.

Version info

lighteval main; inspect-ai 0.3.252 vs 0.3.253; Python 3.13.5, Windows 11.

(Claude helped write this up; I ran the repro myself, on both wheels.)

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions