Repository navigation
Fix gpqa-fi inspect backend: Finnish prompt + Finnish answer parsing - #27
Merged
Merged
Conversation
added 2 commits
September 28, 2026 12:30
The inspect-ai path used multiple_choice() with inspect's default English
template and the choice() scorer, which only parses 'ANSWER: X'. Finnish
responses ('Vastaus: X') were therefore scored incorrect.
- Render the Finnish instruction via a multiple_choice template, shared
with the native prompt functions.
- Score with multichoice_scorer(language=Language.FINNISH), which uses
lighteval's language-aware index extraction ('vastaus' anchor).
- multichoice_scorer: accept a language, and build regexes for all
choices (previously len_choices=1, so only 'A' was extractable).
Metrics.gpqa_instruct_pass_at_k extracts with English anchors. Use a Finnish counterpart (same config, Language.FINNISH) so 'Vastaus: X' is recognised.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Reported by Kai Hakala:
gpqa-fi:diamondon the inspect-ai backend (what TW runs) sent inspect's default English multiple-choice prompt and scored withchoice(), which only parsesANSWER: X. Finnish answers (Vastaus: X) were scored incorrect.Changes
gpqa_fi.py: inspect solver ismultiple_choice(template=...)using the same Finnish instruction as the native prompt functions (GPQA_FI_INSTRUCTION, now shared); scorer ismultichoice_scorer(language=Language.FINNISH). Applied to bothgpqa-fiandgpqa-fi:diamond; versions bumped.metrics.py:multichoice_scorertakes alanguage(default English, no other callers) and builds regexes forlen(state.choices)letters. Previously it usedlen_choices=1, so onlyAcould ever be extracted.Verification (TW, vLLM 0.20 container, deployed runtime + this patch)
mockllm smoke, 6 samples of
gpqa-fi:diamondwith scripted completions:...Vastaus: **A**(correct)Vastaus: A(correct)vastaus: C.(correct)Vastaus: C(wrong)... Vastaus: C(wrong)ANSWER: B(correct)Prompt before:
Answer the following multiple choice question...; after:Vastaa seuraavaan monivalintakysymykseen....Native backend:
gpqa-fi:diamondnow uses a Finnish counterpart ofgpqa_instruct_pass_at_k(same config,Language.FINNISH).Crafted-completion check (native Finnish / old English / inspect Finnish), all through the real scorer code on TW:
Vastaus: A,**Vastaus: A**,vastaus: (A).→ correct in allVastaus: A. Vaihtoehto B on väärä.→ Finnish correct, old English metric picks the distractorBKnown limitation, shared with English
gpqa:diamond: inlazy_indices_regexthe letter suffix does not accept a newline, soVastaus: A\n\n<more text mentioning other letters>falls back to the last bare letter. Left unchanged on purpose to keep parity with the English task.