Repository navigation
Fix MBPP+ grader scoring 0.0 (unconditional check(entry_point) call) - #25
Merged
Merged
Conversation
…tests MBPP+ test harnesses assert against the target function directly at module scope and define no `check` wrapper, whereas HumanEval+ tests define `def check(candidate)` and must be invoked as `check(entry_point)`. `_build_program` appended `check(entry_point)` unconditionally, so every MBPP+ problem raised `NameError: name 'check' is not defined` and the whole task scored 0.0 (confirmed on a real iter16000 run: mbpp_plus=0.0000 with sub-0.06 truncation). Detect the `check` definition and only emit the call when present; otherwise the harness's module-level asserts run directly. Add MBPP-style (direct-assert) grader tests plus program-shape assertions so the two harness styles are both covered — the previous tests only exercised the HumanEval `check()` shape, which is why CI stayed green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
A real iter16000 run surfaced mbpp_plus = 0.0000 across all 378 problems (with sub-0.06 truncation, so generations were fine). Root cause:
execution._build_programappendedcheck(entry_point)to every problem, but the two EvalPlus benchmarks structure theirtestfield differently:def check(candidate): ...→ must be invoked ascheck(entry_point).checkwrapper; its assertions call the target function directly at module scope.So every MBPP+ problem raised
NameError: name 'check' is not defined→ the whole task scored 0. HumanEval+ was unaffected (0.622 on the same run).Fix
Only append
check(entry_point)when the harness actually definescheck(regex on theteststring); otherwise concatenategeneration + testso the module-level asserts execute directly.Tests
The prior tests only exercised the HumanEval
check()shape, which is why CI stayed green. Added:check(...)call appended; HumanEval one doesVerified locally: humaneval correct/wrong pass/fail unchanged; mbpp correct now passes (was failing → 0.0).
🤖 Generated with Claude Code