Skip to content

Add HumanEval+/MBPP+ (EvalPlus) as native sandboxed LightEval tasks - #24

Merged
ezosa merged 1 commit into
mainfrom
add-evalplus-tasks
Sep 10, 2026
Merged

ezosa merged 1 commit into
mainfrom
add-evalplus-tasks

Conversation

@ezosa

@ezosa ezosa commented Sep 10, 2026

Copy link
Copy Markdown

What

Adds humaneval_plus and mbpp_plus (EvalPlus) as native inspect-ai LightEval tasks, migrating these code-generation benchmarks off lm-eval-harness so they use LightEval's wiring, dashboard sample logging, and reasoning-trace separation.

New package src/lighteval/tasks/tasks/evalplus/ (auto-discovered like bfcl/):

  • main.py — BFCL inspect-ai shape (record_to_sample + solver=[generate(cache=True)] + @scorer). Instruct-style prompt; extracts code from the model's post-reasoning final answer (with reasoning_parser=qwen3 the completion is the answer, not the <think> trace); grades by executing against EvalPlus's augmented unit tests. pass@1 estimated over N samples via the wrapper's epochs (mean reducer) — no custom metric.
  • execution.py — check_correctness(sample, generation, timeout) / run_test(sample, test, timeout) grader that deliberately mirrors lcb's signature + multiprocessing.Process pattern, so the TensorWave network sandbox (sandbox_grading.py) wraps it via a single generic wrapper with no per-task code. The sample carries a one-element input_output purely so the wrapper's timeout math (which indexes sample["input_output"]["inputs"]) works unchanged. reliability_guard is vendored from OpenAI human-eval (defensive only; real isolation is the sandbox's CLONE_NEWUSER|CLONE_NEWNET).
  • build_data.py — normalizes evalplus/humanevalplus + evalplus/mbppplus into a uniform multi-config HF dataset (ezosa/evalplus, configs humaneval_plus/mbpp_plus), deriving MBPP's entry point from its reference code.
  • tests/unit/tasks/test_evalplus.py — pure-stdlib grading tests: correct→pass, wrong/raise/timeout/no-function→fail, and code extraction (incl. post-reasoning answers).

Why

The lm-eval *_instruct code tasks force an empty <think></think> block into the prompt, so the model never reasons. This migration keeps reasoning ON and gives proper LightEval logging.

Safety

Execution is unsafe and must run under eval_lighteval_sandboxed.sbatch. The consuming TensorWave repo registers the task prefixes (humaneval, mbpp) in sandboxed_evals.SANDBOXED_TASK_PREFIXES and the grader in GRADER_PATCHES (separate PR).

Verification

  • Unit tests pass (grader validated locally under fork).
  • ruff check + format clean.
  • Registry auto-discovers evalplus/main.py; humaneval_plus/mbpp_plus resolve.

🤖 Generated with Claude Code

Adds `humaneval_plus` and `mbpp_plus` as native inspect-ai tasks (BFCL shape:
record_to_sample + generate(cache=True) solver + @scorer), migrating these code
benchmarks off lm-eval-harness so they use LightEval's wiring, dashboard sample
logging, and reasoning-trace separation.

- evalplus/main.py: instruct prompt, extracts code from the (post-reasoning)
  final answer, grades by executing against EvalPlus's augmented tests. pass@1
  is estimated over N samples via the wrapper's epochs (mean reducer).
- evalplus/execution.py: check_correctness/run_test grader deliberately mirroring
  lcb's signature + multiprocessing.Process pattern so the TensorWave network
  sandbox (sandbox_grading.py) wraps it with no custom wrapper; the sample carries
  a one-element input_output so the generic wrapper's timeout math works unchanged.
- evalplus/build_data.py: normalizes evalplus/humanevalplus + evalplus/mbppplus
  into a uniform multi-config HF dataset (ezosa/evalplus).
- tests/unit/tasks/test_evalplus.py: pure-stdlib grading tests (pass/fail/raise/
  timeout/no-func + code extraction incl. post-reasoning answers).

Reasoning stays ON (unlike the lm-eval *_instruct tasks, which force an empty
think block); execution MUST run under eval_lighteval_sandboxed.sbatch.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ezosa
ezosa merged commit c69a5a6 into main Sep 10, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant