python_bugfix_diff

Code review skill — real bugfix-commit detection

A code-review Claude Skill (SKILL.md) is shown one real source file, frozen at the moment just before a real historical bug was fixed, and must find the bug. Ground truth is the actual fix commit — not a synthetic injected bug.

Why this task

Community "AI code reviewer" skills are one of the most duplicated categories in the Claude Skills ecosystem — everyone has built one, nobody knows which is actually good at catching real bugs versus producing plausible-sounding comments. This task is a shootout: same 10 real diffs, same scoring, whichever skill catches the most real bugs wins.

Files

File	Role
gold.cases.json	Source of truth — 10 real bugfix-commit cases. Edit here.
build_cases.py	Generates inputs/ + expected/ and validates invariants.
judge.py	Per-case scoring.
grader.py	Run-level aggregation.
inputs/<id>/question.txt	GENERATED — the prompt shown to the skill.
expected/<id>/answer.json	GENERATED — the gold bug location + keywords.

Regenerate after editing cases: python3 build_cases.py

Input / output contract

The skill receives inputs/<id>/question.txt: one file's contents shown at its real absolute line numbers, plus instructions to reply with a single JSON object:

{"findings": [{"file": "<path>", "line": <int>, "description": "<1-2 sentences>"}, ...]}

Findings should be ordered most-confident-first. Only the first 5 findings are scored (anti-shotgun) — flagging every line in the file does not help.

Scoring

• Per case (judge.py, deterministic, no LLM judge): a finding "hits" the gold bug only if ALL THREE hold — (1) file matches the case's file by basename, (2) line is within line_tolerance (default 2, tightened to 1 for case_07 and case_10, where the buggy line sits at the very edge of the shown snippet) of the real bug line, (3) description contains at least one of the case's pre-curated keywords as a whole word (word-boundary match, case-insensitive). Score is 1.0 if any of the first 5 findings hits, else 0.0. Non-hitting runs still surface best_match_signals (which of the 3 signals came closest) as an ungraded diagnostic. Keyword curation note: each case's keywords list includes several natural synonyms/inflections of the core term (e.g. case_01 has both off-by-one and off by one; case_04 has concurrent, concurrently, and concurrency) — added after real testing against community skills showed correct diagnoses getting scored as misses purely because the skill's phrasing didn't exactly match a single curated phrase. This doesn't eliminate the risk (a good but differently-worded description can still miss), but it substantially narrows it.
• Per run (grader.py): mean score across cases; n_passed counts full hits; by_category breaks score down by bug category. Run passes at mean ≥ 0.5 (deliberately lower than connections' 0.75 — these are real bugs in real unfamiliar code, not a closed-form puzzle).

Sources & licensing

Every case's snippet is a verbatim excerpt of a real pre-fix file, reproduced under the source repo's own permissive license (attribution below satisfies the license notice requirement for the small excerpts used here):

case	bug category	source	license
case_01	off-by-one	pallets-eco/croniter@3ddcd13	MIT
case_02	null/None deref	googlefonts/gftools@f28317c	Apache-2.0
case_03	logic error	sgl-project/SpecForge@361f5a1	MIT
case_04	race condition	google/bumble@55d8171	Apache-2.0
case_05	resource leak	miketheman/pytest-socket@2aaaee1	MIT
case_06	missing auth check	homeassistant-ai/ha-mcp@9f5b085	MIT
case_07	mutable default arg	xarray-contrib/xarray-spatial@5103dc1	MIT
case_08	broad except	xarray-contrib/xarray-spatial@c6daae5	MIT
case_09	cache invalidate KeyError	Suor/funcy@a96b449	BSD-3-Clause
case_10	float truncation	spulec/freezegun@9591645	Apache-2.0

Known limitation — leakage risk: because these are real commits (not synthetic bug injection), a skill's underlying model may have seen the exact commit during pretraining. Mitigated, not eliminated, by preferring smaller repos (roughly 277–4,517 stars at time of writing, not framework-scale mega-repos) and recent commits.

Why case_09/case_10 exist: the original 8 cases included two "textbook famous" Python gotchas (case_07 mutable default argument, case_08 bare except) that every community skill tested against v1 caught instantly — likely pattern-matched from widely-discussed blog posts rather than reasoned about from the code itself, giving them poor long-term discriminative power. case_09 and case_10 were deliberately sourced to be subtler and much less discussed (a cache-invalidation KeyError and a float-truncation-order bug), while still meeting the same real-commit/permissive-license/modest-repo bar. One candidate considered and rejected for this round was a real, cleanly verified path-traversal CVE (formally tracked with a GHSA advisory) — passed over specifically because a formally cataloged CVE is arguably more likely to be indexed in training data than an obscure, unlabeled bugfix commit, which would work against the same discriminative-power goal. The design doc also floated renaming identifiers as an additional mitigation; v1 ships the snippets verbatim instead (see "Out of scope" in the implementation plan for why). See docs/superpowers/specs/2026-07-10-code-review-skill-task-design.md in the trapstreet-tasks repo for the full design writeup and tradeoffs.

Run

python3 build_cases.py                 # (re)generate cases
python3 -m pytest tests/ -v            # unit tests