Skip to content

Sync mutate_core.py: refuse a file the binary under test is not built from - #186

Merged
martinus merged 1 commit into
masterfrom
claude/oans-testing-improvements-wslk93
Aug 23, 2026
Merged

martinus merged 1 commit into
masterfrom
claude/oans-testing-improvements-wslk93

Conversation

@martinus

Copy link
Copy Markdown
Owner

Re-copied from unordered_dense, where the change was made and where
scripts/test_mutate.py covers it (martinus/unordered_dense#206).

A file no test binary includes scores every mutant survived, so the report
reads as an indictment of the tests when nothing was measured at all — the
flattering direction, which is the one this tool must never fail in. It was
found in oans, whose src/tests.c #includes most of src/*.c but deliberately
not oans.c: sweeping that planned 1,394 mutants that would all have come back
survived, presented as a 0% kill rate.

The probe appends an #error to the target and builds. If the build still
succeeds, nothing the test binary is made of ever included the file — which
needs to know nothing about the build system, and so works the same under cmake
here, meson there and make in oans.

Nanobench cannot hit that case today: src/include/nanobench.h is included by
every translation unit in the suite, so the probe accepts it. Verified end to end
here rather than assumed, since this is the cmake backend and the change was
developed against make — the probe passes, the baseline builds, and mutants score
caught exactly as before.

Why this is a separate PR rather than nothing

This is the gap vendoring introduces, and it bit. The file was copied into all
three working trees weeks ago, but only oans's copy reached a default branch —
so this repository has been running the old core with lint-mutate-core.py
green, because it records the hash of whatever it has committed. No lint in
any of the three can see the other two.

Only mutate_core.py and its .sha256 change; the adapter beside it is
untouched, and nothing in src/include/ or the test suite is affected.


Generated by Claude Code

… from

Re-copied from unordered_dense, where the change was made and where
scripts/test_mutate.py covers it. A file no test binary includes scores
every mutant `survived`, so the report reads as an indictment of the tests
when nothing was measured at all - the flattering direction, which is the
one this tool must never fail in.

Nanobench cannot hit that case today: src/include/nanobench.h is included
by every translation unit in the suite, so the probe accepts it. Verified
end to end here rather than assumed, since this is the cmake backend and
the change was developed against make: the probe passes, the baseline
builds, and mutants score `caught` exactly as before.
@martinus
martinus merged commit 61f9e8a into master Aug 23, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants