Next release: data safety, scoring correctness, recovery and package validation checklist

[reacher-z]

Goal

Use this as the next-release quality checklist, with a bounded set of verified defects and existing review work rather than opening duplicate refactor requests. Suggested ordering below is for maintainer review; no release version or date is assumed.

Suggested release blockers: data safety and truthful results

• [P1] reproduce: cleanup deletes unrelated files in the user-supplied work directory #361 — reproduction cleanup deletes unrelated user files in an existing --work-dir. Fix in the focused PR linked below; test PASS, FAIL, download/judge errors, interruption and overlapping invocations.
• [P1] reproduce: validate sample coverage and selected rubrics before reporting PASS/FAIL #362 — reproduction ignores sample coverage and compares unrequested rubric columns as zero. Define valid full versus partial reproduction before returning PASS.
• [P1] rescore: stale judge caches are silently attributed to a newly requested judge model #363 — a new judge model can silently reuse old verdicts and relabel the summary. Require truthful provenance/cache matching.

Recovery and existing CI coverage

• [P2] rescore: truncated judge cache is permanently skipped instead of retried #364 — corrupt judge JSON is skipped before worker recovery. Share cache validation, use atomic verdict writes, and define partial-error reporting.
• [P2] CI: runtime-server changes select zero downstream harness image builds #365 — runtime-server/ changes select no downstream native harness builds. Correct/test the existing dependency selector.

Review existing work instead of duplicating it

• fix(batch): resume from run-meta.json, not from log-file existence #329 / batch --resume skips failed jobs (log-file heuristic) and overwrites batch-summary.json with 'skipped' rows #297: batch resume must be driven by run metadata and preserve prior results.
• fix(judge): collapse judge_llm.py onto judge.py's transport #319 / judge_llm.py has diverged from judge.py (no Gemini support, wrong responses endpoint, 800-token cap) — collapse the duplication #300: shared judge transport behavior.
• refactor(interceptor): one implementation of Stage-1 matching, with a parity matrix #354 / Stage-1 interceptor matching is duplicated between runtime-server and edgebench_judge, and the live copy has zero tests #301: shared Stage-1 interceptor semantics and parity tests.
• feat(batch): report Stage 1 and Stage 2 together, never one alone #340 / Always report both Stage-1 (interception) and Stage-2 (judged); raw interception overcounts ~2× #243: report Stage 1 and Stage 2 together with explicit denominators.
• fix(rescore): resolve paths from the workspace instead of a maintainer's home #313 / rescore: public reproducibility CLI defaults to a maintainer's home paths and silently no-ops elsewhere #296: portable rescore paths and explicit invalid-input failures.
• feat(runner): record code, corpus, and agent revisions in run-meta.json #347: record run/corpus/agent provenance; coordinate with evaluator cache provenance in [P1] rescore: stale judge caches are silently attributed to a newly requested judge model #363.

Maintainers can explicitly defer a candidate, with the reason and remaining limitation recorded here. This checklist does not declare every existing feature PR a release blocker.

Validate the actual release candidate

• Run the existing host tests, Ruff, Pyright, corpus validation and Markdown checks on the combined candidate commit; results from separate PR heads do not prove the combined candidate is sound.
• Build wheel and sdist, run metadata checks, then install each in a clean environment outside the source checkout. Verify packaged CLI --help, bundled corpus discovery/JSON loading, and a one-task offline adapter conversion. This catches source-checkout-only imports and missing package data.
• Ensure changes to native runtime dependencies trigger the correct existing image build matrix ([P2] CI: runtime-server changes select zero downstream harness image builds #365).
• Update only Unreleased during development; choose the release version/date and reconcile pyproject.toml, tag and changelog when maintainers cut the release.
• Release notes distinguish verified functionality from untested runtime/provider combinations and state any deferred correctness limitations.

Harbor scope

Keep #331 as the Harbor integration tracker; review #353, #346 and #360 there, with #358/#359 covering remaining lifecycle/upstream acceptance work. Do not reopen deferred Harbor CI #350 or make image publication a new prerequisite. Loader checks are not browser/provider E2E or parity evidence.

Audit evidence

The five new bugs were checked against main 9dd9d44; reproduce/rescore sources match the current working branch. Offline temporary-directory/mock reproductions verified user-file deletion, sample-count acceptance, stale-cache attribution and corrupt-cache skipping. The CI selector was exercised with a runtime-server-only input. No live model calls, release publication or real user-data deletion were performed.

@Perry2004 please review the suggested blocker ordering, the linked fixes, and which items must be completed versus explicitly deferred for the next release.

[reacher-z]

@Perry2004 — the first release-quality fix is ready for review: #367 (fixes #361), preserving existing user files by isolating reproduction downloads in per-invocation child directories.

Validation on the PR revision: 11 new offline cleanup regressions; full suite 294 passed / 3 skipped; Ruff and Pyright pass; wheel and sdist build successfully. Formal review requested from you on the PR.

Suggested next order: #362 and #363 for truthful reproduction/scoring, then #364 recovery and #365 existing CI selection. Existing #313/#319/#329/#340/#347/#354 remain their own review work. Please confirm or adjust the proposed release blockers in this checklist; no release has been published.

[reacher-z]

@Perry2004 — public-facing release readiness now has a focused correction PR and three publicity deliverables:

• docs: align public metadata and distinguish historical benchmark scores #373 fixes docs: correct public discovery metadata and label historical benchmark scores #369: canonical package/contribution links, historical V1 score labeling in both READMEs, shipping corpus counts in repository llms.txt, and v0.10.0 citation date. Formal review requested from you. Markdown links, wheel/sdist build and built wheel metadata checks passed.
• publicity: turn the model leaderboard call-to-action into an evidence-based submission guide #370: a concrete model-result submission guide/form so publicity readers can become contributors with reviewable evidence.
• publicity: prepare a reproducible next-release demo and bilingual announcement kit #371: next-release annotated demo, working CLI examples and bilingual announcement draft tied to actual shipped fixes.
• publicity: audit existing outreach PRs and make featured-in claims traceable #372: audit existing outreach PRs and substantiate the featured-in count before adding more submissions.

These items concern repository artifacts and release materials. The website quick-start mismatch is documented as an owner handoff in #371; private website work remains outside this repository, consistent with your earlier triage. Nothing here claims a new release, website deployment or official Harbor registration.

[reacher-z]

@Perry2004 — #374 now fixes #365 with an extracted, testable native harness build selector. Formal review requested.

Verification: 17 targeted tests; full suite 300 passed / 3 skipped; Ruff and Pyright pass. The actual workflow selection step with stubbed Git inputs selects 12 harnesses for shared runtime-server changes, 1 for Hermes, 12 for manual runs, and 0 native harnesses for Harbor-only changes. No local Docker build is claimed.

This is a concrete maintenance-quality improvement for the next release: shared dependencies are explicitly modeled, deleted paths participate in selection, and matrix output is covered by tests. #365 should close when #374 merges, not merely because the PR exists.