VALIDATION_RESULTS.md

Validation Results: agent-shell-contract

2026-09-23 - Primary public wheel onboarding

Selection evidence: the public v0.2.0 release has a verified wheel and source distribution; the README exposed both but led the quickstart with the source archive. The current authenticated window through 2026-09-22 reports 0 views / 0 unique viewers and 104 clones / 35 unique cloners. These are awareness signals, not confirmed usage.

User outcome: a maintainer can copy the first quickstart and install the exact released shell-contract CLI from the prebuilt wheel without Git, a checkout, or PyPI; CI now exercises that same public URL.

Implementation: README now leads with the exact wheel, keeps the source distribution as the platform fallback, and retains the immutable archive for environments that can build from source. Added a checkout-free public-wheel-install job and regression coverage for its pinned URL, no-checkout shape, and README ordering.

Local verification: Python 3.14.6 passed all 20 tests, compileall, workflow YAML parsing, git diff --check, and a scoped credential-pattern scan with 0 findings. The public wheel URL returned HTTP 200 and 23,066 bytes. A separate clean virtual environment installed agent-shell-contract==0.2.0, listed fixtures, and passed the output-before-exit reference smoke.

Public verification: Actions run 35835955110 passed all 13 jobs for b2b8e4e, including the Python 3.10/3.11/3.12 Ubuntu/macOS/Windows matrix, Git install, wheel install, immutable archive install, and optional-adapter smokes. GitHub reported only pre-existing Node.js 20 and ubuntu-latest migration warnings; no change-specific failure occurred.

Adoption remains UNKNOWN; clone traffic is an awareness signal and does not establish product usage.

2026-09-18 - Direct release-asset onboarding

Selection evidence: the public v0.2.0 release has a wheel and source distribution, but the README exposed only Git and source-archive install paths. The latest known awareness window is 9 views / 2 unique viewers and 82 clones / 26 unique cloners through 2026-09-14; these are awareness signals, not confirmed usage.

User outcome: a maintainer can install the exact released shell-contract CLI from a prebuilt wheel, or from the release source distribution, without Git, a checkout, or PyPI publication.

Implementation: README documents the exact v0.2.0 wheel and source-distribution URLs; tests/test_workflow.py prevents the two release-asset URLs from drifting or being duplicated.

Local verification: Python 3.14.6 passed all 19 tests, compileall, workflow YAML parsing, git diff --check, and a scoped credential-pattern scan. Both release asset URLs returned HTTP 200; the wheel was 23,066 bytes and the source distribution was 22,410 bytes. Separate clean virtual environments installed each asset, reported package version 0.2.0, listed fixtures, and passed the CLI help smoke.

Public verification: Actions run 35378627603 passed all 12 jobs for commit eae416e, including the Python 3.10/3.11/3.12 Ubuntu/macOS/Windows matrix, public release install, immutable archive install, and optional adapter smokes. GitHub reported Node 20 action-runtime and future Ubuntu label migration warnings; no job failure or change-specific warning occurred.

2026-09-11 - Checkout-free public archive install

Selection evidence: the public v0.2.0 release has no assets, PyPI returns HTTP 404, and the README's released-install paths required Git. The last-known awareness snapshot is 78 clones / 16 unique cloners.

User outcome: a maintainer can install the exact released shell-contract CLI without Git, a repository checkout, or PyPI publication; CI independently verifies that path.

Implementation: README documents the immutable GitHub source archive for release commit c018e00433d09a82c71a83d648d214d296b90f6b; the no-checkout public-archive-install job installs that archive and runs the reference suite; tests/test_workflow.py prevents the README and workflow from drifting to another archive or adding checkout to the archive job.

Local verification: Python 3.14.6 passed all 18 tests, including the archive/workflow regression; compileall, workflow YAML parsing, git diff --check, and the mutable-action and credential-pattern scans passed. A clean temporary virtual environment installed the exact public archive, reported package version 0.2.0, listed fixtures, and passed the output-before-exit reference smoke. The archive endpoint returned HTTP 200 and the release tag resolves to the same commit.

Public verification: Actions run 34631891779 completed successfully for f22dd0d, including the new archive-install job and the existing Python 3.10/3.11/3.12 Ubuntu/macOS/Windows matrix, public-install, and optional-adapter jobs. Remote main resolves to f22dd0d111cceca4ba03ff66fcf3723523b70356.

Adoption remains UNKNOWN; clone traffic and release readiness do not establish product usage.

2026-09-03

Maintenance validation:

• Replaced all five mutable actions/checkout@v4 and actions/setup-python@v5 workflow refs with the verified full commit pins for the same action versions.
• Added tests/test_workflow.py to enumerate every workflow action call and fail on pin drift or a non-40-character ref.

Local verification:

• PASS: 16 focused unit tests, including the workflow-pin regression and all non-background reference-adapter tests.
• BLOCKED: the full 17-test suite's background-server-lifecycle test is denied by the host's macOS sandbox with PermissionError: [Errno 1] Operation not permitted before it can bind its owned port; this is an environment limitation, not a workflow-pin failure.
• PASS: Python compileall, workflow YAML parsing with the installed Ruby/Psych version, git diff --check, and a scoped mutable-action-ref scan with no findings.
• Upstream tag verification: actions/checkout v4 resolves to 11d5960a326750d5838078e36cf38b85af677262; actions/setup-python v5 resolves to a26af69be951a213d495a4c3e4e4022e16d87065.
• Public verification: Actions run 33732183838 passed all nine Python 3.10/3.11/3.12 Ubuntu/macOS/Windows matrix jobs, the public release install smoke, and the optional adapter smoke job.

Adoption remains UNKNOWN; this maintenance change improves CI reproducibility and does not establish product usage.

2026-06-27 11:43 Europe/Amsterdam

Lifecycle mode: VALIDATE.

Target validation:

• Validate the optional Pydantic AI Harness adapter in an isolated dependency environment.
• Confirm whether the fixture runner produces meaningful contract results beyond the local subprocess reference adapter.
• Decide whether the project should continue toward a second stable adapter shape or be repositioned upstream as Pydantic-only.

Environment:

• Command: uv run --python /opt/homebrew/bin/python3.12 --with pydantic-ai-harness --with-editable . python -m agent_shell_contract run --adapter pydantic-ai-harness --format json
• Python: 3.12.11
• Platform: macOS-15.5-arm64-arm-64bit
• Pydantic AI Harness package import succeeded from uv cache.
• uv created local .venv and uv.lock; both were removed after validation because they were incidental tool artifacts.

Result:

• PASS: 7
• FAIL: 1
• SKIP: 1
• ERROR: 0
• UNSUPPORTED: 0

Fixture outcomes:

• timeout-child-pipe: PASS; timeout returned without hanging on child pipes.
• timeout-process-tree: PASS; timeout terminated the owned process tree.
• background-server-lifecycle: PASS; background start/check/stop closed the owned port.
• output-before-exit: PASS; stdout, stderr, and exit status survived.
• output-after-kill: FAIL; timeout returned promptly but bounded pre-timeout stdout/stderr was not included in the timeout result.
• cwd-isolation: PASS; adapter declares sticky cwd support and behavior was recorded.
• env-boundary: PASS; explicit env boundary excluded the host sentinel variable.
• pty-detection: PASS; observed no-PTY behavior matched adapter declaration.
• windows-tree-termination: SKIP; Windows-only fixture skipped on macOS.

Gate impact:

• Pydantic AI Harness adapter feasibility: PASS.
• Meaningful external harness validation: PASS; the suite produced actionable pass/fail contract output against a real harness adapter.
• Specific Pydantic timeout-output contract: FAIL; timeout results drop bounded pre-kill output.
• Project-not-Pydantic-only decision: PASS; the runner has a reference adapter plus a working external harness adapter, and the failure is a reusable shell-semantics contract case rather than a Pydantic-specific unit test.
• Publication validation: UNKNOWN; still needs a second stable external adapter shape or a documented route showing ACP/native-client feasibility without UI automation or paid model calls.

Decision:

Continue incubation. Do not publish. Do not reposition upstream yet. The Pydantic result is useful validation, but publication still needs broader adapter evidence.

Exact next action:

Validate a second stable adapter shape. Prefer ACP terminal methods if a local implementation or conformance target can be run without UI automation; otherwise inspect Codex/Claude headless shell surfaces and kill or hand off if the remaining useful work is only a Pydantic AI Harness fixture contribution.

2026-06-27 12:20 Europe/Amsterdam

Lifecycle mode: VALIDATE.

Target validation:

• Validate a second adapter shape after the Pydantic AI Harness run.
• Prefer direct shell-runner APIs that do not require UI automation or paid model calls.

Adapters validated:

• acp-local-terminal: optional adapter using the official agent-client-protocol Python schema and ACP v1 terminal method names with a local in-process terminal client.
• codex-app-server: experimental adapter over Codex CLI app-server command/exec, using direct JSON-RPC shell execution without model calls.

Environment:

• ACP command: uv run --python /opt/homebrew/bin/python3.12 --with agent-client-protocol --with-editable . python -m agent_shell_contract run --adapter acp-local-terminal --json reports/acp-local-terminal.json --markdown reports/acp-local-terminal.md
• Codex command: PYTHONPATH=src python3 -m agent_shell_contract run --adapter codex-app-server --json reports/codex-app-server.json --markdown reports/codex-app-server.md
• Codex version: codex-cli 0.133.0
• Codex app-server methods tested: initialize, command/exec, command/exec/outputDelta, and command/exec/terminate.
• Platform: macOS-15.5-arm64-arm-64bit

Results:

Adapter	PASS	FAIL	SKIP	Notes
subprocess-reference	8	0	1	Baseline local adapter.
pydantic-ai-harness	7	1	1	output-after-kill drops bounded pre-timeout output.
acp-local-terminal	8	0	1	Protocol-shape validation, not a specific ACP product claim.
codex-app-server	8	0	1	Real local Codex shell surface, but app-server is experimental.

Saved reports:

• reports/subprocess-reference.json
• reports/subprocess-reference.md
• reports/pydantic-ai-harness.json
• reports/pydantic-ai-harness.md
• reports/acp-local-terminal.json
• reports/acp-local-terminal.md
• reports/codex-app-server.json
• reports/codex-app-server.md

Gate impact:

• Second adapter validation: PASS, narrowly. Codex app-server is a real local direct shell surface that ran the full suite without model calls; ACP local validates a protocol-shaped adapter. Caveat: Codex app-server is experimental and ACP local is not a pass claim for a specific ACP client.
• Real-world cases: PASS for fixture classes mapped to Codex timeout/output/process issues, Claude/Pydantic background lifecycle issues, and Copilot/terminal output/cwd pain in the demand evidence.
• Publication validation: PASS for public release readiness after CI/docs/safety/discoverability checks and final GitHub Actions verification.

Decision:

Continue to public publication after the final release-candidate CI run passes.

2026-06-28 21:28 Europe/Amsterdam

Lifecycle mode: MAINTAIN.

Target validation:

• Harden the experimental Codex app-server adapter against schema/version drift.
• Keep the compatibility claim bounded to the local Codex CLI version and methods tested.

Environment:

• Codex version: codex-cli 0.133.0
• Codex app-server methods tested: initialize, command/exec, command/exec/outputDelta, and command/exec/terminate.
• Live command: PYTHONPATH=src python3 -m agent_shell_contract run --adapter codex-app-server --format json
• Platform: macOS-15.5-arm64-arm-64bit

Result:

• PASS: 8
• SKIP: 1
• FAIL: 0
• ERROR: 0

Gate impact:

• Codex app-server compatibility guard: PASS; the adapter reports JSON-RPC errors, non-object results, missing or mistyped exitCode / stdout / stderr, and malformed output-delta notifications as method-specific compatibility errors instead of silently accepting mismatched response shapes.
• Stable Codex API support claim: FAIL; app-server remains experimental and version-sensitive.

Decision:

Continue maintaining codex-app-server as experimental validation evidence only. Do not claim stable Codex API compatibility.

2026-06-29 17:58 Europe/Amsterdam

Lifecycle mode: MAINTAIN.

Target validation:

• Reproduce the Pydantic AI Harness output-after-kill contract failure from owner-created issue #2.
• Produce a minimal upstream-ready report without claiming maintainer interest or doing external upstream work from the OWNED lane.

Environment:

• Command: uv run --python /opt/homebrew/bin/python3.12 --with pydantic-ai-harness --with-editable . python -m agent_shell_contract run --adapter pydantic-ai-harness --fixture output-after-kill --json reports/pydantic-ai-harness-output-after-kill.json --markdown reports/pydantic-ai-harness-output-after-kill.md
• Pydantic AI Harness version: 0.4.0
• Python: 3.12.11
• Platform: macOS-15.5-arm64-arm-64bit

Result:

• FAIL: 1
• Fixture: output-after-kill
• Observed result: timeout returned promptly, but stdout and stderr were empty and raw output was only [Command timed out after 1.0s].

Saved reports:

• reports/pydantic-ai-harness-output-after-kill.json
• reports/pydantic-ai-harness-output-after-kill.md
• reports/pydantic-ai-harness-output-after-kill-report.md

Decision:

Keep the report local and upstream-ready. Do not open a Pydantic AI Harness upstream issue from this OWNED-lane run; a separate UPSTREAM run should first check that repository's contribution rules and current behavior.

2026-07-29 10:24 Europe/Amsterdam

Lifecycle mode: MAINTAIN.

Target validation:

• Complete owner issue #1 by validating the Windows process-tree fixture in hosted CI.
• Keep the Windows claim limited to the reference adapter and the tested GitHub Actions runner/Python versions.

Implementation:

• Commit 7d3b1f9 added Windows taskkill /PID <pid> /T /F tree termination and the windows-latest matrix.
• Commit 3c6a597 fixed Windows shell quoting by base64-encoding fixture Python source before the -c bootstrap; this was required after the first Windows matrix run failed in unit tests.

Environment:

• GitHub Actions run 30435210271 on windows-latest with Python 3.10, 3.11, and 3.12.
• Local macOS Python 3.12.11 verification: 16 unittests, compileall, direct Windows-style encoded-command smoke, reference suite 8 PASS / 1 SKIP, git diff --check, and scoped credential scan.

Result:

• PASS: all 9 matrix jobs completed successfully, including all three Windows jobs and the windows-tree-termination fixture; the separate optional-adapter smoke job also completed successfully.
• The original failure was Windows-incompatible quoting of embedded Python -c source; the encoded bootstrap correction passed the rerun.

Public action:

• Pushed 3c6a597 to main.
• Closed owner issue #1 with the implementation and CI evidence.

Decision:

Maintain at low intensity. Do not claim Windows behavior for other adapters or client products from this reference-adapter result.