Add an inspectable fault-injection reliability benchmark

[PranavMishra28]

Summary

• add an executable, adapter-backed reliability benchmark with matched baseline and faulted runs
• support deterministic schedules for sandbox restart, lost dispatch response, SIGKILL mid-tool-call, and network partition faults
• persist inspectable event, fault, recovery, metrics, and irreversible-effect evidence before grading
• grade completion/resume, exactly-once irreversible effects, and recovery overhead with explicit reason codes
• add JSON and Markdown scorecards, a reliability-eval CLI, an example scenario, and an SDK event callback bridge
• document the verified benchmark/SDK/agent-server integration seams in ARCHITECTURE.md and the public contract in DESIGN.md

Design boundary

The benchmark core is fully executable and tested. Infrastructure-specific operations remain explicit adapter capabilities (restart_sandbox, drop_dispatch_response, sigkill_mid_tool_call, partition_network, and heal_network) because their concrete implementation depends on the selected runtime and its privileges. This keeps the graders and schedule portable while making every injected action inspectable.

The SDK bridge runs persistence callbacks before observing an event for fault injection. This preserves the central persist-before-inject invariant and avoids depending on hidden model reasoning.

Evidence and validation

• uv run pre-commit run --files ...
  • Ruff format: passed
  • Ruff lint: passed
  • pycodestyle: passed
  • Pyright strict: passed
• uv run pytest tests/test_reliability.py tests/test_async_evaluation.py tests/test_iterative_resume.py
  • 29 passed
• make build
  • passed
• broader uv run pytest tests
  • 538 passed
  • 13 ProgramBench shell-hook tests could not run on macOS because GNU timeout is not installed; those failures are unrelated to this change

Engagement and context

• RFC: RFC: a fault-injection reliability benchmark for agent runs #764
• Existing remote-run failure: ACP Agent: workspace_keepalive not preventing all 'Remote conversation ended with error' failures #488
• Sandbox restart/session recovery: ACP: conversations cannot be resumed after sandbox restart (session_id is not persisted) OpenHands#14260
• Sandbox startup and persistence failure: [Bug]: Sandbox failed to start within 120s OpenHands#13578

[PranavMishra28]

The implementation is complete and this is ready for maintainer review. RFC/context: #764; related reliability failures: #488, OpenHands/OpenHands#14260, and OpenHands/OpenHands#13578.

@neubig, I would especially value your feedback on whether this benchmark core belongs here and how the first SWE-bench runtime adapter should plug into the shared evaluation runner. @simonrosenberg, I would value your review of the SDK event/dispatch boundary and the persist-before-inject contract.