Skip to content

Harbor upstream: agree live-site oracle evidence and parity acceptance protocol #359

Description

@reacher-z

Decision needed for upstream acceptance

Parent #331 proposes a documented no-op oracle because ClawBench uses live websites. Harbor's current adapter contribution guide requires solvability/oracle validation and parity with the same agents, models, and settings; it recommends discussing difficult tasks case by case. Our solve_script() only prints a message and exits successfully; it provides no intercepted request or successful judged outcome. A shell exit of 0 is not oracle evidence.

Source: https://github.com/harbor-framework/harbor/blob/fd0049126a060f930372f9b0b0aa330caf1f19e5/docs/content/docs/datasets/adapters.mdx (oracle validation, parity planning, structured reporting).

Acceptance criteria

  • Obtain and record an upstream-agreed approach for live-site solvability evidence (real reference trajectories or an explicit exception); do not label the no-op script an oracle pass.
  • Decide whether upstream adapter submission uses the conventional root layout or accepts our supported steps/run layout. Harbor's loader accepts steps; contribution acceptance is a separate question.
  • Specify a representative task subset, exact corpus/agent/model/judge revisions, browser/tool configuration, timeouts, retries, run count, and rate limits before experiments.
  • Report Stage 1, lenient/strict Stage 2, judge-inconclusive and infrastructure counts separately, with denominators and mean ± sample SEM across independent runs.
  • Produce actual adapter_metadata.json, parity_experiment.json, reference YAML, and reproducible evidence before claiming official integration.
  • Keep upstream docs-only PR docs: reference ClawBench as an external benchmark harbor-framework/harbor#2495 distinct from an adapter or registered dataset submission.

This issue narrows #331 §4 rather than reopening the overall integration request. Honor Perry2004's decision to defer image publication and CI; neither is made a prerequisite here. No parity results have been fabricated or experiments launched by this issue.

@Perry2004 please review the live-site oracle approach and parity protocol and coordinate the upstream acceptance decision.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions