You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Parent #331 proposes a documented no-op oracle because ClawBench uses live websites. Harbor's current adapter contribution guide requires solvability/oracle validation and parity with the same agents, models, and settings; it recommends discussing difficult tasks case by case. Our solve_script() only prints a message and exits successfully; it provides no intercepted request or successful judged outcome. A shell exit of 0 is not oracle evidence.
Obtain and record an upstream-agreed approach for live-site solvability evidence (real reference trajectories or an explicit exception); do not label the no-op script an oracle pass.
Decide whether upstream adapter submission uses the conventional root layout or accepts our supported steps/run layout. Harbor's loader accepts steps; contribution acceptance is a separate question.
Specify a representative task subset, exact corpus/agent/model/judge revisions, browser/tool configuration, timeouts, retries, run count, and rate limits before experiments.
Report Stage 1, lenient/strict Stage 2, judge-inconclusive and infrastructure counts separately, with denominators and mean ± sample SEM across independent runs.
Produce actual adapter_metadata.json, parity_experiment.json, reference YAML, and reproducible evidence before claiming official integration.
This issue narrows #331 §4 rather than reopening the overall integration request. Honor Perry2004's decision to defer image publication and CI; neither is made a prerequisite here. No parity results have been fabricated or experiments launched by this issue.
@Perry2004 please review the live-site oracle approach and parity protocol and coordinate the upstream acceptance decision.
Decision needed for upstream acceptance
Parent #331 proposes a documented no-op oracle because ClawBench uses live websites. Harbor's current adapter contribution guide requires solvability/oracle validation and parity with the same agents, models, and settings; it recommends discussing difficult tasks case by case. Our
solve_script()only prints a message and exits successfully; it provides no intercepted request or successful judged outcome. A shell exit of 0 is not oracle evidence.Source: https://github.com/harbor-framework/harbor/blob/fd0049126a060f930372f9b0b0aa330caf1f19e5/docs/content/docs/datasets/adapters.mdx (oracle validation, parity planning, structured reporting).
Acceptance criteria
steps/runlayout. Harbor's loader accepts steps; contribution acceptance is a separate question.adapter_metadata.json,parity_experiment.json, reference YAML, and reproducible evidence before claiming official integration.This issue narrows #331 §4 rather than reopening the overall integration request. Honor Perry2004's decision to defer image publication and CI; neither is made a prerequisite here. No parity results have been fabricated or experiments launched by this issue.
@Perry2004 please review the live-site oracle approach and parity protocol and coordinate the upstream acceptance decision.