refactor(claw-eval): use the standard <task>/task.json layout

[vaibhavdabas16]

Closes #292. Rewritten to take @Perry2004's suggested approach — this PR no longer touches the workflow at all. See the review reply below for what changed and why.

The problem

test-cases/claw-eval/ shipped as flat <task-identifier>.json files while every native corpus uses <task-identifier>/task.json. validate-task selects on task.json in four separate places:

Location	Rule
paths: trigger (PR + push)	test-cases/**/task.json
workflow_dispatch fallback	find test-cases -path "*/task.json"
schema-changed glob	repo.glob("test-cases/**/task.json")
per-changed-path test	path.name == "task.json"

None of them ever saw the suite. Nineteen tasks could be added, edited or broken without the job noticing.

The fix

Move the 19 tasks into directories. The workflow then covers them with no change to the workflow — every rule above already does the right thing once the layout matches.

Discovery collapses back to a plain */task.json search:

• batch._all_cases_in drops _flat_case_files entirely
• batch.discover_cases drops its "…or a bare .json file" pattern branch
• tui.load_cases drops its parallel flat glob
• test_host_tasks stops assembling task files from two globs, and can now assert every discovered case is a directory containing task.json

The eligibility-report.json exclusion goes with them — it existed only to keep a non-task file out of the flat glob.

Net: 9 insertions, 30 deletions outside the file moves.

Verification

Ran the workflow's own selection logic, unmodified, against the new tree:

task files selected: 320  of which claw-eval: 19
errors: 0 in claw-eval

301 before, 320 after. All 19 validate cleanly against task.schema.json, including the extra_info path-existence check — so this turns the job on for the suite without turning it red, which is worth confirming separately: a trigger fix that surfaced 19 broken tasks would be a different PR.

213 passed, 4 skipped — unchanged from main. (test_host_tasks.py::…[v1-lite] fails on main too on Windows: v1-lite is symlinks, which check out as text files. Same cause as the 20 v1-lite JSON errors in the local validation run above. Unrelated to this change.)

Also checked by hand: clawbench-batch --cases-dir test-cases/claw-eval --all-cases discovers all 19, and pattern matching (--cases 'ce-T046*') still resolves.

One correction to the issue text

#292 states claw-eval was entirely unchecked. Not quite: tests/test_host_tasks.py globbed the flat files and called validate_task_data(), which checks four fields. What never reached the suite was the full task.schema.json validation and the extra_info path-existence check. That is the real gap, and it is what this closes.

Supersedes #337

#337 taught the CI script to validate flat suites. With no flat suites left, there is nothing for it to teach — closing it.

[Perry2004]

Thanks for the contribution. Though I'd suggest an alternative approach:

• Instead of changing the workflow it's better to use this chance to re-organize the claw-eval port's structure. It should go with the standard test-cases/claw-eval/<task-identifier>/task.json structure as native ClawBench tasks.
• After the structure change we can then simplify the TUI and batch discovery logic to a standard task.json search.
• I believe there's no need for extracting the json schema validation logic to a separate script and add tests on it given the simplicity of the logic.

[vaibhavdabas16]

Thanks — took all three points; the PR is rewritten rather than patched, so it is worth reading fresh.

Instead of changing the workflow it's better to use this chance to re-organize the claw-eval port's structure.

Agreed, and it is the better fix for a reason I had missed: with the layout corrected, the workflow needs no change at all. All four of its selection rules — the paths: trigger, the workflow_dispatch find, the schema-changed glob, and the per-changed-path test — are already task.json-shaped and already correct. The bug was never in the workflow; it was that one suite did not look like the others. .github/workflows/validate-task.yml is untouched in the new diff.

After the structure change we can then simplify the TUI and batch discovery logic to a standard task.json search.

Done, and it removed more than I expected:

• batch._flat_case_files — deleted
• batch.discover_cases — the elif d.is_file() and d.suffix == ".json" pattern branch, gone
• tui.load_cases — its parallel flat glob, gone
• test_host_tasks._task_files_for_suite — was unioning two globs, now one
• test_builtin_case_suites_are_discoverable — had an if case.is_dir() / else fork to accommodate both shapes; now asserts every case is a directory with a task.json, which is a stronger claim than it could previously make

The eligibility-report.json exclusion went too — it existed only to keep a non-task file out of the flat glob.

9 insertions, 30 deletions outside the file moves.

I believe there's no need for extracting the json schema validation logic to a separate script and add tests on it given the simplicity of the logic.

Dropped — scripts/ci/validate_tasks.py and tests/test_validate_tasks_script.py are not in this diff. My reasoning for extracting it was that four copies of the same wrong rule had survived inside an untestable YAML heredoc, but that is moot now: the rules were not wrong, and the reason they looked wrong is fixed at the source. The inline heredoc stays as it is.

Verification. Ran the workflow's own selection logic, unmodified, against the restructured tree: it selects 320 task files where it previously selected 301, the 19 new ones being claw-eval, and all 19 validate cleanly against task.schema.json including the extra_info path-existence check. So this turns the job on for the suite without turning it red — worth confirming separately, since a trigger fix that surfaced 19 broken tasks would be a different conversation.

Test suite unchanged from main at 213 passed, 4 skipped. Also checked by hand that --all-cases discovers all 19 and --cases 'ce-T046*' still resolves.

I will close #337 as superseded — it taught the CI script to validate flat suites, and there are none left.

[vaibhavdabas16]

CI green, and this run is the demonstration the previous version of this PR could not give.

validate-task fired on the unmodified workflow and reported:

Validated 19 task file(s) and 19 changed JSON file(s).

All 19 are claw-eval. Before this change the same job on the same suite selected zero — the run on the old branch reported Validated 0 task file(s), correctly, because nothing there was named task.json. So the suite is now covered by the real job on the real runner, not just by a local reproduction of its selection logic.

_{Run: https://github.com/TIGER-AI-Lab/ClawBench/actions/runs/33666380236}

[Perry2004]

LGTM🫡