[Leaderboard] Permute EQ - Claude Opus 5 - 91.33% Pass@1

[ericmillsio]

Permute EQ - Leaderboard Submission

Agent name: Permute EQ
Team: Permute (permute.ai)
Contributors: @Landbus and @ericmillsio
Backbone LLM: Claude Opus 5 (max reasoning effort)
Evidence refinement: Claude Sonnet 4.6
Hints: Yes
Tuned prompt: Yes
Trials: 5 per query (270 total)
Stratified Pass@1: 91.33%

Architecture

Permute EQ combines a novel model architecture for learned evidence construction over structured data with an independent analytical execution agent.

The model is trained over structured data systems using semi-supervised, invariant learning techniques. It learns reusable evidence about their semantics, structure, behavior, and relationships. Unlike conventional tabular predictive models, it does not directly predict a target outcome or answer. Instead, it constructs a general evidence layer designed to support downstream reasoning across many questions over the same data system.

For each question, the system selects a bounded, relevant view of the learned evidence. A single pre-execution Claude Sonnet 4.6 refinement pass organizes that evidence into structured context and an execution plan.

For every trial, a fresh Claude Opus 5 agent receives the same frozen evidence context for that question and access to the sanctioned task data through DuckDB. The agent independently queries the underlying records, verifies material assumptions and relationships, performs the requested analysis, and produces the final response.

The learned evidence guides execution but does not contain benchmark ground-truth answers, validator output, prior-run results, or other submissions. The evidence and execution instructions are fixed before the trials and are not updated using validator outcomes.

The execution agent has two tools:

• duckdb_query: runs analytical SQL over the sanctioned task data.
• calculator: performs deterministic numerical calculations.

No web, shell, document-search, external-fetch, or general-purpose filesystem tool is exposed. The benchmark source data remains read-only; the agent may create temporary in-memory DuckDB tables during analysis.

Given our training and testing split, we used two fixed prompts: one for the first 27 questions and one for the second 27. The prompts are otherwise identical; the second adds two question-specific formatting rules affecting presentation only, not evidence, analysis, or answer selection. No other question-specific tuning was used.

Results

Dataset	Pass@1
agnews	0.50
bookreview	1.00
crmarenapro	1.00
DEPS_DEV_V1	0.50
GITHUB_REPOS	1.00
googlelocal	1.00
music_brainz_20k	1.00
PANCANCER_ATLAS	1.00
PATENTS	1.00
stockindex	1.00
stockmarket	0.96
yelp	1.00
Stratified Pass@1	0.9133

Notes

• 270 entries: 54 queries × 5 complete trials.
• Individual trial scores were 51/54, 51/54, 51/54, 50/54, and 51/54.
• The aggregate raw result was 254/270.
• All five trials were complete benchmark runs performed under one frozen configuration; no partial-run or question-level splicing was used.
• Every trial used a fresh agent session with max reasoning effort and a 100-step limit.
• The learned evidence and execution instructions were fixed before the trials began.
• Validator results were produced only after each trial was completed and were never exposed to the agent.
• The learned evidence was reviewed and does not provide benchmark ground-truth answers, validator output, prior-run results, or other submissions.
• All answers were produced from the sanctioned benchmark data.
• Results were evaluated against the official per-query validators at benchmark commit 9a55e34271a2500a41cbc009e239edd94599e374.
• Submission answers match their corresponding final trace responses in all 270/270 trials.
• Trace audit found no attempts to access validate.py, ground-truth files, benchmark answer artifacts, prior-run results, or other submissions.
• Trace audit found no external dataset or web-access attempts.
• No general-purpose filesystem tool was exposed. Some stockmarket trials used DuckDB-native file functions to locate and inspect sanctioned Parquet fixtures, including one read of benchmark-provided hint text that was already present in the task input. Path enumeration surfaced some unrelated fixture filenames, but no unrelated file contents, ground-truth data, validators, prior results, or submissions were read or used.

Submission file: leaderboard_submissions/permute_eq.json
Traces: permute_eq_traces.zip