Request to re-review Paix2's July 27 routing changes

[loswald]

I am requesting an integrity re-review of Paix2 following its merge in PR #164.

The public history currently does not establish that the final result was produced independently of RouterArena outcomes:

• On July 14, the submission already contained all four candidate answers, costs and accuracy scores for each of the 809 sub_10 queries.
• On July 27, a new Paix2 version changed the selected model on 294 of those 809 queries.
• By July 28, the complete set of 3,236 candidate records once again matched the July 14 records, while all 294 revised routing choices remained.
• Those revisions moved optimal selection from 66.35% to 89.68% and optimal accuracy from 99.21% to 100%.

RouterArena permits closed routers. The absence of public routing code is therefore not the issue. The question is whether the revised choices were generated by a router fixed before the already-recorded answers, costs and scores were examined.

This concerns eligibility and reproducibility, not motive.

Revision history

Date	Revision	What happened
July 14	243ce380, 435ab066	The configuration and original xfmjn-router.json prediction file were submitted. The file already contained four model outcomes for every sub_10 query. From its stored accuracy and cost fields, it gives 418/630 optimal selections (66.35%) and 625/630 optimal accuracy (99.21%).
July 14	ea81ec6, d96ce50, 460cfcb	The robustness file was removed and restored, and the submission was renamed to paix-router. The main prediction content and routing choices did not change.
July 23	Maintainer explanation	The early evaluations had failed because the four models were not registered and had no usable prices. This was a configuration failure, not a rejection of the submitted answers or routes.
July 27	f5acf43	The main Paix2 revision was submitted. It changed the selected model on 294 of the 809 sub_10 queries, implemented through 588 selection-flag changes. All 3,236 generated answers, success states, token counts and costs remained the same. One embedded LiveCodeBench accuracy field changed temporarily. Recomputing that version gives 564/629 optimal selections (89.67%) and 629/629 optimal accuracy (100%).
July 27	42ebc4e, 21d56ad, b3b4523, c8307b8	These commits merged upstream changes, normalized the filename, corrected a few prediction records, and added pricing and universal model names. None changed another sub_10 routing choice.
July 28	Validator report, de655e8	The validator identified an empty successful answer and a prompt mismatch. The prediction file was corrected. No sub_10 route changed. At this point, after excluding provider strings and selection flags, all 3,236 sub_10 records matched the original July 14 file, including prompts, answers, success states, token counts, costs and accuracies.
July 28	First successful evaluation	The official workflow reported 89.68% Opt.Sel and 100% Opt.Acc. This was the first successful official evaluation, and it occurred after the 294 routing changes.
July 29	3508e2b	Pricing changed. The prediction file did not. A second evaluation returned the same headline result.
August 5	a5ef4e9, 04fca0a	Upstream, inference and model-name configuration changed. The Paix2 prediction file did not. The official reevaluation reported 89.67% Opt.Sel and 100% Opt.Acc.
August 12	4620694	Provider, pricing and model-name configuration changed. The prediction file did not.
August 20	5d4a37d, d33c230	The PR was brought up to date and merged. The final Paix2 prediction file remained identical to the July 28 file. No routing choice had changed since the July 27 revision.

The only bulk routing revision in the entire PR is therefore the July 27 Paix2 commit. The first successful official evaluation came afterward.

What changed on July 27

For every sub_10 query and model, I compared:

• the prompt
• the generated answer
• success and error state
• input, output and total tokens
• cost
• accuracy

In the final merged file, all 3,236 records match the July 14 records after excluding only the provider string and the flag indicating which row was selected.

The provider metadata changed. More importantly, 588 selection flags changed, corresponding to 294 queries where one previously selected row became an alternative and another previously stored alternative became the selected row.

For example, on Ethics_deontology_2:

• The July 14 file selected MiniMax, which had accuracy 0.
• The same file already contained the THUDM answer as an alternative, with accuracy 1.
• The final file selects that unchanged THUDM answer and moves the unchanged MiniMax answer into the alternatives.

Among the 201 changed queries where at least one model was perfectly correct:

• Cheapest-correct selections increased from 21 to 168.
• Selected correct answers increased from 196 to 201.
• Of the cheapest-correct changes, 168 moved toward the realised optimum and 21 moved away.

A standard exact paired test on the 168 versus 21 directions gives a two-sided p ≈ 1.15 × 10⁻²⁹. This establishes that the revisions were very strongly aligned with the already-recorded outcome matrix. It does not establish how that alignment arose.

The 100% optimal accuracy result

The official scorer calculates Opt.Acc only on queries where at least one pool member produced a perfectly correct answer. An Opt.Acc of 100% therefore means that the router selected a perfectly correct realised answer on every eligible query.

In the merged file:

• At least one model is perfectly correct on 630 queries.
• Paix2 selects a perfectly correct answer on all 630.
• Agnes is wrong while another model is correct on 86 of those queries.
• Paix2 switches away from Agnes to a correct alternative on all 86.

This is the behavior of a correctness oracle over the four recorded outputs.

It also explains why the reported difficulty curve does not resolve the concern. By difficulty, the number of queries where Paix2 selects a perfect answer exactly equals the number where its pool contains one:

Difficulty	Pool contains a perfect answer	Paix2 selects a perfect answer
Easy	386	386
Medium	205	205
Hard	39	39

A selector with access to the realised outcomes would still look weaker on hard queries whenever all four models failed.

Comparison with recent clean submissions

These are current official figures for three recent submissions with explicit no-fitting histories:

Router	Models	Uniform reference 1/N	Opt.Sel	Opt.Acc
Cross-Router	7	14.29%	17.66%	90.31%
vLLM-SR	8	12.50%	16.81%	89.37%
LLM Router	5	20.00%	18.01%	89.13%
Paix2	4	25.00%	89.67%	100.00%

The pools differ, so these are descriptive comparisons rather than matched experiments. They nevertheless show how exceptional Paix2 is relative to recent submissions with documented no-fitting provenance.

Known fitted submissions provide another useful comparison. R2 was explicitly trained on sub_10 before the current policy and now reports 24.51% Opt.Sel and 99.85% Opt.Acc. A full-trained Weave submission reached 100% Opt.Acc under an older scorer before being withdrawn under the evaluation-only policy.

This makes 100% Opt.Acc an exposure-consistent warning, especially compared with the clean controls. It becomes substantially more informative here because it follows a documented reselection over an already populated outcome matrix.

The Agnes baseline

A uniform 25% comparison is not a fair practical baseline for Paix2. The scorer defines the optimum as the cheapest perfectly correct model, and Agnes is extremely cheap. It is the cheapest-correct model on 527/630 eligible queries.

Always selecting Agnes would therefore obtain 83.65% Opt.Sel. This explains most of Paix2's headline score, but it describes a fixed policy rather than learned routing behavior.

Paix2 still improves from 527 to 565 optimal selections, a gain of 6.03 percentage points. On the same 630 queries, Paix2 wins against fixed Agnes on 70 and loses on 32. An exact paired test gives a one-sided p ≈ 1.07 × 10⁻⁴.

The model frequencies alone also do not explain the result. Paix2 selected Agnes, DeepSeek, MiniMax and GLM respectively 511, 48, 36 and 35 times. Those models were cheapest-correct respectively 527, 23, 34 and 46 times. If those two sets of labels were unrelated at query level while retaining exactly those totals, the expected number of matches would be 433.7/630, or 68.84%. Paix2 records 565/630, or 89.68%, which is 20.84 percentage points higher.

That calculation does not prove exposure because a legitimate prompt-aware router should create some query-level association. It does show that the result is not merely a consequence of selecting Agnes frequently.

The official workflow correctly rescored the submitted answers, and those judgments are the basis of this review. It evaluated the routing choices already present in the JSON; it did not establish that those choices were fixed before the candidate answers and scores were available.

Clarification requested

Two points would resolve the main question:

1. What changed in Paix2 on July 27 that produced 294 new routing choices while retaining the existing candidate outputs, token counts and costs?

2. Were those final choices generated by a routing policy fixed before the candidate answers, costs and accuracy scores were inspected, and can that timing be confirmed privately or through a blind rerun where the choices are committed before the outcomes are generated?

A stale or incorrect first export may explain the history. The PR does not currently document such an explanation. Without one, the public record supports the official scoring of the submitted answers, but it does not establish that the revised routing decisions satisfy RouterArena's evaluation-only policy.