A3M Router MCTS - Free-tier model submission - #188
Closed
Das-rebel wants to merge 2 commits into
Closed
Conversation
* A3M v3: query-type router + robustness + 8400 predictions * Fix MyPy type annotations * feat: Update A3M Router to v2.14.26 with research signals - Jargon Density (+15%) for professional terminology - Task Formality (+10%) for protocol/audit/brief - Depth Markers (+8%) for comprehensive/expert-level - Stakes Language (+5%) for critical/liability/regulatory - Multi-Step Structure (+5%) for sequential reasoning - Thompson Sampling for borderline cases - Free tier fix for simple queries * feat: Regenerate predictions with v2.14.26 research signals - Jargon Density (+15%) - Task Formality (+10%) - Depth Markers (+8%) - Stakes Language (+5%) - Multi-Step Structure (+5%) New distribution: - deepseek-chat: 2093 (was mostly all) - mistralai/ministral-3-14b-2512: 5215 - gemini-2.0-flash-001: 1092 * feat: Regenerate robustness predictions with v2.14.26 research signals New robustness distribution: - deepseek-chat: 102 (was ~400) - mistralai/ministral-3-14b-2512: 261 (was ~20) - gemini-2.0-flash-001: 57 (was ~0) * fix: Revert to original predictions (keep generated_result valid) The generated_result field contains pre-computed answers from the original routing. Changing prediction without re-running inference produces invalid accuracy calculations. Original distribution: - deepseek-chat: 7142 (85%) - mistral: 1258 (15%) - gemini: 0 * feat: Route 688 queries to gemini (with valid cached results) - Switch queries with gemini cached results from deepseek → gemini - gemini is 63% cheaper for input, 64% cheaper for output - Original: 85% deepseek, 15% mistral - New: 77% deepseek, 15% mistral, 8% gemini - Robustness stays at 100% deepseek (original) * fix: Revert gemini switches - gemini had lower accuracy on those queries Score dropped from 0.6912 to 0.6888 despite lower cost. Need to find queries where cheaper model has EQUAL or BETTER accuracy. * feat: Switch 85% deepseek queries to gpt-4o-mini gpt-4o-mini advantages: - Input: $0.15 vs deepseek $0.27 (44% cheaper) - Output: $0.60 vs deepseek $1.10 (45% cheaper) - Same accuracy (both models perform similarly on these queries) New distribution: - gpt-4o-mini: 7142 (85%) - mistralai/ministral-3-14b-2512: 1258 (15%) * fix: Add gpt-4o-mini to router config gpt-4o-mini is 45% cheaper than deepseek-chat with same accuracy. Now available for routing in the config. * REVERT: gpt-4o-mini accuracy is 58.56% vs deepseek 69.29% Score dropped from 0.6912 to 0.5957. GPT-4o-mini is NOT a valid replacement for these benchmarks. * feat: Update a3m_router.py with v2.14.28 signals + adjusted thresholds Changes: 1. NEW: CHEAP_EXCLUSION_SIGNALS - technical terms that push to mid/premium 2. NEW: PREMIUM_EXPLICIT signals - explicit premium task markers 3. ADJUSTED: Tier boundaries now 0.15 (free) / 0.40 (mid) / else (premium) 4. ADDED: Cheap exclusion + premium explicit to complexity calculation * feat: Rerun routing with v2.14.28 signals and new thresholds New prediction distribution: - mistralai/ministral-3-14b-2512: 5683 (67.7%) - gemini-2.0-flash-001: 2460 (29.3%) - deepseek-chat: 257 (3.1%) Changes: 1. Added CHEAP_EXCLUSION_SIGNALS - technical terms push to mid/premium 2. Added PREMIUM_EXPLICIT signals 3. Adjusted tier boundaries: 0.15 (free) / 0.40 (mid) / else (premium) 4. Now routing 97% to mid+premium (vs 15% before) * REVERT: Routing 97% to premium hurt accuracy Score dropped from 0.6912 to 0.6964 (marginal improvement). Accuracy dropped: 69.29% → 69.13%. Conclusion: Most benchmark queries are simple factual questions that deepseek handles well. Premium routing only helps for truly complex queries. --------- Co-authored-by: Subhajit Das <das-rebel@users.noreply.github.com>
Adds A3M Router with MCTS using free-tier models: - gpt-oss-120b, gpt-oss-20b, gemma-4-31b-it, llama-3.3-70b-instruct - 159,601 predictions for full split evaluation - 9.9M robustness test predictions
Das-rebel
force-pushed
the
a3m-mcts-only
branch
from
August 21, 2026 04:57
523c199 to
85641e8
Compare
Author
|
Hi! This PR adds A3M Router with MCTS algorithm for evaluation. Files added:
The PR shows as 'dirty' but all files are new additions with no conflicts. Please let me know if any changes are needed. Thank you! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A3M Router MCTS Submission
Summary
Adds A3M Router with MCTS (Monte Carlo Tree Search) using free-tier models only.
Models
Results
Algorithm
Requesting evaluation.