Skip to content

Feature/multi dataset support - #85

Open
Henry-Shan wants to merge 7 commits into
RouteWorks:mainfrom
Henry-Shan:Feature/multi-dataset-support
Open

Feature/multi dataset support#85
Henry-Shan wants to merge 7 commits into
RouteWorks:mainfrom
Henry-Shan:Feature/multi-dataset-support

Conversation

@Henry-Shan

Copy link
Copy Markdown
Collaborator

Integrate ARC, HellaSwag, WinoGrande, TriviaQA, and HLE datasets

Add support for 5 new benchmark datasets into the RouterArena evaluation pipeline.

Key changes:

Add evaluation configs (config/eval_config/zero-shot/):

  • ARC-Challenge.json, ARC-Easy.json (mcq_accuracy)
  • HellaSwag.json (mcq_accuracy)
  • WinoGrande.json (mcq_accuracy)
  • TriviaQA.json (triviaqa_exact_match)
  • HLE.json (exact_match)

Create data preparation scripts (scripts/):

  • prepare_arc_data.py: Load ARC-Challenge/Easy from HuggingFace,
    normalize numeric answer keys (1/2/3/4 → A/B/C/D), combine all splits
  • prepare_hellaswag_data.py: Load HellaSwag, convert label indices to
    letters, skip unlabeled test split
  • prepare_winogrande_data.py: Load all 6 WinoGrande subsets, handle
    binary choice format
  • prepare_triviaqa_data.py: Load all 8 TriviaQA subsets, save full
    alias lists for flexible answer matching
  • prepare_hle_data.py: Load HLE test split, filter image-based questions,
    handle both exactMatch and multipleChoice answer types

Update evaluation pipeline (llm_evaluation/run.py, evaluate_models.py):

  • Add split handling in load_ground_truth_dataset() for all 5 datasets
  • Add dataset name detection in evaluate_models.py
  • Add triviaqa_exact_match metric (alias list matching) to metrics.py
  • Register triviaqa_exact_match in eval_reasoning.py

Update inference pipeline (router_inference/generate_prediction_file.py,
llm_inference/run.py):

  • Add new splits to DATASET_PATHS and filename routing
  • Add split support for router predictions and argparse choices

@yl231

yl231 commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Apologies for the long silence, @Henry-Shan — maintainer triage. This is a sizable change (~992 lines across 15 files) for multi-dataset support and predates several months of pipeline evolution, so it no longer applies cleanly to main. Before investing in a full review: is this still active, and is it aimed at the current RouterArena evaluation flow? If yes, a rebase onto the latest main would let us review it properly; if it's been superseded, let us know and we'll close it. Thanks for the contribution either way.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants