Repository navigation
[MRG] Clarify MixVal AccAvg scoring versus cross-candidate rank aggregation - #378
Open
AHMETHAKANBEZIR1 wants to merge 1 commit into
Open
AHMETHAKANBEZIR1 wants to merge 1 commit into
AHMETHAKANBEZIR1 wants to merge 1 commit into
Conversation
Co-authored-by: OpenAI Codex <codex@openai.com>
This was referenced Oct 22, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
MixValScorer(ice_type="both") averages the available ICE score values for one model. With accuracy scoring, that is the AccAvg ablation, whereas the published MixVal procedure ranks all candidate models separately on intra/inter ICE and then averages their ranks. A single-estimator scorer cannot perform that cross-candidate aggregation by itself.
Clarify this distinction in the parameter description and Notes, describe how to collect separate intra/inter scores with a shared random_state, and include an executable four-candidate example: rank aggregation selects index2 while accuracy averaging selects index1. The example uses finite, untied scores to avoid implying an unapproved tie/missing-score policy. No scoring logic, default, public API or helper function is changed.
Fixes #374 via the documentation option in the report. The source was checked against Section3, Algorithm1 and Section4.3 of Hu et al. (2023): https://proceedings.neurips.cc/paper_files/paper/2023/file/7721f1fea280e9ffae528dc78c732576-Paper-Conference.pdf . Thanks to VictorBauler for identifying the mismatch.
Validation: metrics doctest plus unchanged scorer tests32passed (including the new executable example); all seven full-repository pre-commit hooks passed, normal commit hooks retained, diff check passed. This documentation-only change does not claim a runtime bug fix, new model benchmark or improved accuracy. Paper reproduction/model-selection policy beyond the documented distinction remains for users and maintainers to choose.
AI disclosure: the account owner requested OSS work. OpenAI Codex checked the sources, wrote this documentation/example and ran the checks; no independent human code review has occurred.
Integration check: these three independent PR heads (#376/#377/#378) were applied together to a private worktree based on main 31f2ff8. The official base-dependency Minimal CI profile passed with 273 passed, 21 skipped, 98 deselected and 4 xfailed. The combined scorer/subspace tests and metrics doctest also passed in the separate Torch CPU environment (72 passed). This is a local compatibility check, not an upstream merge or remote-CI result. GitHub Testing workflows currently require maintainer approval.