Leaderboard link - #27
Merged
Merged
Conversation
The last of the survey's thirteen trustworthy papers that had no implementation here, and the one it singles out: "the only paper with no obvious problems in the exposed entry-point code that also provides usable results", reporting over five seeds. RankSCL is supervised contrastive learning with two departures. Positives are augmented in the embedding space rather than the input space, by jittering the representation of a randomly drawn same-class neighbour, and the loss is rank-based: for each positive it counts, through a sigmoid, the negatives that sit at least as close to the anchor, sums them, and compresses with arctan so a badly ranked positive saturates instead of dominating the batch. An SVM on the encoder representations does the classifying, the TS2Vec protocol the authors adopt, including its MAX_SAMPLES cap. Transcribed rather than vendored. The authors' modules import each other absolutely and pull in matplotlib and a logger writing to a hard-coded path, while the parts that matter are small: the FCN encoder, the augmentation and the loss. The loss is checked against a hand computation on four points on a line rather than only for running, since a transcription of it could be quietly wrong and still train. Two behaviours are ours: a single-class batch has nothing to rank and returns no loss where the original raises on an empty stack, and a collection smaller than batch_size is refused, since the original drops the last incomplete batch and would train on nothing. Defaults are the authors' UEA settings from scripts/uea.sh. 21/21 aeon checks, 14 tests. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The UEA archive is the 30-dataset collection almost every published multivariate result is quoted on, so a table restricted to it is what a reader comparing against the literature needs. It is a subset view of the same runs behind the Multiverse-core leaderboard, not a separate experiment: the same function with a different dataset list. Coverage is stated rather than left to be inferred, because a partial-coverage table is precisely what the 2026 MTSC survey criticises in the literature. Four of the thirty are not in Multiverse-core and have no results at all here, BasicMotions, FingerMovements, InsectWingbeat and SelfRegulationSCP2, which the page names. Of the twenty-six that remain, twenty-three have a result for every estimator and those are what the ranking uses. leaderboard_markdown gains a collection parameter. Its caption was hard-coded to "Multiverse-core", so the UEA table would have claimed to average over Multiverse-core datasets while covering 23 UEA ones. HC2 leads on this subset too, though the order below it differs from Multiverse-core: RDST and MRHydra come second and third here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
65 of the 66 core datasets at resample 0; EmoPain is the only gap, and aeon rejects it before fit for every classifier on its variance check. The port enters the leaderboard at rank 13.63, between LiteTIME and ConvTran. Against aeon's estimator on the same 65 datasets it is +0.126 mean accuracy, 50 wins to 13 with 2 ties, Wilcoxon p = 3.3e-07, and on the 23 UEA datasets we hold, closest to the paper's own evaluation set, +0.170 at 20/2/1. ERing, the dataset the bug was traced on, goes 0.589 to 0.956 against 0.964 published. aeon's results stay under DisjointCNN-Aeon and stay excluded from the table. The common set is unchanged at 51, since EmoPain was already outside it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The only conflict was docs/leaderboard.md. main's docs pass trimmed the file to the "generating a leaderboard" how-to and moved the archive descriptions into docs/datasets.md, while this branch had added a UEA section to the same region. Resolved to main's structure plus the two things only this branch has: the UEA section, which holds the UEA_LEADERBOARD markers that tables.py writes into and so cannot be dropped without breaking the build, and the max_cd_estimators warning, which main's text still refers to as "see the warning below" but no longer contained. The empty Multiverse, Multiverse-core, EEG archive and UEA archive stubs are gone, as main intended. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Checklist
results/schema.md.