Put average rank first in the README leaderboard - #20
Closed
TonyBagnall wants to merge 12 commits into
Closed
Conversation
65 of the 66 Multiverse-core datasets, ingested with multiverse.experiments.ingest. TimesNet places 22nd of 24 on average accuracy rank, ahead of 1NN-DTW and Dummy. EigenWorms and Alzheimers are both present, which is worth recording: at 17984 and 15000 points they exceed the fixed 5000 position table TimesNet inherited from TSLib, and would have failed with a tensor size mismatch before that was sized to the series. The run also used d_model 32 and d_ff 64 with the type1 learning rate schedule, so it picked up the published classification defaults rather than the earlier values. EmoPain is missing. No TimesNet job logs were copied across, so the reason is recorded as inferred: aeon rejects EmoPain before fit for having 1733 case/channel pairs with std <= 1e-07, which stops every aeon classifier, and is the logged cause for ConvTran, PatchMTSC and DisjointCNN. It costs no datasets, since those three already exclude it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
PatchMTSC previously failed on PenDigits with
ValueError: patch_len exceeds the number of timepoints
because the series is 8 points long and the default patch length is 16. The
port clamps the patch geometry to the observed series instead of rejecting the
dataset, and a rerun with that code completes it at 0.9814. 65 of 66 datasets
now, EmoPain remaining.
PenDigits stays out of the comparison, since MRHydra and STSF are still missing
it, so the leaderboard is unchanged at 24 estimators on 52 datasets. MRHydra
fails it for the same underlying reason from the other direction: MultiRocket
requires at least 9 points.
Nothing else on disk has changed since the last ingest. The remaining
estimators with raw results are the ones deliberately left out: CIF-500 and
DrCIF-500, PULSAR pending publication, RSTSF and ProximityTree for the datasets
they cost, a duplicate STC run, and several with coverage too low to compare.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TSLib defaults to lradj type1, applied every five epochs. From
learning_rate 0.001 that reaches 2.0e-6 by epoch 10 and 6.1e-8 by epoch 15, so
training effectively stops a third of the way through a 30 epoch run. It was
reproduced here as a fidelity fix, which was the wrong call: it is harmless in
TSLib only because they select the retained epoch on the test set, keeping an
early epoch from before the collapse. This wrapper selects on a held-out split
of the training data, so it keeps a model that has stopped learning.
Measured on ERing, 30 training cases and 270 test, against a published 0.915:
as benchmarked (32/64/2, val 0.2) 0.356
bigger model (64/256/3, val 0.2) 0.578
no held-out split (32/64/2, val 0) 0.426
bigger + no split (64/256/3, val 0) 0.578
bigger, no split, no lr decay 0.933
The schedule is worth 35 points here, more than the model size and far more
than the validation split. Set lr_adjust="type1" to reproduce TSLib.
The ingested TimesNet results are removed rather than left in place, since they
were produced with the schedule on and understate the method: its ERing result
was 0.419. It will be rerun.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TS2Vec is the direct comparator to TimesURL: the survey notes they are the only two foundation-style methods with broad archive coverage that are comparable through available code, and both are in its clean 13. The authors' package is vendored under _ts2vec_original and driven by a thin wrapper, as TimesURL is. The only change to their code is the relative-import rewrite in ts2vec.py. A correction to an earlier assumption: TimesURL is a fork of TS2Vec, not a user of it, so its vendored components could not be reused. Its losses.py calls mixup variants TS2Vec does not have and its encoder.py carries a reconstruction head. The two packages are kept separate. The default probe is the authors' UEA protocol, an SVC chosen by grid search over C, since train.py passes eval_protocol='svm'. Their grid sets probability=False, which leaves an SVC unable to produce probability estimates, so it is enabled here because aeon classifiers must implement predict_proba; that adds Platt scaling fitted by internal cross-validation on training data only. probe="logistic" selects their linear alternative, which is the probe the TimesURL wrapper uses, so the two encoders can be compared without the probe differing between them. Equivalence is checked by comparing the vendored modules against upstream after canonicalising import style, and by asserting the wrapper drives the authors' class and uses their C grid. Set MULTIVERSE_TS2VEC_SRC to run those. On BasicMotions all three score 0.975; on ERing TS2Vec reaches 0.896 against TimesURL's 0.881, at a fifth of the runtime. Also adopts three improvements made to the tsml-eval copy of the TimesURL package: the shape print is now guarded by verbose rather than removed, imports are reformatted, and a missing trailing newline is added. The equivalence test that asserted the print was deleted now asserts it is guarded, which is the stronger check. 54 tests pass, aeon conformance 21/21 for all five ported classifiers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
XCM is in the survey's clean 13 and is the only member of that group with no aeon equivalent, so it needed a port rather than a run. Two parallel branches see the input differently: a 2D branch convolves along time within each channel separately, so its activations stay attributable to individual channels, and a 1D branch convolves across all channels together. That attributability is what the paper's explanations rest on, so the layer names it refers to are preserved and a test asserts they are still there. The network is one 97 line Keras function, so it is transcribed rather than vendored. Equivalence is checked by building both the authors' graph and ours and comparing layer types, shape-determining arguments, ordering, parameter counts and output shape. This is the first Keras port here, following the authors, so tensorflow joins torch in the deep-learning extra. Two changes were needed. The original imports Conv1D and Conv2D from keras.layers.convolutional, a path removed in Keras 3, so they come from tensorflow.keras.layers instead; layers and arguments are unchanged. And the kernel length int(window_size * n) is zero for series shorter than five points, which builds an invalid layer, so it is floored at one. Their training procedure is reproduced as is: a fixed 100 epochs on the whole training collection, no validation split, no epoch selection. Their main.py also runs a five fold cross validation but only to report per fold accuracies. On real data: BasicMotions 1.000 against a published 1.000, Epilepsy 0.949 against 0.993, NATOPS 0.911 against 0.978. The authors tune window_size per dataset, carrying a Win_pct column in their results table; NATOPS uses 40 where the default is 20, and at 0.4 we reach 0.933, so their window closes part but not all of that gap. 75 tests pass, aeon conformance 21/21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The 0.2 default was taken from the authors' config.yml, which is a worked example pinned to BasicMotions rather than a default. Their results table shows window_pct tuned per dataset over a five point grid: 20, 40 and 60 on five datasets each, 80 on thirteen, and 100 on two. 0.8 is the modal value, so it is the better single choice, on the same reasoning used for TimesNet's d_model and d_ff. Because window_size is a fraction the kernel grows with the series, and 0.8 of EigenWorms' 17984 points is a 14387 point kernel. max_window bounds it at 100, which changes nothing for series under 125 points and keeps the long Multiverse problems tractable. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
65 of the 66 Multiverse-core datasets. TimesURL places 22nd of 24 on average accuracy rank, mean accuracy 0.695. EmoPain is missing, the usual case: aeon rejects it before fit for having 1733 case/channel pairs with std <= 1e-07, which stops every aeon classifier. No TimesURL job logs were copied across, so the reason is recorded as inferred. It costs no datasets, since ConvTran, PatchMTSC and DisjointCNN already exclude it. Note the run used the tsml-eval implementation, whose parameters have moved ahead of the copy in this repository: batch_size 8 rather than 16, learning_rate 1e-4 rather than 1e-3, an eval_protocol option, standardisation, max_train_length and several others. The results here therefore describe that version, not multiverse.classification.TimesURLClassifier. Reconciling the two is a separate change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The TimesURL results were produced by the tsml-eval version, whose parameters and behaviour had moved ahead of the copy here: batch_size 8 rather than 16, learning_rate 1e-4 rather than 1e-3, plus an eval_protocol option, per-channel standardisation, max_train_length, temporal_unit, temperature and sgd. The ingested results therefore described an implementation this repository did not have. This adopts that version so the two agree. Its probe handling is also better than what it replaces: rather than forcing probability=True on the SVC, which adds Platt scaling and an internal cross validation, it softmaxes the decision scores when the probe has no predict_proba. Kept from the previous version: the module docstring recording provenance, the MIT licence and attribution, and the three changes made to the vendored package; the Attributes, References and Examples sections; and shape validation on predict, without which a mismatched collection fails somewhere inside the encoder with an error that does not say what was wrong. 75 tests pass, aeon conformance 21/21. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The directory holds accuracies reported in the MTSC literature, one file per surveyed paper, which is what it is used for here: comparing our runs against what was published. "survey" named where the data came from rather than what it is, and read as though it belonged to one paper. All 129 files moved with git mv so history follows. Nothing in the package or the documentation referenced the path except the repository layout in the README, which is updated. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The directory had per-category notes recording which table each file came from, but nothing describing what the collection is or how to read it. The README adds that, along with the four things that have caught us out while using it: - value scale is not consistent, percentages in some files and fractions in others, so 0.13 is ambiguous without checking the file - 13 of the 118 files are not wide, leading with Table, Scope, Method and others, because the paper's table did not fit that shape - files carry summary rows such as Average Rank, Wins and p-val, which parse as datasets with implausible scores - repeated values across papers are transcribed baselines, not independent runs, so the modal value is one measurement quoted many times It also notes the two kinds of file that are not extracted from papers, the dataset name variations, and that published numbers are not always produced under a comparable protocol. Links the survey from both READMEs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Renaming survey/ to published_results/ broke links to github.com/aeon-toolkit/multiverse/tree/main/survey, and GitHub does not redirect directory URLs. A README at the old path is what a browser lands on, so it points at the new location and shows how to rewrite a link to a specific file. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The rows are ordered by average accuracy rank, but that column sat at the far right, so the table read as though it were sorted on accuracy, the leftmost metric, and mis-sorted: HC2 above MRHydra is right on rank, and CIF above QUANT is not what the accuracy column alone would give. Moving the ordering column to the front makes the ordering legible. Column order only. No value changes, and the HTML page keeps its score-and-rank pair per metric, where the ordering is already visible from the rank columns. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TonyBagnall
force-pushed
the
leaderboard-link
branch
from
September 1, 2026 09:24
1df917e to
04fe44c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The rows are ordered by average accuracy rank, but that column sat at the far right, so the table read as though it were sorted on accuracy, the leftmost metric, and mis-sorted: HC2 above MRHydra is right on rank, and CIF above QUANT is not what the accuracy column alone would give. Moving the ordering column to the front makes the ordering legible.
Column order only. No value changes, and the HTML page keeps its score-and-rank pair per metric, where the ordering is already visible from the rank columns.
Claude-Session: https://claude.ai/code/session_01LyaaJWNBSejmDY7XRQKok3
Summary
Checklist
results/schema.md.