Leaderboard link - #28
Merged
Merged
Conversation
LiteTIME is the univariate LITE architecture; the multivariate variant of the same method already reports as LITETime-MV. FreshPRINCE and 1NN-DTW cannot finish the archive on the resources available, and because scoring uses the datasets every estimator completed, one estimator's gaps are subtracted from everyone's table. Removing the three returns five datasets to the scored set: BIDMC32HR_disc, BIDMC32SpO2_disc, FaceDetection, FordChallenge and Skoda. The Multiverse-core leaderboard goes from 27 estimators on 51 datasets to 24 on 56. The four withheld estimators are now named on every page under "Estimators not listed", each with its reason, rather than being absent without explanation. The exclusion list and those reasons are one structure, WITHHELD_ESTIMATORS, so the page cannot drift from what main() actually excludes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eight of RankSCL's core datasets hit TIMEOUT at the 60 hour wall, and the cause was not dataset size. LSST is 2459 training cases of length 36 and reached "Epoch 91 ----- loss 0.239" before Slurm killed it, which at batch size 4 is 614 steps an epoch and works out at about 3.5 seconds per optimiser step for a small encoder on an H200. The loop was the cost. At the archive settings a batch holds batch_size * (2 * aug_positives + 1) = 44 embeddings with every case repeated eleven times, so each anchor has around ten positives and each step ran roughly 440 Python iterations, every one launching several small CUDA kernels and adding autograd nodes. Kernel launch latency decided the runtime while the GPU sat idle. The same quantity is one dense expression over an (anchor, positive, other) difference tensor, 85k elements at the defaults. Checked against the loop on 500 random collections, values and the None cases agree exactly, and two new tests pin both the value and the gradient against the loop kept as a reference. Memory is now cubic in the batch, which is noted in the docstring: a much larger batch_size or aug_positives would need chunking over anchors. Nothing about the method changes, so the 50 datasets that already finished are unaffected. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
65 of the core 66 at resample 0, EmoPain the only gap, which was already outside the scored set, so the leaderboard goes to 25 estimators still on 56 datasets. XCM enters at rank 16.58. These are the fixed-parameter results: one fit at window 0.8 with batch 32, not the per-dataset cross-validated search the paper describes. The search was run over the whole core and did not pay for itself. Against the single fit on the 65 shared datasets it was 0.017 mean accuracy worse, 31 wins to 31 with 3 ties, Wilcoxon p = 0.63. On the 14 datasets where the search selected 0.8, the same window as the fixed run, so initialisation is the only difference, the two runs still differed by 0.11 mean absolute accuracy and by up to 0.48. At one resample XCM's run-to-run variance is larger than the effect being tuned, which makes the selection largely a choice over noise. A reader cannot infer that from a table row, so ESTIMATOR_NOTES carries the caveat onto every page alongside WITHHELD_ESTIMATORS, and docs/classifiers.md now leads with which run is published rather than describing the protocol as if it were the one reported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The dense ranking loss fixed RankSCL's training: seven reruns reached epoch 91 in four to thirty-nine minutes, against sixty hours before. They then sat in the probe for 42 hours and are heading for the same wall, with nothing in the log after the last epoch because the probe prints nothing. The probe was built as SVC(probability=True) and handed to a GridSearchCV over ten values of C with five folds. libsvm implements Platt scaling by an internal five-fold cross-validation inside every fit, so that search costs about 300 SVC trainings on 10000 x 320 rather than 50. The authors set probability=False in their own grid; only the final estimator needs probabilities, because aeon classifiers must implement predict_proba. _fit_probe now selects with it off and refits once at the chosen C with it on. The selection is unchanged: GridSearchCV scores with predict, which reads the decision function whether or not Platt scaling is fitted, so this is strictly more faithful to the authors than what it replaces. TS2Vec carried the same line, and its four timeouts were attributed solely to the missing MAX_SAMPLES cap. The cap was real, but this was the larger half, and those four are queued to rerun. Tests pin probability=False on both the searched estimator and the degenerate-case bypass, and that the fitted probe still offers predict_proba. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
00ec539 moved Platt scaling out of the grid search but left the module docstring describing the behaviour it replaced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Twelve gap records resolved across ten estimators. The leaderboard stays at 25 estimators on 56 datasets, because none of the ten dropped datasets was closed outright, but every one of them is nearer. HC2 on AustraliaRainfall_disc and USCActivity was not run. Both were rebuilt from their components with tsml_eval's FromFileHIVECOTE: HC2 is a CAWPE ensemble of STC, DrCIF, Arsenal and TDE, so where all four have a train and a test result its output is determined, the weights being each component's train accuracy to the fourth power. The standalone components were run at HC2's own internal defaults, so the configuration is faithful; it is not bitwise what a direct run gives, since the components inside HC2 would be fitted from HC2's random state. AustraliaRainfall_disc 0.7654, USCActivity 0.7473. Summary and TSF on AustraliaRainfall_disc were run locally, at the parameters recorded in their other result files: 113s and 1771s respectively. Both are now complete at 66. EmoPain returned for ConvTran, PatchMTSC, TimesNet and TS2Vec once aeon was upgraded past #3598, which relegated its low-variance check to a warning. The data was never at fault: the flagged channels are sparse rather than mis-scaled. The remaining gaps concentrate on a small number of causes. aeon issue 3738, LAPACK integer overflow in RidgeClassifierCV's SVD, accounts for RDST and ROCKET on AustraliaRainfall_disc and RDST and MRHydra on Tiselac. TS2Vec's four are the probe timeouts now fixed. ConvTran's three are CUDA OOM on the A100 and are queued for the H200. HC2 and TDE on STEW wait on a single TDE run, which would let HC2 be rebuilt from file as well. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both are held out rather than reported as missing, because no amount of scheduling closes them and leaving them in the denominator makes the scored fraction look like a queue still draining. PenDigits has series of length 8 and MRHydra requires at least 9, so it cannot complete while MRHydra is a column. AustraliaRainfall_disc has 112186 cases and three estimators that fail on it for reasons compute cannot fix: RDST and ROCKET hit LAPACK integer overflow in RidgeClassifierCV's SVD, aeon issue 3738, after 14 and 12 attempts, and MRHydra exhausted 128 GB over 13. DEFERRED_DATASETS sits beside WITHHELD_ESTIMATORS and ESTIMATOR_NOTES, and is rendered on every page under "Datasets not included" with its reason, so the omission is stated rather than inferred. The deferral applies to the UEA table as well: a dataset MRHydra cannot fit is no more completable inside the UEA 30 than inside Multiverse-core. Scoring is unchanged at 56 datasets; the honest denominator is now 64 rather than 66, and the results already collected for both stay in the repository. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Checklist
results/schema.md.