Leaderboard link - #23
Merged
Merged
Conversation
Set up so our port and aeon's DisjointCNNClassifier can be run against each other under tsml-eval, to attribute the gap in aeon issue #3775, where aeon scored 20.3 accuracy points below the published numbers on all 23 shared UEA datasets. Comparing the two graphs found the likely cause, and it is not the training procedure. aeon applies a Permute after the final block, so the tensor entering the pooling is (time, filters, 1) rather than (time, 1, filters). GlobalAveragePooling2D reduces the two leading axes, so aeon's classifier head receives one scalar per case where it should receive 64 features. Its Dense(128) has 256 parameters, 1*128 + 128, which confirms the input width is 1. That explains the shape of the failure in the issue: accuracy correlates -0.425 with class count and collapses toward chance on the many-class problems, because a single scalar cannot separate 26 handwriting classes. Head to head on ERing, 200 epochs, default split: aeon 0.641, this port 0.952, published 0.964. The port is the authors' Keras network transcribed, with their training procedure: class-weighted loss, batch size min(n_cases // 10, 8), 500 epochs, and learning rate schedule and retained epoch both monitoring validation loss. aeon does none of those four. They are documented on the class so that a run of the two can separate the head bug from the training differences. Two things recorded rather than reproduced. The authors' DCNN_4L applies the third block's spatial convolution to conv2 rather than to its own temporal convolution, so a layer is computed and discarded; this port builds the clean stack, matching aeon, so the architecture is not a second moving part. And the data pipeline is not a difference at all: Main.py passes normalise=False, so the authors train on raw series, as aeon does. Registered in tsml-eval as DisjointCNN-MV, beside aeon's DisjointCNN rather than replacing it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The leaderboard answers "which estimator is best". This answers "what is this dataset worth", which is the question an archive has to keep asking of its own problems. Per dataset: the Dummy floor, the median, best and worst over the other estimators, which estimator was best, the gain over Dummy, the spread, and how many estimators contributed. Sorted by gain, so the problems where nothing yet beats the baseline are at the top. Two departures from the columns as first sketched. A single range would have been close to redundant, since Dummy is almost always the weakest entry, so a best-minus-worst range mostly restates best-minus-Dummy; they are split into gain, meaning how much skill was found at all, and spread, meaning how much the choice of estimator mattered, because those answer different questions. And the median is reported beside the best, because best is a maximum over 23 estimators and so optimistic by construction. Unlike the leaderboard this keeps every dataset rather than only those every estimator finished, since a partly covered dataset is still informative; the estimator count carries the caveat. Rows are shaded where a dataset separates estimators poorly: amber for a gain of 0.05 or less, green for a best of 0.99 or more. On the current results that is 4 and 15 of 66. AtrialFibrillation and KINECAL-QSEO have a gain of exactly zero, so no estimator has yet beaten the majority class on them. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A placeholder for runtime and memory comparisons, with nothing published yet and the reasons why written down. The measurements already exist: tsml-eval records fit_time, predict_time, memory_usage and benchmark_time per run, and the ingest simply does not bring them across. The reason to hold off is that the numbers we hold are not comparable with each other. Results span H200, A100 and CPU-only nodes; GPU and CPU methods are not on one axis, so a ratio between them is partly a statement about hardware; wall-clock includes queueing and the controller's memory escalation retries; a deep learner's fit time is close to linear in an epoch count that is a choice rather than a property; memory_usage is peak host memory, so a model doing its work on a GPU looks cheap while occupying device memory nobody measured; and the recorded device is unreliable, since the tooling used for these runs inspects TensorFlow only and reports CPU for PyTorch estimators either way. The page also sets out what a fair comparison would need, including normalising CPU times by benchmark_time, which exists for that purpose, and notes there is no GPU equivalent. This is the same reasoning that keeps time and memory columns off the leaderboard, so the two now cross-reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The page opened on the numbers not being comparable, which reads as a data problem to be cleaned up. The more basic point is that no timing experiment has been designed or run: every result here came from runs set up to measure predictive performance, where partition, cores, epochs and memory retries were chosen to get accurate results out cheaply and left free across estimators because nothing depended on them. The timings are a by-product. That now leads, and the reasons below it are framed as the conditions such an experiment would have to fix rather than as defects in the current figures. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Same shape as the runtime page and leading on the same point: no experiment has been structured to compare memory. A job was given whatever ceiling got it to finish, on whatever node was free, with whatever core count came with the partition, none of it held constant across estimators. Memory has enough of its own difficulties to warrant a separate page rather than a section. What is recorded is one number, host side, fit only. Peak process memory includes the interpreter, the imported framework and the dataset, so without a per-framework baseline it largely ranks frameworks; on problems like EigenWorms the data dominates the model. Host and device memory are different quantities and only the host one is recorded. Framework allocators hold memory they are not using, so a naive device reading measures allocator policy. The point that decided the separate page is censoring. Where memory mattered most we have no number at all, only a bound: missing_results.csv records ConvTran hitting CUDA out of memory on Alzheimers, EigenWorms and PhotoStimulation, and FreshPRINCE hitting OOM at 128 GB on FaceDetection, FordChallenge and Skoda after eight attempts. A table built from successful runs alone would be survivorship-biased in a way a runtime table is not. The runtime page is narrowed to runtime and the two cross-reference. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
65 of 66 Multiverse-core datasets, seven metrics, from the i7_h200 run at the
port's defaults: window_size 0.8 capped at 100 points, batch size 32, 100
epochs. EmoPain is the one gap and is the archive-wide one, aeon's input
validation rejecting it before fit for every classifier, now recorded in
missing_results.csv beside the other four.
XCM lands 21st of 25 on average accuracy rank, mean accuracy 0.6690.
The ingest logged "y_prob values do not sum to one" on many datasets. The
probabilities are fine: the largest deviation across all 65 is 2.8e-07, ordinary
float32 rounding from the Keras softmax, well inside anything that would affect
log loss or AUROC.
Against the paper's own table, on the 23 shared datasets, we average 0.699 where
Fauvel et al. report 0.761, a gap of 6.2 points. That is the expected size: they
grid-search window_size per dataset over {20,40,60,80,100}% by cross-validation
on the training set, and report a 7.0% mean relative accuracy drop from using a
suboptimal window, where this run uses the modal 0.8 everywhere. Unlike
DisjointCNN's 20.3 points, this gap is explained by a documented protocol
difference rather than an implementation fault.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The port fixed window_size at 0.8, the modal published value, where the paper
sets it per dataset. Section 4.3 grid-searches window_size over
{0.2,0.4,0.6,0.8,1.0} and batch_size over {1,8,32} by stratified five-fold
cross-validation of the training set, so their figures are tuned but not leaked.
Both parameters now accept a sequence, which runs that search and refits on the
whole training set; a scalar fits once exactly as before, so the default is
unchanged. The selection is recorded on window_fraction_, batch_size_ and
cv_results_. Folds are reduced when a class is smaller than cv_folds, and the
search is skipped when the rarest class appears once, since there is nothing to
select on.
max_window now accepts None. The 100 point bound is ours rather than the
authors', who do run kernels of the order 40% of EigenWorms, 7193 points, so
None is needed to reproduce them.
The default stays a single fit because of cost. The fixed-window pass over
Multiverse-core took 1.2 GPU-hours; searching the window alone is about 20 times
that, and adding the batch grid about 37 times again, since batch 1 takes 32
times the gradient steps of batch 32. That puts the full grid near 900
GPU-hours for a batch choice the published table makes on 4 of 30 datasets, so
XCM-Tuned in tsml-eval searches the window only.
For context on what this is worth: fixed at 0.8 we average 0.699 over the 23
datasets shared with the paper's table against their 0.761, and they report a
7.0% mean relative drop from a suboptimal window.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
TimesNet: 65 of 66 datasets, seven metrics, from the rerun with lr_adjust at the implementation default of None. EmoPain is the usual gap. Mean accuracy 0.7013, 22nd of 25 on rank. Against the paper's own table, on the nine shared datasets, we average 0.678 where they report 0.754. A 7.6 point gap is what the protocol difference predicts: TSLib selects the retained epoch on the test set, so their figure is not a held-out estimate. The XCM results are removed rather than kept. They came from a single fit at window_size 0.8, which is the modal published value but not a configuration the authors ever ran: they select the window per dataset by cross-validation on the training set. Reporting the fixed version measures something the paper does not claim, and the 6.2 point shortfall against their table is the size the paper's own sensitivity figure predicts for a suboptimal window. XCM in the tsml-eval lookup now supplies the window grid, so the reported run follows the authors' protocol, and XCM-Fixed keeps the single-fit variant for comparison. docs/classifiers.md states which is reported and why batch size is held at the published modal 32 rather than searched. The missing_results row for XCM goes with the results and will be re-recorded when the tuned run is ingested. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
placeholders for time and memory