From a175733b89317ba8608065519a8ae8616a2d219c Mon Sep 17 00:00:00 2001 From: Tony Bagnall Date: Mon, 31 Aug 2026 15:25:28 +0100 Subject: [PATCH 01/12] Ingest TimesNet results 65 of the 66 Multiverse-core datasets, ingested with multiverse.experiments.ingest. TimesNet places 22nd of 24 on average accuracy rank, ahead of 1NN-DTW and Dummy. EigenWorms and Alzheimers are both present, which is worth recording: at 17984 and 15000 points they exceed the fixed 5000 position table TimesNet inherited from TSLib, and would have failed with a tensor size mismatch before that was sized to the series. The run also used d_model 32 and d_ff 64 with the type1 learning rate schedule, so it picked up the published classification defaults rather than the earlier values. EmoPain is missing. No TimesNet job logs were copied across, so the reason is recorded as inferred: aeon rejects EmoPain before fit for having 1733 case/channel pairs with std <= 1e-07, which stops every aeon classifier, and is the logged cause for ConvTran, PatchMTSC and DisjointCNN. It costs no datasets, since those three already exclude it. Co-Authored-By: Claude Opus 5 --- README.md | 51 +++++++------- .../multiverse/TimesNet/TimesNet_accuracy.csv | 66 +++++++++++++++++++ .../multiverse/TimesNet/TimesNet_auroc.csv | 66 +++++++++++++++++++ .../multiverse/TimesNet/TimesNet_balacc.csv | 66 +++++++++++++++++++ results/multiverse/TimesNet/TimesNet_f1.csv | 66 +++++++++++++++++++ .../multiverse/TimesNet/TimesNet_logloss.csv | 66 +++++++++++++++++++ .../TimesNet/TimesNet_sensitivity.csv | 66 +++++++++++++++++++ .../TimesNet/TimesNet_specificity.csv | 66 +++++++++++++++++++ results/multiverse/leaderboard.html | 4 +- results/multiverse/missing_results.csv | 1 + 10 files changed, 491 insertions(+), 27 deletions(-) create mode 100644 results/multiverse/TimesNet/TimesNet_accuracy.csv create mode 100644 results/multiverse/TimesNet/TimesNet_auroc.csv create mode 100644 results/multiverse/TimesNet/TimesNet_balacc.csv create mode 100644 results/multiverse/TimesNet/TimesNet_f1.csv create mode 100644 results/multiverse/TimesNet/TimesNet_logloss.csv create mode 100644 results/multiverse/TimesNet/TimesNet_sensitivity.csv create mode 100644 results/multiverse/TimesNet/TimesNet_specificity.csv diff --git a/README.md b/README.md index 6e83707..b8d251a 100644 --- a/README.md +++ b/README.md @@ -34,31 +34,32 @@ The current paper version describes: | # | Estimator | Accuracy | Balanced accuracy | AUROC | F1 | Log loss ↓ | Sensitivity | Specificity | Accuracy rank | |---|---|---|---|---|---|---|---|---|---| -| 1 | HC2 | **0.7909** | 0.7518 | **0.8990** | 0.7273 | **0.5383** | 0.7459 | **0.7943** | **7.64** | -| 2 | MRHydra | 0.7837 | **0.7564** | 0.8105 | **0.7316** | 7.7974 | **0.7642** | 0.7757 | 8.14 | -| 3 | RDST | 0.7734 | 0.7333 | 0.7912 | 0.6991 | 8.1667 | 0.7109 | 0.7874 | 9.02 | -| 4 | RIST | 0.7720 | 0.7397 | 0.8748 | 0.7147 | 0.6218 | 0.7408 | 0.7655 | 9.60 | -| 5 | DrCIF | 0.7747 | 0.7429 | 0.8813 | 0.7173 | 0.6484 | 0.7397 | 0.7708 | 9.86 | -| 6 | FreshPRINCE | 0.7743 | 0.7487 | 0.8745 | 0.7211 | 0.6007 | 0.7414 | 0.7770 | 9.87 | -| 7 | CIF | 0.7781 | 0.7471 | 0.8908 | 0.7212 | 0.6430 | 0.7441 | 0.7753 | 9.94 | -| 8 | QUANT | 0.7720 | 0.7462 | 0.8831 | 0.7189 | 0.6175 | 0.7521 | 0.7581 | 10.29 | -| 9 | Arsenal | 0.7680 | 0.7321 | 0.8457 | 0.7024 | 3.8631 | 0.7257 | 0.7732 | 10.41 | -| 10 | ROCKET | 0.7690 | 0.7326 | 0.7925 | 0.7019 | 8.3249 | 0.7200 | 0.7764 | 10.58 | -| 11 | LITETime-MV | 0.7506 | 0.7299 | 0.8513 | 0.6820 | 1.3206 | 0.7132 | 0.7637 | 10.92 | -| 12 | STSF | 0.7724 | 0.7477 | 0.8804 | 0.7080 | 0.6432 | 0.7345 | 0.7826 | 11.29 | -| 13 | H-InceptionTime | 0.7408 | 0.7190 | 0.8496 | 0.6838 | 1.3227 | 0.7223 | 0.7378 | 11.39 | -| 14 | LiteTIME | 0.7341 | 0.7104 | 0.8394 | 0.6680 | 1.4776 | 0.7113 | 0.7336 | 12.08 | -| 15 | PatchMTSC | 0.7428 | 0.6897 | 0.8261 | 0.6533 | 0.7655 | 0.6852 | 0.7352 | 12.77 | -| 16 | ConvTran | 0.7462 | 0.7102 | 0.8592 | 0.6767 | 0.8190 | 0.7159 | 0.7345 | 12.89 | -| 17 | Catch22 | 0.7475 | 0.7181 | 0.8697 | 0.6922 | 0.7147 | 0.7240 | 0.7374 | 12.93 | -| 18 | STC | 0.7545 | 0.7172 | 0.8744 | 0.6940 | 0.6391 | 0.7185 | 0.7537 | 13.63 | -| 19 | TSF | 0.7515 | 0.7236 | 0.8740 | 0.6883 | 0.7252 | 0.7093 | 0.7606 | 13.63 | -| 20 | TDE | 0.7262 | 0.6813 | 0.8374 | 0.6382 | 0.8869 | 0.6714 | 0.7344 | 14.21 | -| 21 | Summary | 0.6858 | 0.6574 | 0.8268 | 0.6230 | 0.9123 | 0.6574 | 0.6844 | 16.12 | -| 22 | 1NN-DTW | 0.6712 | 0.6454 | 0.7197 | 0.6136 | 11.8506 | 0.6521 | 0.6636 | 17.82 | -| 23 | Dummy | 0.3645 | 0.3029 | 0.5000 | 0.1507 | 1.4067 | 0.2855 | 0.3816 | 20.95 | - -Test results for default train/test split. All classifiers trained with default settings. Average over the 52 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. +| 1 | HC2 | **0.7909** | 0.7518 | **0.8990** | 0.7273 | **0.5383** | 0.7459 | **0.7943** | **7.83** | +| 2 | MRHydra | 0.7837 | **0.7564** | 0.8105 | **0.7316** | 7.7974 | **0.7642** | 0.7757 | 8.34 | +| 3 | RDST | 0.7734 | 0.7333 | 0.7912 | 0.6991 | 8.1667 | 0.7109 | 0.7874 | 9.19 | +| 4 | RIST | 0.7720 | 0.7397 | 0.8748 | 0.7147 | 0.6218 | 0.7408 | 0.7655 | 9.74 | +| 5 | FreshPRINCE | 0.7743 | 0.7487 | 0.8745 | 0.7211 | 0.6007 | 0.7414 | 0.7770 | 10.00 | +| 6 | DrCIF | 0.7747 | 0.7429 | 0.8813 | 0.7173 | 0.6484 | 0.7397 | 0.7708 | 10.04 | +| 7 | CIF | 0.7781 | 0.7471 | 0.8908 | 0.7212 | 0.6430 | 0.7441 | 0.7753 | 10.09 | +| 8 | QUANT | 0.7720 | 0.7462 | 0.8831 | 0.7189 | 0.6175 | 0.7521 | 0.7581 | 10.47 | +| 9 | Arsenal | 0.7680 | 0.7321 | 0.8457 | 0.7024 | 3.8631 | 0.7257 | 0.7732 | 10.69 | +| 10 | ROCKET | 0.7690 | 0.7326 | 0.7925 | 0.7019 | 8.3249 | 0.7200 | 0.7764 | 10.85 | +| 11 | LITETime-MV | 0.7506 | 0.7299 | 0.8513 | 0.6820 | 1.3206 | 0.7132 | 0.7637 | 11.25 | +| 12 | STSF | 0.7724 | 0.7477 | 0.8804 | 0.7080 | 0.6432 | 0.7345 | 0.7826 | 11.46 | +| 13 | H-InceptionTime | 0.7408 | 0.7190 | 0.8496 | 0.6838 | 1.3227 | 0.7223 | 0.7378 | 11.68 | +| 14 | LiteTIME | 0.7341 | 0.7104 | 0.8394 | 0.6680 | 1.4776 | 0.7113 | 0.7336 | 12.39 | +| 15 | PatchMTSC | 0.7428 | 0.6897 | 0.8261 | 0.6533 | 0.7655 | 0.6852 | 0.7352 | 12.98 | +| 16 | ConvTran | 0.7462 | 0.7102 | 0.8592 | 0.6767 | 0.8190 | 0.7159 | 0.7345 | 13.09 | +| 17 | Catch22 | 0.7475 | 0.7181 | 0.8697 | 0.6922 | 0.7147 | 0.7240 | 0.7374 | 13.15 | +| 18 | TSF | 0.7515 | 0.7236 | 0.8740 | 0.6883 | 0.7252 | 0.7093 | 0.7606 | 13.89 | +| 19 | STC | 0.7545 | 0.7172 | 0.8744 | 0.6940 | 0.6391 | 0.7185 | 0.7537 | 13.94 | +| 20 | TDE | 0.7262 | 0.6813 | 0.8374 | 0.6382 | 0.8869 | 0.6714 | 0.7344 | 14.59 | +| 21 | Summary | 0.6858 | 0.6574 | 0.8268 | 0.6230 | 0.9123 | 0.6574 | 0.6844 | 16.53 | +| 22 | TimesNet | 0.6643 | 0.6287 | 0.8105 | 0.5904 | 1.0058 | 0.6305 | 0.6593 | 17.71 | +| 23 | 1NN-DTW | 0.6712 | 0.6454 | 0.7197 | 0.6136 | 11.8506 | 0.6521 | 0.6636 | 18.31 | +| 24 | Dummy | 0.3645 | 0.3029 | 0.5000 | 0.1507 | 1.4067 | 0.2855 | 0.3816 | 21.79 | + +Average over the 52 Multiverse-core datasets with results for every estimator on every metric, ordered by average accuracy rank. Best in each column in bold. Rebuilt with `python -m multiverse.experiments.tables`, which also writes a sortable diff --git a/results/multiverse/TimesNet/TimesNet_accuracy.csv b/results/multiverse/TimesNet/TimesNet_accuracy.csv new file mode 100644 index 0000000..9893800 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_accuracy.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.37209302325581395 +AppliancesEnergy_disc,0.7857142857142857 +ArticularyWordRecognition,0.9466666666666667 +AsphaltObstaclesCoordinates,0.7263427109974424 +AsphaltRegularityCoordinates,0.9307589880159787 +AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.7785611780121046 +AutomotiveRoadTrials,0.7792207792207793 +BIDMC32HR_disc,0.5256356815339724 +BIDMC32SpO2_disc,0.7027928303459775 +BeijingPM10Quality_disc,0.8230982567353408 +BeijingPM25Quality_disc,0.8785657686212361 +BenzeneConcentration_disc,0.8859190393182258 +Blink,0.7333333333333333 +BoneIntensitiesAgeGroup,0.7258426966292135 +BoneProbAgeGroup,0.5370786516853933 +CharacterTrajectories,0.9811977715877437 +CounterMovementJump,0.6871508379888268 +Cricket,0.8888888888888888 +CrowdSourced,0.7317331450758913 +DuckDuckGeese,0.42 +ERing,0.4185185185185185 +EigenWorms,0.4732824427480916 +Epilepsy,0.8115942028985508 +EthanolConcentration,0.2889733840304182 +EyesOpenShut,0.5952380952380952 +FaceDetection,0.6677071509648127 +FordChallenge,0.8888888888888888 +HandMovementDirection,0.527027027027027 +Handwriting,0.15529411764705883 +Heartbeat,0.6829268292682927 +HouseholdPowerConsumption1_disc,0.924198250728863 +HouseholdPowerConsumption2_disc,0.6282798833819242 +IEEEPPG_disc,0.34262048192771083 +IRDS-SFL,0.7413793103448276 +JapaneseVowels,0.8513513513513513 +KERAAL-RTK,0.42857142857142855 +KIMORE-PR-C,0.2857142857142857 +KINECAL-QSEO,0.9411764705882353 +LSST,0.4744525547445255 +Libras,0.42777777777777776 +Locust2022,0.9072446195816914 +LowCost,0.6433333333333333 +MindReading,0.36294027565084225 +MotionSenseHAR,0.9018867924528302 +MotorImagery,0.48 +NATOPS,0.6611111111111111 +PEMS-SF,0.6127167630057804 +PenDigits,0.9785591766723842 +PhonemeSpectra,0.13301521025946914 +PhotoStimulation,0.3888888888888889 +RacketSports,0.5921052631578947 +STEW,0.7462121212121212 +SelfRegulationSCP1,0.8191126279863481 +Skoda,0.9342058719490626 +SpokenArabicDigits,0.9768076398362893 +StandWalkJump,0.26666666666666666 +TactileTextureRecognition,0.9162995594713657 +Tiselac,0.8094539939332659 +UCDHE-Rowing-MC,0.7454545454545455 +UCIActivity,0.9695960016659725 +UIPRMD-DS-C,0.6388888888888888 +USCActivity,0.6690461725394897 +UWaveGestureLibrary,0.790625 +WISDM,0.8744446590689589 diff --git a/results/multiverse/TimesNet/TimesNet_auroc.csv b/results/multiverse/TimesNet/TimesNet_auroc.csv new file mode 100644 index 0000000..cf124e0 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_auroc.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.5601764234161989 +AppliancesEnergy_disc,0.463235294117647 +ArticularyWordRecognition,0.9968055555555556 +AsphaltObstaclesCoordinates,0.9123217502870393 +AsphaltRegularityCoordinates,0.9827481024331418 +AtrialFibrillation,0.3466666666666667 +AustraliaRainfall_disc,0.8634142388219334 +AutomotiveRoadTrials,0.8312159709618875 +BIDMC32HR_disc,0.5953081985105932 +BIDMC32SpO2_disc,0.43076103983863867 +BeijingPM10Quality_disc,0.8844808777621117 +BeijingPM25Quality_disc,0.9407219774895562 +BenzeneConcentration_disc,0.9633165297099491 +Blink,0.97954 +BoneIntensitiesAgeGroup,0.859510458528784 +BoneProbAgeGroup,0.6887822224013965 +CharacterTrajectories,0.9994980589164253 +CounterMovementJump,0.8655775159225702 +Cricket,0.9974747474747474 +CrowdSourced,0.8176827286899594 +DuckDuckGeese,0.6990000000000001 +ERing,0.824164609053498 +EigenWorms,0.718388522476044 +Epilepsy,0.959455672434579 +EthanolConcentration,0.5653979528156827 +EyesOpenShut,0.7414965986394558 +FaceDetection,0.7330057810170828 +FordChallenge,0.9456426176562067 +HandMovementDirection,0.7396379044684129 +Handwriting,0.6462913698440567 +Heartbeat,0.7318634423897582 +HouseholdPowerConsumption1_disc,0.9697122530049018 +HouseholdPowerConsumption2_disc,0.5701212735874284 +IEEEPPG_disc,0.5353214645905319 +IRDS-SFL,0.8713768115942029 +JapaneseVowels,0.9865154676979703 +KERAAL-RTK,0.8958333333333334 +KIMORE-PR-C,0.5 +KINECAL-QSEO,0.5625 +LSST,0.8046668528791036 +Libras,0.9001653439153439 +Locust2022,0.808517776733223 +LowCost,0.6967111111111111 +MindReading,0.6966572150087443 +MotionSenseHAR,0.978391972002462 +MotorImagery,0.44999999999999996 +NATOPS,0.926074074074074 +PEMS-SF,0.8808031009894832 +PenDigits,0.999382338517799 +PhonemeSpectra,0.7839772428931948 +PhotoStimulation,0.4712067562067562 +RacketSports,0.8241386064232483 +STEW,0.8157471018956242 +SelfRegulationSCP1,0.953499207902339 +Skoda,0.9948829449552933 +SpokenArabicDigits,0.9996008588337867 +StandWalkJump,0.5199999999999999 +TactileTextureRecognition,0.9974426855960103 +Tiselac,0.9623010032487034 +UCDHE-Rowing-MC,0.9414196610667199 +UCIActivity,0.9990284187495908 +UIPRMD-DS-C,0.7222222222222222 +USCActivity,0.958313351560365 +UWaveGestureLibrary,0.9689062500000001 +WISDM,0.9648921913644327 diff --git a/results/multiverse/TimesNet/TimesNet_balacc.csv b/results/multiverse/TimesNet/TimesNet_balacc.csv new file mode 100644 index 0000000..88352e9 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_balacc.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.33164983164983164 +AppliancesEnergy_disc,0.4852941176470588 +ArticularyWordRecognition,0.9466666666666668 +AsphaltObstaclesCoordinates,0.7239738848141597 +AsphaltRegularityCoordinates,0.9303929914166135 +AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.4012438850443972 +AutomotiveRoadTrials,0.588021778584392 +BIDMC32HR_disc,0.39531645613163274 +BIDMC32SpO2_disc,0.5741082977539786 +BeijingPM10Quality_disc,0.7754261379919071 +BeijingPM25Quality_disc,0.8561546854335564 +BenzeneConcentration_disc,0.8333372774343781 +Blink,0.7595000000000001 +BoneIntensitiesAgeGroup,0.7037761415350002 +BoneProbAgeGroup,0.45082156728048 +CharacterTrajectories,0.9798461115661856 +CounterMovementJump,0.6878531073446328 +Cricket,0.8888888888888888 +CrowdSourced,0.7317944132786303 +DuckDuckGeese,0.41999999999999993 +ERing,0.4185185185185185 +EigenWorms,0.2979108813891423 +Epilepsy,0.8112082670906201 +EthanolConcentration,0.2884032634032634 +EyesOpenShut,0.5952380952380952 +FaceDetection,0.6677071509648127 +FordChallenge,0.8747557137310917 +HandMovementDirection,0.530952380952381 +Handwriting,0.15038873399040267 +Heartbeat,0.6401730678046468 +HouseholdPowerConsumption1_disc,0.8637379676661375 +HouseholdPowerConsumption2_disc,0.48565564318988974 +IEEEPPG_disc,0.3708925748661307 +IRDS-SFL,0.7753623188405797 +JapaneseVowels,0.8351985691974568 +KERAAL-RTK,0.5 +KIMORE-PR-C,0.5833333333333334 +KINECAL-QSEO,0.5 +LSST,0.29218620313725446 +Libras,0.42777777777777787 +Locust2022,0.5656042863079302 +LowCost,0.6433333333333333 +MindReading,0.3508160498636541 +MotionSenseHAR,0.8779953202845631 +MotorImagery,0.48 +NATOPS,0.6611111111111111 +PEMS-SF,0.6070261139826357 +PenDigits,0.9788324983474237 +PhonemeSpectra,0.13302465887965206 +PhotoStimulation,0.3111111111111111 +RacketSports,0.5951485245260895 +STEW,0.7462121212121212 +SelfRegulationSCP1,0.8195881092162892 +Skoda,0.9254113989967009 +SpokenArabicDigits,0.9768119551681196 +StandWalkJump,0.26666666666666666 +TactileTextureRecognition,0.9142122043777865 +Tiselac,0.6237679935905254 +UCDHE-Rowing-MC,0.7559761904761906 +UCIActivity,0.9696253555252076 +UIPRMD-DS-C,0.6388888888888888 +USCActivity,0.6761836489300211 +UWaveGestureLibrary,0.790625 +WISDM,0.5499107526502924 diff --git a/results/multiverse/TimesNet/TimesNet_f1.csv b/results/multiverse/TimesNet/TimesNet_f1.csv new file mode 100644 index 0000000..d93b0d0 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_f1.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.3083127164769916 +AppliancesEnergy_disc,0.0 +ArticularyWordRecognition,0.9447295291707192 +AsphaltObstaclesCoordinates,0.7247463798100227 +AsphaltRegularityCoordinates,0.9279778393351801 +AtrialFibrillation,0.21666666666666667 +AustraliaRainfall_disc,0.7616817448566727 +AutomotiveRoadTrials,0.32 +BIDMC32HR_disc,0.5067507072674818 +BIDMC32SpO2_disc,0.34527089072543615 +BeijingPM10Quality_disc,0.6838938053097345 +BeijingPM25Quality_disc,0.799213887979037 +BenzeneConcentration_disc,0.7907637655417407 +Blink,0.7683397683397684 +BoneIntensitiesAgeGroup,0.7231843654079018 +BoneProbAgeGroup,0.5181537964851896 +CharacterTrajectories,0.9812488580364788 +CounterMovementJump,0.6950658623728717 +Cricket,0.8895909645909647 +CrowdSourced,0.7713598074608905 +DuckDuckGeese,0.4033511586452762 +ERing,0.38176931128876207 +EigenWorms,0.35942968120789176 +Epilepsy,0.7884851881255935 +EthanolConcentration,0.2722492035191654 +EyesOpenShut,0.6792452830188679 +FaceDetection,0.6552840741831027 +FordChallenge,0.8471748198710656 +HandMovementDirection,0.5334621841559374 +Handwriting,0.10939521947870232 +Heartbeat,0.4881889763779528 +HouseholdPowerConsumption1_disc,0.9243358966284373 +HouseholdPowerConsumption2_disc,0.6121944360702697 +IEEEPPG_disc,0.29862067776974394 +IRDS-SFL,0.5714285714285714 +JapaneseVowels,0.8461624758662067 +KERAAL-RTK,0.6 +KIMORE-PR-C,0.2857142857142857 +KINECAL-QSEO,0.0 +LSST,0.3976143241095737 +Libras,0.38902188198725685 +Locust2022,0.2233502538071066 +LowCost,0.6385135135135135 +MindReading,0.3382508586266084 +MotionSenseHAR,0.9049992060482481 +MotorImagery,0.4090909090909091 +NATOPS,0.6593343119205188 +PEMS-SF,0.6019313305058753 +PenDigits,0.978579711883136 +PhonemeSpectra,0.13105612511029194 +PhotoStimulation,0.23333333333333334 +RacketSports,0.5833452665457628 +STEW,0.7274370950730752 +SelfRegulationSCP1,0.8408408408408409 +Skoda,0.9338560883310366 +SpokenArabicDigits,0.9768053196986767 +StandWalkJump,0.2545454545454546 +TactileTextureRecognition,0.9154304314374941 +Tiselac,0.8000370725018578 +UCDHE-Rowing-MC,0.7401906666335435 +UCIActivity,0.9694915832342339 +UIPRMD-DS-C,0.5517241379310345 +USCActivity,0.6563872199532003 +UWaveGestureLibrary,0.7704187365622013 +WISDM,0.8573365686974957 diff --git a/results/multiverse/TimesNet/TimesNet_logloss.csv b/results/multiverse/TimesNet/TimesNet_logloss.csv new file mode 100644 index 0000000..2b4bd91 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_logloss.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,1.248236038572558 +AppliancesEnergy_disc,0.5412784291807121 +ArticularyWordRecognition,0.8492528242945858 +AsphaltObstaclesCoordinates,0.7084097487726752 +AsphaltRegularityCoordinates,0.17954995823930306 +AtrialFibrillation,1.19812890448403 +AustraliaRainfall_disc,0.5008335291420626 +AutomotiveRoadTrials,0.4623138101766679 +BIDMC32HR_disc,3.832847325944141 +BIDMC32SpO2_disc,2.4123649216187286 +BeijingPM10Quality_disc,0.4257424758681892 +BeijingPM25Quality_disc,0.3181176717426027 +BenzeneConcentration_disc,0.29121361148804753 +Blink,0.6506888433736286 +BoneIntensitiesAgeGroup,0.6180810049964339 +BoneProbAgeGroup,0.9047540752213848 +CharacterTrajectories,0.07699067139539278 +CounterMovementJump,1.1845161607337837 +Cricket,0.7860967552483313 +CrowdSourced,1.6282154666878466 +DuckDuckGeese,1.5251056419662503 +ERing,1.5811664187709722 +EigenWorms,2.103077366172099 +Epilepsy,0.5498186174501463 +EthanolConcentration,1.4244886272948445 +EyesOpenShut,0.6353703673229939 +FaceDetection,0.8802738810169449 +FordChallenge,0.36841739423824155 +HandMovementDirection,1.2043115104844277 +Handwriting,3.206927412554193 +Heartbeat,0.5865044980705334 +HouseholdPowerConsumption1_disc,0.35263437730264885 +HouseholdPowerConsumption2_disc,1.33788754475725 +IEEEPPG_disc,2.700281326291473 +IRDS-SFL,0.4817444192621376 +JapaneseVowels,1.1181788717388894 +KERAAL-RTK,0.879617704562996 +KIMORE-PR-C,2.033304487545635 +KINECAL-QSEO,0.4393486598310768 +LSST,1.5122659524349011 +Libras,2.028803280243944 +Locust2022,0.25395280676626236 +LowCost,0.655424023355451 +MindReading,1.998291445399421 +MotionSenseHAR,0.7757995881404698 +MotorImagery,1.1773923833475948 +NATOPS,1.1499764605087988 +PEMS-SF,1.1547337532691098 +PenDigits,0.07047689242779125 +PhonemeSpectra,4.001894781448025 +PhotoStimulation,1.3357289781649082 +RacketSports,1.1551709817966538 +STEW,0.7312802797783392 +SelfRegulationSCP1,0.3899655112514777 +Skoda,0.1973274428464228 +SpokenArabicDigits,0.07744734280980718 +StandWalkJump,1.121926403618857 +TactileTextureRecognition,0.4198618383424086 +Tiselac,0.7436609220512062 +UCDHE-Rowing-MC,0.7006641043256774 +UCIActivity,0.1038315670986873 +UIPRMD-DS-C,0.6068853848762614 +USCActivity,1.3873971507868537 +UWaveGestureLibrary,0.8681987960496491 +WISDM,1.2681664616054182 diff --git a/results/multiverse/TimesNet/TimesNet_sensitivity.csv b/results/multiverse/TimesNet/TimesNet_sensitivity.csv new file mode 100644 index 0000000..9909ba4 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_sensitivity.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.37209302325581395 +AppliancesEnergy_disc,0.0 +ArticularyWordRecognition,0.9466666666666667 +AsphaltObstaclesCoordinates,0.7263427109974424 +AsphaltRegularityCoordinates,0.9054054054054054 +AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.7785611780121046 +AutomotiveRoadTrials,0.21052631578947367 +BIDMC32HR_disc,0.5256356815339724 +BIDMC32SpO2_disc,0.2752562225475842 +BeijingPM10Quality_disc,0.6625514403292181 +BeijingPM25Quality_disc,0.799475753604194 +BenzeneConcentration_disc,0.6947565543071161 +Blink,0.995 +BoneIntensitiesAgeGroup,0.7258426966292135 +BoneProbAgeGroup,0.5370786516853933 +CharacterTrajectories,0.9811977715877437 +CounterMovementJump,0.6871508379888268 +Cricket,0.8888888888888888 +CrowdSourced,0.905367231638418 +DuckDuckGeese,0.42 +ERing,0.4185185185185185 +EigenWorms,0.4732824427480916 +Epilepsy,0.8115942028985508 +EthanolConcentration,0.2889733840304182 +EyesOpenShut,0.8571428571428571 +FaceDetection,0.6316685584562997 +FordChallenge,0.8174167581412367 +HandMovementDirection,0.527027027027027 +Handwriting,0.15529411764705883 +Heartbeat,0.543859649122807 +HouseholdPowerConsumption1_disc,0.924198250728863 +HouseholdPowerConsumption2_disc,0.6282798833819242 +IEEEPPG_disc,0.34262048192771083 +IRDS-SFL,0.8333333333333334 +JapaneseVowels,0.8513513513513513 +KERAAL-RTK,1.0 +KIMORE-PR-C,1.0 +KINECAL-QSEO,0.0 +LSST,0.4744525547445255 +Libras,0.42777777777777776 +Locust2022,0.15017064846416384 +LowCost,0.63 +MindReading,0.36294027565084225 +MotionSenseHAR,0.9018867924528302 +MotorImagery,0.36 +NATOPS,0.6611111111111111 +PEMS-SF,0.6127167630057804 +PenDigits,0.9785591766723842 +PhonemeSpectra,0.13301521025946914 +PhotoStimulation,0.3888888888888889 +RacketSports,0.5921052631578947 +STEW,0.6773288439955106 +SelfRegulationSCP1,0.958904109589041 +Skoda,0.9342058719490626 +SpokenArabicDigits,0.9768076398362893 +StandWalkJump,0.26666666666666666 +TactileTextureRecognition,0.9162995594713657 +Tiselac,0.8094539939332659 +UCDHE-Rowing-MC,0.7454545454545455 +UCIActivity,0.9695960016659725 +UIPRMD-DS-C,0.4444444444444444 +USCActivity,0.6690461725394897 +UWaveGestureLibrary,0.790625 +WISDM,0.8744446590689589 diff --git a/results/multiverse/TimesNet/TimesNet_specificity.csv b/results/multiverse/TimesNet/TimesNet_specificity.csv new file mode 100644 index 0000000..c44e716 --- /dev/null +++ b/results/multiverse/TimesNet/TimesNet_specificity.csv @@ -0,0 +1,66 @@ +Resamples:,0 +Alzheimers,0.37209302325581395 +AppliancesEnergy_disc,0.9705882352941176 +ArticularyWordRecognition,0.9466666666666667 +AsphaltObstaclesCoordinates,0.7263427109974424 +AsphaltRegularityCoordinates,0.9553805774278216 +AtrialFibrillation,0.26666666666666666 +AustraliaRainfall_disc,0.7785611780121046 +AutomotiveRoadTrials,0.9655172413793104 +BIDMC32HR_disc,0.5256356815339724 +BIDMC32SpO2_disc,0.872960372960373 +BeijingPM10Quality_disc,0.888300835654596 +BeijingPM25Quality_disc,0.9128336172629188 +BenzeneConcentration_disc,0.97191800056164 +Blink,0.524 +BoneIntensitiesAgeGroup,0.7258426966292135 +BoneProbAgeGroup,0.5370786516853933 +CharacterTrajectories,0.9811977715877437 +CounterMovementJump,0.6871508379888268 +Cricket,0.8888888888888888 +CrowdSourced,0.5582215949188426 +DuckDuckGeese,0.42 +ERing,0.4185185185185185 +EigenWorms,0.4732824427480916 +Epilepsy,0.8115942028985508 +EthanolConcentration,0.2889733840304182 +EyesOpenShut,0.3333333333333333 +FaceDetection,0.7037457434733257 +FordChallenge,0.9320946693209466 +HandMovementDirection,0.527027027027027 +Handwriting,0.15529411764705883 +Heartbeat,0.7364864864864865 +HouseholdPowerConsumption1_disc,0.924198250728863 +HouseholdPowerConsumption2_disc,0.6282798833819242 +IEEEPPG_disc,0.34262048192771083 +IRDS-SFL,0.717391304347826 +JapaneseVowels,0.8513513513513513 +KERAAL-RTK,0.0 +KIMORE-PR-C,0.16666666666666666 +KINECAL-QSEO,1.0 +LSST,0.4744525547445255 +Libras,0.42777777777777776 +Locust2022,0.9810379241516967 +LowCost,0.6566666666666666 +MindReading,0.36294027565084225 +MotionSenseHAR,0.9018867924528302 +MotorImagery,0.6 +NATOPS,0.6611111111111111 +PEMS-SF,0.6127167630057804 +PenDigits,0.9785591766723842 +PhonemeSpectra,0.13301521025946914 +PhotoStimulation,0.3888888888888889 +RacketSports,0.5921052631578947 +STEW,0.8150953984287318 +SelfRegulationSCP1,0.6802721088435374 +Skoda,0.9342058719490626 +SpokenArabicDigits,0.9768076398362893 +StandWalkJump,0.26666666666666666 +TactileTextureRecognition,0.9162995594713657 +Tiselac,0.8094539939332659 +UCDHE-Rowing-MC,0.7454545454545455 +UCIActivity,0.9695960016659725 +UIPRMD-DS-C,0.8333333333333334 +USCActivity,0.6690461725394897 +UWaveGestureLibrary,0.790625 +WISDM,0.8744446590689589 diff --git a/results/multiverse/leaderboard.html b/results/multiverse/leaderboard.html index 47dad2d..b2930a6 100644 --- a/results/multiverse/leaderboard.html +++ b/results/multiverse/leaderboard.html @@ -60,12 +60,12 @@ details { margin-top: .6rem; } summary { cursor: pointer; color: var(--accent); } code { font-family: ui-monospace, SFMono-Regular, Menlo, monospace; font-size: .9em; } -

Multiverse-core leaderboard

23 estimators on 52 datasets · 7 metrics · ordered by average accuracy rank · built 2026-08-31

#EstimatorAccuracyBalanced accuracyAUROCF1Log loss ↓SensitivitySpecificity
ScoreRankScoreRankScoreRankScoreRankScoreRankScoreRankScoreRank
1HC20.79097.640.75188.280.89905.940.72738.080.53836.080.74598.490.79437.23
2MRHydra0.78378.140.75647.850.810516.240.73167.707.797418.890.76428.040.77578.92
3RDST0.77349.020.73339.540.791216.810.69919.888.166719.140.710910.500.78748.48
4RIST0.77209.600.739710.400.87487.950.714710.150.62188.290.740810.510.765510.52
5DrCIF0.77479.860.742910.260.88138.310.717310.600.64849.150.739711.120.770810.58
6FreshPRINCE0.77439.870.748710.260.87457.880.721110.160.60076.150.741410.600.777010.70
7CIF0.77819.940.747110.170.89088.140.721210.210.64309.130.744111.030.77539.99
8QUANT0.772010.290.746210.240.88317.490.718910.310.61757.330.752110.400.758111.19
9Arsenal0.768010.410.732110.290.845712.770.702410.413.863116.030.725710.520.773210.27
10ROCKET0.769010.580.732610.350.792517.520.701910.838.324919.870.720011.400.776410.73
11LITETime-MV0.750610.920.72999.600.85139.820.68209.701.320611.650.71329.580.763710.88
12STSF0.772411.290.747710.940.88049.710.708011.520.64328.230.734511.820.782611.88
13H-InceptionTime0.740811.390.719010.600.849610.240.683810.291.322712.420.722310.160.737812.14
14LiteTIME0.734112.080.710411.260.839411.330.668011.551.477612.600.711310.390.733611.97
15PatchMTSC0.742812.770.689713.540.826112.450.653313.180.76559.190.685212.850.735212.61
16ConvTran0.746212.890.710213.060.859210.860.676712.610.81909.290.715912.490.734513.23
17Catch220.747512.930.718113.380.869710.590.692213.420.714710.650.724013.270.737413.28
18STC0.754513.630.717213.820.874411.080.694013.660.63919.750.718513.850.753713.56
19TSF0.751513.630.723613.200.874011.420.688313.580.725210.370.709314.210.760613.44
20TDE0.726214.210.681314.380.837412.070.638213.880.886911.150.671413.460.734412.65
21Summary0.685816.120.657415.890.826814.880.623015.890.912312.960.657415.810.684416.29
221NN-DTW0.671217.820.645417.060.719720.450.613617.0711.850621.990.652115.800.663617.88
23Dummy0.364520.950.302921.640.500022.050.150721.321.406715.670.285519.700.381617.59

Average score and average rank over the 52 datasets with results for every estimator on every metric. Best in each column is highlighted. Metrics marked ↓ are better when lower.

Missing results

  • HC2 — AustraliaRainfall_disc (Time limit); STEW, Tiselac, USCActivity (cancelled before completion)
  • MRHydra — AustraliaRainfall_disc (OOM at 128GB); PenDigits (ValueError: n_timepoints must be >= 9, but found 8); Tiselac (LAPACK integer overflow in the RidgeClassifierCV SVD (aeon issue 3738))
  • RDST — AustraliaRainfall_disc, Tiselac (LAPACK integer overflow in the RidgeClassifierCV SVD (aeon issue 3738)); USCActivity (OOM at 64GB)
  • FreshPRINCE — FaceDetection, FordChallenge, Skoda, Tiselac (OOM at 128GB)
  • ROCKET — AustraliaRainfall_disc (LAPACK integer overflow in the RidgeClassifierCV SVD (aeon issue 3738))
  • STSF — AustraliaRainfall_disc, PenDigits (not recorded)
  • LiteTIME — BIDMC32HR_disc, BIDMC32SpO2_disc, USCActivity (not recorded)
  • PatchMTSC — EmoPain (ValueError: input collection has too little variation (std <= 1e-07)); PenDigits (ValueError: patch_len exceeds the number of timepoints)
  • ConvTran — Alzheimers, EigenWorms, PhotoStimulation (CUDA out of memory); EmoPain (ValueError: input collection has too little variation (std <= 1e-07))
  • TSF — AustraliaRainfall_disc (not recorded)
  • TDE — AustraliaRainfall_disc, Tiselac, USCActivity (Time limit); STEW (cancelled before completion)
  • Summary — AustraliaRainfall_disc (not recorded)
  • 1NN-DTW — BIDMC32HR_disc (Time limit); BIDMC32SpO2_disc (not recorded)

Scoring uses the 52 datasets every estimator completed, so a dataset any one of them is missing is left out for all. Reasons are from the job logs of these runs.

Reproducing this page

from aeon.datasets.tsc_datasets import multiverse_core
+
Multiverse-core leaderboard
24 estimators on 52 datasets · 7 metrics · ordered by average accuracy rank · built 2026-08-31
#EstimatorAccuracyBalanced accuracyAUROCF1Log loss ↓SensitivitySpecificity
ScoreRankScoreRankScoreRankScoreRankScoreRankScoreRankScoreRank
1HC20.79097.830.75188.460.89906.120.72738.260.53836.270.74598.710.79437.44
2MRHydra0.78378.340.75648.040.810516.800.73167.907.797419.760.76428.270.77579.16
3RDST0.77349.190.73339.760.791217.380.699110.118.166719.990.710910.790.78748.64
4RIST0.77209.740.739710.590.87488.110.714710.320.62188.440.740810.750.765510.76
5FreshPRINCE0.774310.000.748710.410.87458.010.721110.300.60076.330.741410.800.777010.87
6DrCIF0.774710.040.742910.440.88138.440.717310.800.64849.370.739711.400.770810.84
7CIF0.778110.090.747110.350.89088.260.721210.370.64309.350.744111.270.775310.21
8QUANT0.772010.470.746210.420.88317.610.718910.490.61757.520.752110.660.758111.44
9Arsenal0.768010.690.732110.530.845713.170.702410.653.863116.880.725710.800.773210.55
10ROCKET0.769010.850.732610.610.792518.120.701911.098.324920.750.720011.680.776411.05
11LITETime-MV0.750611.250.72999.850.851310.120.68209.961.320612.190.71329.840.763711.17
12STSF0.772411.460.747711.080.88049.880.708011.670.64328.420.734512.060.782612.07
13H-InceptionTime0.740811.680.719010.850.849610.540.683810.571.322712.980.722310.410.737812.53
14LiteTIME0.734112.390.710411.540.839411.620.668011.811.477613.150.711310.620.733612.31
15PatchMTSC0.742812.980.689713.810.826112.690.653313.480.76559.420.685213.120.735212.78
16ConvTran0.746213.090.710213.250.859211.010.676712.800.81909.460.715912.740.734513.42
17Catch220.747513.150.718113.630.869710.800.692213.700.714710.880.724013.570.737413.60
18TSF0.751513.890.723613.410.874011.630.688313.830.725210.650.709314.530.760613.66
19STC0.754513.940.717214.120.874411.350.694014.000.63919.980.718514.180.753713.90
20TDE0.726214.590.681314.740.837412.450.638214.220.886911.600.671413.810.734413.01
21Summary0.685816.530.657416.280.826815.320.623016.270.912313.420.657416.250.684416.75
22TimesNet0.664317.710.628717.760.810516.380.590417.681.005813.830.630517.040.659317.14
231NN-DTW0.671218.310.645417.530.719721.200.613617.4911.850622.950.652116.200.663618.38
24Dummy0.364521.790.302922.550.500022.980.150722.241.406716.400.285520.500.381618.31
Average score and average rank over the 52 datasets with results for every estimator on every metric. Best in each column is highlighted. Metrics marked ↓ are better when lower.
Missing results
  • HC2 — AustraliaRainfall_disc (Time limit); STEW, Tiselac, USCActivity (cancelled before completion)
  • MRHydra — AustraliaRainfall_disc (OOM at 128GB); PenDigits (ValueError: n_timepoints must be >= 9, but found 8); Tiselac (LAPACK integer overflow in the RidgeClassifierCV SVD (aeon issue 3738))
  • RDST — AustraliaRainfall_disc, Tiselac (LAPACK integer overflow in the RidgeClassifierCV SVD (aeon issue 3738)); USCActivity (OOM at 64GB)
  • FreshPRINCE — FaceDetection, FordChallenge, Skoda, Tiselac (OOM at 128GB)
  • ROCKET — AustraliaRainfall_disc (LAPACK integer overflow in the RidgeClassifierCV SVD (aeon issue 3738))
  • STSF — AustraliaRainfall_disc, PenDigits (not recorded)
  • LiteTIME — BIDMC32HR_disc, BIDMC32SpO2_disc, USCActivity (not recorded)
  • PatchMTSC — EmoPain (ValueError: input collection has too little variation (std <= 1e-07)); PenDigits (ValueError: patch_len exceeds the number of timepoints)
  • ConvTran — Alzheimers, EigenWorms, PhotoStimulation (CUDA out of memory); EmoPain (ValueError: input collection has too little variation (std <= 1e-07))
  • TSF — AustraliaRainfall_disc (not recorded)
  • TDE — AustraliaRainfall_disc, Tiselac, USCActivity (Time limit); STEW (cancelled before completion)
  • Summary — AustraliaRainfall_disc (not recorded)
  • TimesNet — EmoPain (ValueError: input collection has too little variation (std <= 1e-07))
  • 1NN-DTW — BIDMC32HR_disc (Time limit); BIDMC32SpO2_disc (not recorded)
Scoring uses the 52 datasets every estimator completed, so a dataset any one of them is missing is left out for all. Reasons are from the job logs of these runs.
Reproducing this page
from aeon.datasets.tsc_datasets import multiverse_core
 from multiverse.experiments.tables import leaderboard
 
 leaderboard(
     datasets=sorted(multiverse_core),
-    estimators=["HC2", "MRHydra", "RDST", "RIST", "DrCIF", "FreshPRINCE", "CIF", "QUANT", "Arsenal", "ROCKET", "LITETime-MV", "STSF", "H-InceptionTime", "LiteTIME", "PatchMTSC", "ConvTran", "Catch22", "STC", "TSF", "TDE", "Summary", "1NN-DTW", "Dummy"],
+    estimators=["HC2", "MRHydra", "RDST", "RIST", "FreshPRINCE", "DrCIF", "CIF", "QUANT", "Arsenal", "ROCKET", "LITETime-MV", "STSF", "H-InceptionTime", "LiteTIME", "PatchMTSC", "ConvTran", "Catch22", "TSF", "STC", "TDE", "Summary", "TimesNet", "1NN-DTW", "Dummy"],
     metrics=["accuracy", "balacc", "auroc", "f1", "logloss", "sensitivity", "specificity"],
     sort_by="accuracy",
 )
Or python -m multiverse.experiments.tables to rebuild it with the defaults.