[taimoorsohail]

This PR acts as an attempt to implement a test suite for the cosima-recipes. The test suite runs all the notebooks on gadi, and reports if any do not run/break for any reason. This then blocks further merging of any PR with new notebooks until the tests pass.

[navidcy]

How do we see the output/log of the tests?

Atm, when I click on the GitHub action I see:

[navidcy]

Also, what NCI project is charged for this?

[taimoorsohail]

Great questions @navidcy -- I am currently just attempting to get the runner active on v46. I hope we can then finalise these details; including what username and project the runner is active on (this will involve minor tweaks to the runner).

[taimoorsohail]

cc @julia-neme @hrsdawson @claireyung @adele-morrison

[taimoorsohail]

OK, the tests are working and all notebooks are failing due to an issue with intake.

[julia-neme]

Woow this is so good @taimoorsohail ! Dang about intake, was there a specific common error or multiple issues?

[taimoorsohail]

I am getting exactly the same error as here: https://forum.access-hive.org.au/t/invalid-utf-8-sequence-error/5454 (thanks @julia-neme for pointing it out!)

But the runner is using analysis3-26.05, so it appears this issue has regressed @charles-turner-1 ?

[charles-turner-1]

That shouldn't be possible... I'll take a look.

Edit: Because the environments are frozen, and were working correctly in May. Something strange must be up.

I also fixed up some back compatibility guarantees so that there should be no possibility of that happening again around the time of that fix. Clearly there's a hole I didn't see when I did that...

[taimoorsohail]

Thanks @charles-turner-1 -- very live possibility my runner is bugged so no pressure; I have pinned it to 25.09 for now just to see if it progresses

EDIT: Pinning to 3-25.09 fixed the bug, suggesting there is an issue with conda/analysis3?

[rbeucher]

It is very similar to what I have been working on, so let us make sure we are not duplicating effort. I am happy to coordinate so we can build on each other's work rather than doing the same thing twice.

See #315

https://github.com/ACCESS-NRI/COSIMA-recipes-workflow
https://access-nri.github.io/COSIMA-recipes-workflow/

My idea was to test to completion across the supported analysis3 environment on xp65.

[charles-turner-1]

Sorry @taimoorsohail, didn't get a chance to look at this earlier - which recipe was triggering the error? Cheers!

[taimoorsohail]

@charles-turner-1 it was every recipe.

[review-notebook-app]

Check out this pull request on 

See visual diffs & provide feedback on Jupyter Notebooks.

---

Powered by ReviewNB

[taimoorsohail]

Current errors picked up by the test:

Notebook 6/38: FAILED 01-Cooking-Tutorials/02-Advanced/CFxarray_and_Pint.ipynb
Cell 47: KeyError: "Receive multiple variables for key 'longitude': ['ULON', 'TLON']. Expected only one. Please pass a list ['longitude'] instead to get all variables matching 'longitude'."
Traceback excerpt:
--> 856 check_results(names, k)
    857 successful[k] = bool(names)
    858 coords.extend(names)
File /g/data/xp65/public/apps/med_conda/envs/analysis3-25.09/lib/python3.11/site-packages/cf_xarray/accessor.py:826, in _getitem.<locals>.check_results(names, key)
    824 def check_results(names, key):
    825     if scalar_key and len(names) > 1:
--> 826         raise KeyError(
    827             f"Receive multiple variables for key {key!r}: {names}. "
    828             f"Expected only one. Please pass a list [{key!r}] "
    829             f"instead to get all variables matching {key!r}."
    830         )
KeyError: "Receive multiple variables for key 'longitude': ['ULON', 'TLON']. Expected only one. Please pass a list ['longitude'] instead to get all variables matching 'longitude'."

[charles-turner-1]

Sorry to potentially derail your main thread here Taimoor - I've retested against some extra environments and I can't reproduce the error.

Josh did run a catalogue rebuild yesterday. The build is supposed to be atomic (ie. 0 downtime between old and new versions being live), but there is a potential gap of up to a few seconds where files get rewritten and directories swapped. In theory if the python kernel read those files during that transition, then the utf8 decode errors could crop up.

It still doesn't really explain why you see the error for more than one notebook though...

At this point, I'm gonna say crossing our fingers and hoping that it was a strange catalog build related oddity is the best route. If it happens again I'll have a deep dive & see what I can figure out.

---

Minor aside, but it's also possible that a utf8 decode error happened as a result of some mis-parsing in the workflow that turns the notebooks into the scripts. There's a lot going on there though - if it seems stable and is working now, I'd rather avoid turning over too many stones - there are lots of way parsing a notebook into a text file to execute could subtly go wrong if the filesystem did anything weird.

[taimoorsohail]

No worries, looks like @rbeucher is building a better version of this, so I am using this PR as a quick stop gap before his becomes online. Just so we can flag and fix notebooks for the JOSE submission.

[rbeucher]

Quick update on the related ACCESS-NRI CI workflow: the updated workflow and dashboard are now available for feedback. The workflow is configured to test all 38 recipe notebooks weekly on Gadi against the six newest versioned analysis3 environments, updating the selection as new versions appear. We have completed an initial full run across all six versions, and the dashboard lets you switch versions to see each notebook’s result.

I’d welcome feedback on the tests, results, or dashboard, and I’m happy to coordinate with the work in this PR.

[taimoorsohail]

@rbeucher thanks! This PR was a quick implementation, far inferior to the dashboard (which looks awesome by the way!).

I'll close this PR in favour of the dashboard now.