Improve software-layer-script workflow

[casparvl]

Changes needed to make builds use everything from the cloned software-layer-scripts, instead of from the deployed stuff in software.eessi.io.

This is a starting point for https://gitlab.com/eessi/support/-/issues/217.

Split off from 2422804 which was tested in e.g. EESSI/software-layer#1351 (comment)

We may not want to merge this until we have sufficient checks (CI, bot) in place to ensure that tarballs can only be deployed if they were build from a merge commit of software-layer-scripts. That means we need:

• Software-layer CI to check if upstream software-layer-scripts commit was merged Clone specific software-layer-commit and implement CI to check merged status software-layer#1353
• Bot improvement to check if build was done using the latest bot/commit_sha (i.e. the same one as the CI check uses).

Edit 10-08-2026: actually, this is somewhat separate from the software-layer work to use a particular commit checksum. Making the build scripts use everything from the software-layer-scripts repository makes sense in any case: it prevents us from first having to deploy certain components before we're actually able to test them. The same applies to the modification in the order in which CI steps are executed, that's also to prevent us from first having to deploy EESSI-extend-easybuild.eb changes to production before all the CI steps are run on those changes.

[bedroge]

Just to summarize how this used to work and how it will work after merging this PR:

Previously, an updated hooks file was copied to the CVMFS repo (with the overlay) before the builds were started (by EESSI-install-software.sh), and this was only done if a PR diff file showed that the hooks file was modified by the PR.

In the new situation it will just use the version from the git clone of software-layer-scripts; the build script clones a specific commit of that repo. Additional CI will make sure that software-layer PR tarballs can only be deployed if that commit belongs to a merged PR.

One thing that maybe requires some attention is possible situations where a software layer PR modifies that software-layer-scripts commit, does the builds, and then undoes those changes regarding the used commit. A reviewer may not immediately spot that the builds were done with a different commit (the "Files changed" tab won't show it, you would have to look at individual commits or notice that there was a commit after the builds were done), but maybe the CI or bot itself could also catch that?

[casparvl]

Let's prove in the build logs that we actually use the eb_hooks.py from software-layer-scripts now instead of from the CVMFS repo.

[casparvl]

bot: build repo:eessi.io-2025.06-software instance:eessi-bot-mc-aws for:arch=x86_64/amd/zen2

[eessi-bot-aws]

New job on instance eessi-bot-mc-aws for repository eessi.io-2025.06-software
Building on: amd-zen2
Building for: x86_64/amd/zen2
Job dir: /project/def-users/SHARED/jobs/2026.08/pr_147/185351

date	job status	comment
Aug 10 14:20:33 UTC 2026	submitted	job id 185351 awaits release by job manager
Aug 10 14:21:05 UTC 2026	released	job awaits launch by Slurm scheduler
Aug 10 14:22:10 UTC 2026	running	job 185351 is running
Aug 10 14:29:38 UTC 2026	finished	😁 SUCCESS (click triangle for details) Details ✅ job output file slurm-185351.out ✅ no message matching FATAL: ✅ no message matching ERROR: ✅ no message matching FAILED: ✅ no message matching required modules missing: ✅ found message(s) matching No missing installations ✅ found message matching .tar.* created! Artefacts eessi-2025.06-software-linux-x86_64-amd-zen2-17863719890.tar.zst size: 0 MiB (101477 bytes) entries: 104 modules under 2025.06/software/linux/x86_64/amd/zen2/modules/all cowsay/3.04.lua EESSI-extend/2025.06-easybuild.lua software under 2025.06/software/linux/x86_64/amd/zen2/software cowsay/3.04 EESSI-extend/2025.06-easybuild reprod directories under 2025.06/software/linux/x86_64/amd/zen2/reprod cowsay/3.04/20260810_142619UTC other under 2025.06/software/linux/x86_64/amd/zen2 no other files in tarball
Aug 10 14:29:38 UTC 2026	test result	😁 SUCCESS (click triangle for details) ReFrame Summary [ OK ] (1/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025-foss-2024a-kokkos %scale=1_node /ade8cad7 @BotBuildTests:x86-64-zen2+default P: perf: 434.834 timesteps/s (r:0, l:None, u:None) [ OK ] (2/6) EESSI_LAMMPS_lj %device_type=cpu %module_name=LAMMPS/22Jul2025_update4-foss-2025b-kokkos %scale=1_node /e121eb9c @BotBuildTests:x86-64-zen2+default P: perf: 448.278 timesteps/s (r:0, l:None, u:None) [ OK ] (3/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_allreduce %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /e4bf9965 @BotBuildTests:x86-64-zen2+default P: latency: 1.3 us (r:0, l:None, u:None) [ OK ] (4/6) EESSI_OSU_coll %benchmark_info=mpi.collective.osu_alltoall %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node %device_type=cpu /3da4890b @BotBuildTests:x86-64-zen2+default P: latency: 2.08 us (r:0, l:None, u:None) [ OK ] (5/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_latency %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /3255009a @BotBuildTests:x86-64-zen2+default P: latency: 0.2 us (r:0, l:None, u:None) [ OK ] (6/6) EESSI_OSU_pt2pt_CPU %benchmark_info=mpi.pt2pt.osu_bw %module_name=OSU-Micro-Benchmarks/7.5-gompi-2025a %scale=1_node /59f4b331 @BotBuildTests:x86-64-zen2+default P: bandwidth: 7979.79 MB/s (r:0, l:None, u:None) [ PASSED ] Ran 6/6 test case(s) from 6 check(s) (0 failure(s), 0 skipped, 0 aborted) Details ✅ job output file slurm-185351.out ✅ no message matching ERROR: ✅ no message matching [\s*FAILED\s*].*Ran .* test case

[casparvl]

Works like a charm:

$ cat /project/def-users/SHARED/jobs/2026.08/pr_147/185351/slurm-185351.out | grep "hooks    "
hooks                                    (E) = /project/60006/SHARED/jobs/2026.08/pr_147/event_a217a4d0-94c6-11f1-9e45-8e3c2afd5d62/run_000/x86_64/amd/zen2/eessi.io-2025.06-software/eb_hooks.py
hooks                                    (E) = /project/60006/SHARED/jobs/2026.08/pr_147/event_a217a4d0-94c6-11f1-9e45-8e3c2afd5d62/run_000/x86_64/amd/zen2/eessi.io-2025.06-software/eb_hooks.py
hooks                                    (E) = /project/60006/SHARED/jobs/2026.08/pr_147/event_a217a4d0-94c6-11f1-9e45-8e3c2afd5d62/run_000/x86_64/amd/zen2/eessi.io-2025.06-software/eb_hooks.py

[casparvl]

CI check looks fine now:

Checking EESSI-extend EasyBuild hooks override
-- Using /tmp/$USER as a temporary working directory for installations, you can
override this by setting the environment variable WORKING_DIR and reloading the
module (e.g., /dev/shm is a common option) 
Configuring for use of EESSI_USER_INSTALL under /home/runner/eessi
-- To create installations for EESSI, you _must_ have write permissions to
/home/runner/eessi/versions/2023.06/software/linux/x86_64/amd/zen3 
-- You may wish to configure a sources directory for EasyBuild (for example,
via setting the environment variable EASYBUILD_SOURCEPATH) to allow you to
reuse existing sources for packages. 
EASYBUILD_HOOKS is correctly set to '/home/runner/work/software-layer-scripts/software-layer-scripts/eb_hooks.py'
✅ No disallowed environment variables with prefix 'EASYBUILD_' found.

The hooks override works, the check works. Only thing left is to actually deploy this new version - which is why the CI is still failing at this time.