[DRAFT] CI: Split up tests evenly on runner VMs

[tonyhutter]

Motivation and Context

Faster CI runs

Description

Our CI spawns two VMs on each github runner, and runs half the test suite on each. It naively splits up the tests by count, and doesn't take into account how long the individual tests groups take to run. This leads to one VM finishing the test suite before the other. For example, one recent run on Fedora 44:

vm1 03:16:28
vm2 02:50:54

This commit attempts to balance the tests on the VMs by runtime. It does this by adding a test completion time database to zfs-tests.sh which is use to portion out the test groups equally. The database is just a big associative array that is generated by the new make-testdb.sh helper script. Just point make-testdb.sh at a test results tarball and it will generate the new test times database.

Note that the database will get stale over time and will periodically have to be re-generated (maybe every couple years). It doesn't hurt anything if it is stale though. If a new test is added that is not in the database, zfs-test.sh is smart enough to assign it a default value of whatever the average test runtime is, so it still stays relatively balanced.

How Has This Been Tested?

CI showed closer runtimes between vm1 & vm2:

OS	vm1	vm2
alma9	03:09:52	03:03:48
alma10	03:01:23	03:01:38
centos-stream10	02:41:08	02:32:17
debian12	02:46:44	02:43:03
debian13	02:48:31	02:40:51
ubuntu24	02:52:47	02:54:36
ubuntu26	03:18:30	03:12:39
fedora43	03:04:54	02:55:51
fedora44	03:09:36	02:58:53

ubuntu22 & alma8 are still 30+min apart though, so marking this as draft. I still need to tune this to pick the right test results to use for the test time database. It's currently using data from Fedora 43 results.

Types of Changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[tonyhutter]

I'm thinking now that I want to get the test times from a single VM running locally. That will isolate the test times from the noisy neighbor problem of running two VMs at once.