CI: added duration for each ZTS group as weight

[tiehexue]

Motivation and Context

As discussed in #18997 , we could make the two VMs in ZTS more balanced by the actual time cost.

Description

The methodology is very simple: 1) added a new key "duration" for each test group; 2) set duration value by a real run (ubuntu22 is picked in this PR); 3) split tags based the test group duration, sorted by duration desc first, and each vm picks one by one.

We also could let one vm pick one, and the second vm pick following several ones to best match the first one. But consider that we have "a lot" test group, the one-by-one is quite good. The Lustre build and in-kernel build is not considered too for simplicity.

Now the two vm differs in less than 20 minutes, e.g.:

run | vm1 ZTS time | vm2 ZTS time | ratio | result
fedora43 | 2:46:03 | 2:49:54 | 1.02 | both exit 0/1*, full run
ubuntu26 | 2:51:39 | 3:00:50 | 1.05 | both exit 0/1*, full run

How Has This Been Tested?

CI of zfs fork

Types of Changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Performance enhancement (non-breaking change which improves efficiency)
• Code cleanup (non-breaking change which makes code smaller or more readable)
• Quality assurance (non-breaking change which makes the code more robust against bugs)
• Breaking change (fix or feature that would cause existing functionality to change)
• Library ABI change (libzfs, libzfs_core, libnvpair and libzfsbootenv)
• Documentation (a change to man pages or other documentation)

Checklist

• My code follows the OpenZFS code style requirements.
• I have updated the documentation accordingly.
• I have read the contributing document.
• I have added tests to cover my changes.
• I have run the ZFS Test Suite with this change applied.
• All commit messages are properly formatted and contain Signed-off-by.

[tiehexue]

According to latest CI results: the most closed between VMs is ubuntu24, differs in seconds, and biggest bias is ubuntu22, around 30 minutes. Ubuntu22 is the one that one of the VMs timeout in 270 minutes, and now it is 215 minutes.

[tonyhutter]

@tiehexue I was working on something very similar to this PR but you beat me to to it: #19084.

I'm seeing the same results as you where most of the tests are now really balanced except for Alma 8 and Ubuntu 22, which finish ~30min between vm1 and vm2.

Some differences with my PR:

• Test times are stored in zts-tests.sh rather than in the runfiles. The benefit there is that we can update the test time data out of band with the addition of new ZTS tests. That means developers won't need to worry about updating a duration field whenever they add a test, or modify an old test to take longer/shorter. We can just update the test times database when we want (like every year or so). Separating the duration field out of the runfiles also minimizes merge conflicts when backporting tests to point releases. All tests will still work even if the test database is stale.
• My PR includes a helper script to scrape the test times from a ZTS results archive.

[tiehexue]

Hi @tonyhutter , I am pretty fine for your PR which is very professional. Either of this two PR merge, should close the other.

• That means developers won't need to worry about updating a duration field whenever they add a test

Actually, this is the "intention" in my PR, we should care about tests we added. However, your points are very sound.

And one more thing, it will be good to add a summary in "Test Summary" section to show each VM's timing explicitly.

[behlendorf]

@tiehexue thanks for posting this so quickly! After reading through both versions I'd like to move forward with #19084, so I'm going to close this PR.

And one more thing, it will be good to add a summary in "Test Summary" section to show each VM's timing explicitly.

Good idea, making it easy see the VM run times would be helpful. It would be nice to have the VM exit status there as well.