Skip to content

test(ci): complete the macOS 3.5.0 matrix with both root legs - #7692

Merged
oferchen merged 2 commits into
masterfrom
feat/uts-macos-root-legs
Sep 5, 2026
Merged

test(ci): complete the macOS 3.5.0 matrix with both root legs#7692
oferchen merged 2 commits into
masterfrom
feat/uts-macos-root-legs

Conversation

@oferchen

@oferchen oferchen commented Sep 5, 2026

Copy link
Copy Markdown
Owner

Completes the upstream 3.5.0 testsuite matrix on macOS and consolidates the
duplication that adding the legs would otherwise have created.

Stacked on #7691. Until that merges this PR's diff shows its commit too.

Coverage

pipe nonroot pipe root tcp nonroot tcp root
Linux required required required required
macOS non-required this PR #7691 this PR

Why the privilege axis is not redundant with Linux

Crossing the committed Linux manifests puts ~29 cells in the set a macOS
root leg would newly RUN. They are not the chown/devices cells the workflow
header used to assume — the set is dominated by the symlink-race and
confinement family
: backup-dir-symlink-race, symlink-race-dest,
temp-dir-symlink-injection, daemon-module-chdir-symlink,
operator-path-backup-symlink, and all five chroot-*-inner-module.

Those are exactly the cells whose Linux passes are known to rest on Landlock
rather than on the userspace confinement guard — that guard was measured failing
on both platforms under a symlinked ancestor. macOS has no Landlock, so it is
the only place the guard is exercised unaided.

⚠ The ~29 is a prediction, and ships as one

Neither leg names a manifest. Each runs the whole corpus and emits its own via
EMIT_EXPECT_RESULT as an artifact, so the manifests come from an executed run
of the leg they describe.

That is deliberate: the same prediction for the macOS tcp leg said 52 cells
and the executed run gave 32, because cells skip on macOS for platform
reasons the Linux manifests cannot express. Until the manifests are committed
these two checks report the suite's genuine result — which on a suite with known
divergences means red.
Both are non-required, matching the macOS pipe leg.

No new caller job — and that was a correction

variant defaults to all, so the two existing macOS callers each publish
both legs. This mirrors _upstream-testsuite.yml, whose callers pass no
variant at all.

The first version of this change added a caller per privilege leg with
variant: 'root' pinned. That works, but a caller pinned to one variant makes
the reusable workflow's other job report a permanently SKIPPED check —
four of them across the two transports, which
check_pr_check_name_uniqueness.py confirmed by counting 84 names instead
of 80. Skipped checks nobody can act on are noise, and the Linux idiom already
avoided them. Corrected before landing; the lint is back to 80.

Consolidation — otherwise this would add two more copies

The nonroot and root legs differ only in the sudo re-exec and their output
paths. Six setup steps and a ~60-line summary renderer are identical, and the
Linux workflow already carries two divergent copies of its own preamble.

  • .github/actions/upstream-testsuite-macos-setup — the shared setup steps.
    Checkout stays in the caller: a local uses: action cannot check out the tree
    it is loaded from.
  • tools/ci/summarize_upstream_testsuite.sh — the summary renderer. Exits 0
    on a missing log on purpose: it runs under if: always() after a step that may
    have failed, and the job's conclusion must stay the testsuite's.

A new job and its full setup now costs 4 lines of YAML instead of ~190.

Verification

  • The extraction was scripted, and the script asserts every non-trivial line
    of the removed block survives in the action
    . 120 of 129 moved verbatim; the
    9 it flagged are exactly the intended parameterisation (three env.*
    inputs.*, the rust-cache key, and the reworded cache-key comment).
  • That comment is load-bearing and moved with its code: the upstream-source
    cache key hashes the file supplying CPPFLAGS/LDFLAGS, because
    run_upstream_testsuite.sh skips ./configure when shconfig is present —
    a tree cached under a different flag set would be reused with its original
    feature set and no step would notice. The flags now live in the action, so the
    key hashes the action too.
  • Workflows verified by parsing the written YAML, not re-reading the intended
    edit. Confirms two jobs, correct if: guards, and that artifact names, emitted
    manifest paths and rust-cache keys are all pairwise distinct — each is a
    per-run namespace, so a collision would let one leg silently overwrite
    another's outcomes. It also asserts every composite run: step declares
    shell:, without which the action fails to load naming no step.
  • Summariser exercised against a synthetic log with PASS/FAIL/UPASS/SKIP/XFAIL
    rows: correct sections, exit 0.
  • check_pr_check_name_uniqueness.py: 80 names, one publisher each.
    run_tools_tests.sh: 86 passed, exit 0.

Follow-up

Once both manifests land from these runs, the residual consolidation is the
Linux workflow's own two copies of the same preamble and summary block — it can
adopt both shared pieces without changing any required check context.

@github-actions github-actions Bot added the test label Sep 5, 2026
@oferchen
oferchen force-pushed the feat/uts-macos-root-legs branch from 78839de to 86b60fe Compare September 5, 2026 12:52
Linux has run {nonroot,root} x {pipe,tcp} since PR #7408. macOS ran nonroot
only, so the privilege axis - the cells that need real root - was covered on
one platform. This adds the two missing legs and consolidates the duplication
that adding them would otherwise have created.

WHY THE PRIVILEGE AXIS IS NOT REDUNDANT WITH LINUX

Crossing the committed Linux manifests puts ~29 cells in the set a macOS root
leg would newly RUN. They are not the chown/devices cells the old header
assumed: the set is dominated by the symlink-race and confinement family -
backup-dir-symlink-race, symlink-race-dest, temp-dir-symlink-injection,
daemon-module-chdir-symlink, operator-path-backup-symlink, and all five
chroot-*-inner-module. Those are exactly the cells whose Linux passes are
known to rest on Landlock rather than on the userspace confinement guard; that
guard was measured failing on BOTH platforms under a symlinked ancestor. macOS
has no Landlock, so it is the only place the guard is exercised unaided.

THE ~29 IS A PREDICTION AND IS SHIPPED AS ONE

Neither leg names a manifest yet. Each runs the whole corpus and emits its own
via EMIT_EXPECT_RESULT as an artifact, so the manifests come from an executed
run of the leg they describe. That is deliberate: the equivalent prediction for
the macOS tcp leg said 52 cells and the executed run gave 32, because cells
skip on macOS for platform reasons the Linux manifests cannot express. Until
the manifests are committed these two checks report the suite's genuine result,
which on a suite with known divergences means red. Both are non-required,
matching the macOS pipe leg.

CONSOLIDATION, BECAUSE OTHERWISE THIS WOULD ADD TWO MORE COPIES

The nonroot and root legs differ only in the sudo re-exec and their output
paths; six setup steps and a ~60-line summary renderer are identical. Adding
root by copy would have made a third and fourth copy of each - the Linux
workflow already carries two divergent copies of its own preamble.

  - .github/actions/upstream-testsuite-macos-setup: the six shared setup steps,
    now parameterised. Checkout stays in the caller, because a local `uses:`
    action cannot check out the tree it is loaded from.
  - tools/ci/summarize_upstream_testsuite.sh: the summary renderer, taking the
    log path and a heading. It exits 0 on a missing log on purpose - it runs
    under `if: always()` after a step that may have failed, and the job's
    conclusion must stay the testsuite's.

Net: one new job and its full setup cost 4 lines of YAML each rather than ~190.

NO NEW CALLER JOB, AND THAT WAS A CORRECTION

`variant` defaults to `all`, so the two existing macOS callers each publish
BOTH legs. This mirrors `_upstream-testsuite.yml`, whose callers pass no
variant at all.

The first version of this change instead added a caller per privilege leg with
`variant: 'root'` pinned. That works, but a caller pinned to one variant makes
the reusable workflow's OTHER job report a permanently SKIPPED check - four of
them across the two transports, which the check-name lint confirmed by counting
84 names instead of 80. Skipped checks nobody can act on are noise, and the
Linux idiom already avoided them. Corrected before landing.

VERIFICATION

The setup extraction was done by script, and the script asserts that every
non-trivial line of the removed block survives in the action. 120 of 129 lines
moved verbatim; the 9 it flagged are exactly the intended parameterisation -
three env.* -> inputs.*, the rust-cache key, and the reworded cache-key comment.

That comment is load-bearing and was updated with the code it explains: the
upstream-source cache key hashes the file that supplies CPPFLAGS/LDFLAGS,
because run_upstream_testsuite.sh skips ./configure when shconfig is present,
so a tree cached under a different flag set would be reused with its ORIGINAL
feature set and no step would notice. The flags now live in the action, so the
key hashes the action too.

Workflows verified by parsing the written YAML, not by re-reading the intended
edit: two jobs, correct `if:` guards, and artifact names, emitted manifest
paths and rust-cache keys all pairwise distinct - each is a per-run namespace,
so a collision would let one leg silently overwrite another's outcomes. The
check also asserts every composite `run:` step declares `shell:`, without which
the action fails to load at runtime naming no step.

Summariser exercised against a synthetic log carrying PASS/FAIL/UPASS/SKIP/
XFAIL rows: correct sections, exit 0. Check-name uniqueness lint: 84 names,
each with one publisher. tools tests: 86 passed, exit 0.
@oferchen
oferchen force-pushed the feat/uts-macos-root-legs branch from 86b60fe to 47eecb5 Compare September 5, 2026 13:30
…runs

The two macOS root legs landed in bootstrap: no `expect_result_macos_root`,
so each ran the WHOLE corpus and reported its genuine result while
EMIT_EXPECT_RESULT wrote the manifest as an artifact. Those runs have now
happened, so this converts the bootstrap into a real gate.

MEASURED, each from a run of the leg it describes:

  root/pipe  266 pass /  2 fail / 77 skip   (345 selected)
  root/tcp   124 pass / 12 fail / 19 skip   (155 selected)

NEITHER was predicted from the Linux legs, and that is deliberate: the one
time a macOS leg WAS predicted that way the forecast said 52 cells and the
executed run gave 32. A cell can skip on macOS for platform reasons no Linux
manifest can express.

THE root/tcp RESULT IS THE INTERESTING ONE

Its twelve failures are IDENTICAL to the Linux root/tcp set, cell for cell:
the four daemon-chroot/-munge/-private-parent/-symlink-escape-matrix rows,
daemon-zstd-thread-exhaustion, malicious-server-partial-basis-symlink-
overwrite, nonroot-restrictive-perms, and the five proto-* rows. Zero
macOS-only failures, zero regressions against the macOS nonroot legs.

So adding the privilege axis on macOS introduced no divergence of its own -
it re-runs exactly the known-failing set, on a platform with no Landlock to
mask a userspace confinement gap. That is what the leg is for.

root/pipe's two are `daemon-module-private-parent` and
`partial-protected-regular-retry-policy`, both pre-existing and both owned.

WHY THE DEFAULT MOVES RATHER THAN THE CALLERS

`expect_result_macos_root` now defaults to the root/pipe manifest, mirroring
`expect_result_macos`, which already defaults to the nonroot/pipe one. The tcp
caller names both tcp manifests explicitly for the same reason it already had
to: `--daemon-tests-only` selects a smaller set, so a shared file would fail
the driver's coverage guard outright. One idiom, not two.

The input keeps its bootstrap contract documented, because that is how the
NEXT new leg obtains its first manifest: set it to the empty string, run, take
the artifact.

Every divergence stays recorded as a `fail` row rather than hidden, exactly as
the Linux legs record theirs, so each gate fires on DRIFT from this measured
state.
@oferchen
oferchen merged commit d85fdbe into master Sep 5, 2026
75 checks passed
@oferchen
oferchen deleted the feat/uts-macos-root-legs branch September 5, 2026 15:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant