Skip to content

TL/UCP: add onesided a2a peer ordering and fragmentation - #1345

Draft
wfaderhold21 wants to merge 1 commit into
openucx:masterfrom
wfaderhold21:topic/onesided-a2a-rank-ordering
Draft

wfaderhold21 wants to merge 1 commit into
openucx:masterfrom
wfaderhold21:topic/onesided-a2a-rank-ordering

Conversation

@wfaderhold21

Copy link
Copy Markdown
Collaborator

What

Adds tuning knobs to onesided alltoall (peer ordering and message fragmentation) and schedules SHM and remote communication in the algorithm.

Why ?

In the current algorithm, performance tends to drop significantly when the number of tokens calculated at init time reaches 1 or lower. This changes the order and rate of message delivery to move performance closer to SOL.

How ?

  1. peer ordering: strided, interleaved, and a full (mixed) ordering
  2. fragmentation: fragmenting the message to reduce required buffering at the switch and endpoint.

Measured performance for messages >= 128 kB shows up to 15% improved bandwidth on IB (Gaia) and 8% improved bandwidth on RoCE.

@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — Lint (codestyle) · commit b20948dd

TL;DR: The codestyle check failed because the commit title TL/UCP: add onesided a2a peer ordering and fragmentation is 56 characters, exceeding the 50-character limit; shorten the commit title to ≤50 chars.

Full analysis

Summary: The "codestyle" job's commit-title lint step exited 1 because the PR's commit message title is too long.

Root cause: The commit-title validation in .github/workflows/codestyle.yaml rejects non-merge titles longer than 50 characters. Commit b20948dd's title TL/UCP: add onesided a2a peer ordering and fragmentation is 56 characters, so the check printed Commit title is too long: 56 / Bad commit title and set exit 1. The header prefix (TL/UCP:) is valid and there's no trailing dot — length is the sole violation.

Implicated commit: [REDACTED:Hex High Entropy String] — "TL/UCP: add onesided a2a peer ordering and fragmentation" (PR #1345, branch topic/onesided-a2a-rank-ordering)

File: .github/workflows/codestyle.yaml (the check_title function; length check if [ ${#msg} -gt 50 ])

Suggested fix: Reword the commit title to 50 characters or fewer while keeping a valid header prefix, then amend/force-push. Example (48 chars): TL/UCP: add onesided a2a ordering + fragmenting — or TL/UCP: onesided a2a peer ordering, fragmenting (47 chars). Any concise variant ≤50 chars that starts with an allowed header (e.g. TL/UCP:) and has no trailing period will pass. If this PR has multiple commits, ensure each commit title also complies.

Related: none

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 31ae8b6b-00e5-4e40-9753-5d0bdd14abe4 in the triage console for the audit trail.

@wfaderhold21
wfaderhold21 force-pushed the topic/onesided-a2a-rank-ordering branch from b20948d to cc211ea Compare August 31, 2026 18:06
TEST/GTEST: add onesided alltoall ordering test

- Fix single_onesided tune to @onesided (@1 selected bruck after the
  bruck algorithm was inserted into the enum)
- Add test_alltoall_onesided.ordering covering seq/stride/ilv/full
  peer orders crossed with fragmentation (nfrags 1/4)
- Give the onesided gtest context a simulated multi-node topology so
  ilv/full orders and the mixed GET/PUT progress path see a real
  local/remote split

TL/UCP: remove ilv onesided a2a peer ordering

- The 2026-09-10 pareto sweep (job 11769, 256 ranks) showed ilv
  dominated by both full and stride on bandwidth and congestion at
  every repeat and percent_bw: 82.17-82.53 GiB/s vs 84.65-85.89 at
  4 MiB/peer, ~6% more CNP/s
- Drop UCC_TL_UCP_ALLTOALL_ONESIDED_ORDER_ILV from the enum and the
  ALLTOALL_ONESIDED_ORDER config (now seq, stride, or full)
- Simplify the mixed GET/PUT progress selection in
  ucc_tl_ucp_alltoall_onesided_init: AUTO + full alone triggers the
  mixed path (ilv shared the same local/remote split layout, so no
  dedicated branch existed in build_peer_order)
- Shrink test_alltoall_onesided.ordering to seq/stride/full
  (6 cases); the ordering gtest passes 150/150 on thor
- Rejecting ORDER=ilv now fails cleanly at config parse time

No behavioral change to the remaining seq/stride/full layouts.

TL/UCP: require split peer layout for mixed under seq/stride

- alltoall_onesided_build_peer_order now takes the selected alg. The
  mixed GET/PUT progress path classifies local vs. remote by position
  in the order (the first/last n_local slots), so it requires the
  local/remote split layout. seq/stride with the mixed alg previously
  fell into the plain ring-walk layout, which leaves n_local = 0; the
  mixed progress path then treated every peer as remote and issued all
  ops as PUT, so the requested mixed (local-GET/remote-PUT) behavior
  silently degraded to all-PUT. They now get the split layout, which
  keeps the requested walk order within each class.
- Move the pos/is_local/is_local_peer locals in mixed_progress out of
  the loops, align a local in finalize, and drop the obsolete
  fragmentation-heuristic comment in _init. No change to the
  seq/stride/full layouts under pure PUT/GET.

TEST/GTEST: seed simulated proc info from local probe

- set_simulated_proc_info now starts from ucc_local_proc before
  overriding the topology coordinates. ucc_context_create_proc_info
  copies the whole struct into the context id (and allgathers it to
  every rank), and topology construction reads cpu_vendor/cpu_model,
  so leaving the remaining fields indeterminate was undefined. Only
  the topology coordinates are now overridden.

TEST/GTEST: add onesided alltoall mixed-alg ordering test

- Add test_alltoall_onesided_mixed.ordering: ALG=mixed crossed with
  ORDER in {stride, full} and NFRAGS in {1, 4}. The auto tune in the
  existing ordering test only reaches the mixed GET/PUT progress path
  via `full` (seq/stride route to GET), so the explicit mixed path under
  the non-full orders was untested. That is the path that depends on
  build_peer_order's local/remote split requirement, so it is the one
  that needs a permanent case. Seq is omitted: it is the default order
  and is covered by the auto path.
@svcnbu-swx-hpcx

Copy link
Copy Markdown
Collaborator

🤖 CI Triage Agent — Lint (codestyle) · commit 278b27b0

TL;DR: The codestyle lint failed on the commit-title check, not on code: four commits in PR #1345 have titles longer than 50 characters and/or use the unsupported TEST/GTEST: prefix. Fix by rewording those commits (interactive rebase) to ≤50 chars with an allowed prefix such as TEST: or TL/UCP:.

Full analysis

Summary: The Check commit title step of the Lint (codestyle) workflow exited 1 after flagging 4 of 6 commits in the PR range as "Bad commit title".

Root cause: .github/workflows/codestyle.yaml enforces two rules per commit: title length ≤ 50 chars (unless it starts with Merge) and a header matching ^((H1)|(H2))+: \w. From the log:

  • TEST/GTEST: add onesided alltoall mixed-alg ordering test — 57 chars (too long)
  • TEST/GTEST: seed simulated proc info from local probe — 53 chars (too long)
  • TL/UCP: require split peer layout for mixed under seq/stride — 60 chars (too long)
  • TEST/GTEST: add onesided alltoall ordering test — "Wrong header": TEST is in H1 and TL/, CL/, MC/, EC/ are in H2, but GTEST is in neither list, so the TEST/GTEST: compound prefix does not match the regex. (The three over-length TEST/GTEST: commits would also fail this header rule; they just short-circuit on length first.)

The two commits using TL/UCP: with ≤50-char titles passed, confirming the checker itself is behaving as designed — this is a commit-message hygiene failure in the branch, not a product/code defect.

Implicated commit: [REDACTED:Hex High Entropy String] (TEST/GTEST: add onesided alltoall mixed-alg ordering test) plus 3 earlier commits on topic/onesided-a2a-rank-ordering

File: .github/workflows/codestyle.yaml:39-53 (checker); offending commit titles on branch topic/onesided-a2a-rank-ordering

Suggested fix: Reword the four commits with git rebase -i remotes/origin/master (mark each as reword) and force-push. Suggested replacements, all ≤50 chars with allowed prefixes:

  • TEST: add onesided a2a mixed-alg order test
  • TEST: seed sim proc info from local probe
  • TL/UCP: split peer layout for mixed a2a
  • TEST: add onesided a2a ordering test

Note specifically that TEST/GTEST: is not an accepted header — use plain TEST:. If the project wants to allow gtest-scoped prefixes, that's a separate change: add GTEST (and TEST/) to the H1/H2 alternation lists in .github/workflows/codestyle.yaml.

Related: PR #1345 (#1345); sibling PR #1344 touches the same onesided-alltoall ordering area.

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 2c65836e-db55-46e4-b440-f4fc14e200aa in the triage console for the audit trail.

@wfaderhold21
wfaderhold21 force-pushed the topic/onesided-a2a-rank-ordering branch from 278b27b to c28ac9b Compare September 16, 2026 17:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants