Skip to content

UCT/IB/RC/VERBS: Support QPs without SRQ - #11933

Open
GuangguanWang wants to merge 8 commits into
openucx:masterfrom
GuangguanWang:master
Open

GuangguanWang wants to merge 8 commits into
openucx:masterfrom
GuangguanWang:master

Conversation

@GuangguanWang

@GuangguanWang GuangguanWang commented Sep 9, 2026 •

Copy link
Copy Markdown

What?

Enable rc_verbs on devices without SRQ by posting receive WRs to each QP.

Add configuration for selecting SRQ usage and setting the receive WR count per
QP. Update flow control, RX CQ accounting and QP cleanup for this mode.

Two pre-existing bugs are fixed as separate commits, because the new mode
reaches both of them. They change existing behavior of the SRQ path:

  • An endpoint is added to iface->ep_list when uct_rc_ep_t is initialized, but
    it was removed outside of the uct_rc_ep_t cleanup, so an endpoint whose
    initialization failed was freed while still on the list. The removal is moved
    to the uct_rc_ep_t cleanup and dropped from rc_verbs, rc_mlx5 and gga_mlx5.
  • A receive descriptor is taken from the interface pool for every receive WR
    posted to the SRQ, but a completion which reported an error skipped the
    release of the descriptor while the SRQ counted the WR as available, so every
    failed receive leaked one descriptor.

Why?

rc_verbs currently requires every device to support SRQ during transport
discovery. Some devices, such as eRDMA, support RC QPs but do not support SRQ,
so rc_verbs is not exposed on them. Supporting QPs without SRQ enables a wider
range of RDMA devices to use UCX through rc_verbs.

How?

  • Add RC initialization attributes and a QP receive WR count.
  • Select SRQ or per-QP receive posting during interface initialization.
  • Track and refill receive WRs per endpoint.
  • Limit the flow control window to the receive capacity of one QP.
  • Defer QP destruction until its receive WR completions are drained, since
    IBV_EVENT_QP_LAST_WQE_REACHED is not generated for a QP without an SRQ.
  • Refuse endpoint creation when the RX CQ has no room left for the receive WRs
    of one more QP, which makes max_num_eps an enforced limit.

The gtest coverage of this mode is a follow-up PR #11938.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@svc-ucx

svc-ucx commented Sep 9, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Codestyle AUTHORS file update check) · commit 1e869fbd

TL;DR: The "AUTHORS file check" step failed because PR #11933's author, Guangguan Wang guangguan.wang@linux.alibaba.com, is not listed in the AUTHORS file; the fix is for the contributor to add that line to AUTHORS (sorted position, after Graham Lopez) and push.

Full analysis

Summary: Azure Pipelines codestyle job "AUTHORS file update check" → step "AUTHORS file check" exited with code 1 because git diff --exit-code showed AUTHORS was modified by contrib/authors_update.sh.

Root cause: Not an infrastructure or timing problem. The check computes the PR's commit range ([REDACTED:Hex High Entropy String]..[REDACTED:Hex High Entropy String]), runs contrib/authors_update.sh over it, and then requires the working tree to stay clean. The script found a commit author whose email/name is not present in AUTHORS and appended it:

Names:
++ Guangguan Wang <guangguan.wang@linux.alibaba.com>
...
+Guangguan Wang <guangguan.wang@linux.alibaba.com>
##[error]Bash exited with code '1'.

Confirmed by reading the repo: AUTHORS jumps from Graham Lopez <lopezmg@ornl.gov> (line 43) straight to Guy Ealey Morag (line 44), with no entry for this contributor. contrib/authors_update.sh:36 matches by email word or name, so neither matched. The check is working as designed — the PR is simply missing the first-time-contributor AUTHORS update.

Implicated commit: [REDACTED:Hex High Entropy String] (Guangguan Wang) — the PR head commit, which lacks the AUTHORS entry. No regression in the CI tooling.

File: AUTHORS:43 (insertion point); check logic in contrib/authors_update.sh:36-46

Suggested fix: In PR #11933, run the same command locally and commit the result:

./contrib/authors_update.sh $(git merge-base origin/master HEAD)..HEAD
git add AUTHORS && git commit -m "AUTHORS: add Guangguan Wang"

Equivalently, manually insert Guangguan Wang <guangguan.wang@linux.alibaba.com> between lines 43 and 44 of AUTHORS (the file is LC_COLLATE=C sorted, so it goes after Graham Lopez and before Guy Ealey Morag). No CI or pipeline change is needed. Note: if the contributor already appears in AUTHORS under a different email, the correct fix instead is to add a .mailmap-style alias or use the already-registered email for the commits, since the script keys off exact email/name matches.

Related: none

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id aaa8b636-b6d4-443d-bcf5-6726c4dd21d3 in the triage console for the audit trail.

Comment thread src/uct/ib/rc/base/rc_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/base/rc_ep.c Outdated
Comment thread src/uct/ib/mlx5/rc/rc_mlx5_iface.c Outdated
@svc-nvidia-pr-review

Copy link
Copy Markdown

Overall: REQUEST_CHANGES (one blocker). pls split the uct_rc_iface_init_attr_t refactor (dc_mlx5, gga_mlx5, gdaki, rc_mlx5) into a separate PR — this one adds ~840 lines, well above the 500 line limit in REVIEW.md.

Note: QP drain now depends on the RX progress callback being registered (the SRQ path used the worker progress_q via the LAST_WQE event). QPs and their TX CQ credits stay pending until iface close if UCT_PROGRESS_RECV was never enabled — bounded, so not raised as a blocker, but pls confirm.

Coverage gap: no test exercises the no-SRQ path with more endpoints than the RX CQ can hold, i.e. the actual overflow behind the new warning; contrib/ibmock (max_srq = 0) presumably becomes the CI carrier for this path but no CI/ibmock change is in the diff.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs.h Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: contrib/ibmock/verbs.c:ibv_create_qp() rejects everything except IBV_QPT_UD/IBV_QPT_DRIVER ("RC is not supported"), so the mock harness cannot exercise the new no-SRQ rc_verbs path as-is — real hardware without SRQ support, or an ibmock RC/ibv_post_recv completion path, would be needed.

@svc-ucx

svc-ucx commented Sep 10, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (EFA_Tests EFA on rhel90_ib) · commit faa3c554

[incomplete: stop_reason=max_tokens]

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread test/gtest/uct/ib/test_ib_event.cc
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap (beyond what was already noted): no test runs the existing rc_verbs suite with UCX_RC_VERBS_SRQ_ENABLE=n, which would exercise the new per-QP receive, repost, and QP-drain paths on ordinary SRQ-capable hardware without needing ibmock RC support.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
@svc-ucx

svc-ucx commented Sep 11, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 1) · commit 91e0f1fb

TL;DR: The only failure in the 8725-test run is the newly added rcx/test_ucp_fault_tolerance.probe_gated_recovery/6 (rc_x/AM), which fails on its EXPECT_TRUE(probe_armed) check because the aux uct_ep_check probe window is sampled by polling and can open/close entirely before (or between) the test's poll iterations — a race in the test, not a product bug. Fix by observing probe arming deterministically (mock ep_check / sticky counter) instead of polling recovery_arg->probe[lane].comp.func.

Full analysis

Summary: make test in the "roce on worker 1" job exited 1: [ FAILED ] rcx/test_ucp_fault_tolerance.probe_gated_recovery/6, where GetParam() = rc_x/AM (8724 passed, 1 failed, no hang — the run reached global tear-down normally).

Root cause: probe_gated_recovery reuses test_am_with_injected_failure(FAILURE_SIDE_TARGET, TEST_OP_AM) and then tries to catch in the act an armed aux probe by polling ep->ext->recovery_arg->probe[lane].comp.func != NULL inside wait_for_cond(...). That window is transient and not sticky:

  • ucp_ep_recovery_arm_probe() sets probe->comp.func (src/ucp/core/ucp_ep.c:2083),
  • ucp_ep_recovery_reset_probe() clears it as soon as the p2p lane reconnects (src/ucp/core/ucp_ep.c:2188, called from ucp_ep_recovery_rebuild_p2p_lane),
  • ucp_ep_recovery_arg_free() frees the whole struct when recovery finishes (src/ucp/core/ucp_ep.c:2044), leaving recovery_arg == NULL.

Recovery is driven by progress, and the helper already runs many AM sends/flushes/progress loops after the failure injection. If recovery completes during those loops, wait_for_cond (test/gtest/common/test.h:105 — condition is evaluated before the first wait) sees ucp_ep_get_failed_lanes(ep) == 0 immediately and exits with probe_armed == false, so EXPECT_TRUE(probe_armed) fires. That the shared helper is fine is confirmed by rcx/test_ucp_fault_tolerance.target_failure/6 passing in the same run — only the probe-observation assertion unique to probe_gated_recovery failed. This makes the test timing-sensitive/flaky, which matches a single sporadic failure on one worker.

Implicated commit: db208ee — "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)", Evgeny Leksikov (added both the probe logic and this test)

File: test/gtest/ucp/test_ucp_fault_tolerance.cc:922-943 (probe polling + EXPECT_TRUE(probe_armed)); product side: src/ucp/core/ucp_ep.c:2073-2097 and :2188

Suggested fix: Make the observation deterministic rather than sampled:

  1. Preferred: use the existing ucs::mock hook (mock_recovery_probe(), already used by teardown_with_outstanding_probe / recovery_retries_exhausted_live_lanes) with an ep_check wrapper that increments a static counter and returns UCS_INPROGRESS (capturing the completion), then assert the counter is non-zero — this cannot be missed regardless of timing; or
  2. Add a sticky counter in the product (e.g. ++ep->worker->counters.ep_recovery_probes in ucp_ep_recovery_arm_probe(), src/ucp/core/ucp_ep.c:2073) and have the test assert on that counter instead of on the volatile probe[lane].comp.func.
    As an interim measure, start the polling loop before the failure injection (or mark the test flaky) so it cannot miss the arming window.

Related: #11563 (introduced the probe gate and this test); PR under test: #11933. No existing issue found for probe_gated_recovery flakiness.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
Comment thread src/uct/ib/rc/base/rc_ep.c
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: beyond what was already noted, nothing exercises rx_remaining-based QP drain, i.e. destroying an endpoint while receive WRs are still posted with SRQ_ENABLE=n — that is the one new state machine in this PR (gc_drain_cqe/gc_drain_check) and it only runs when RX progress is polled.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_impl.h Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
@svc-nvidia-pr-review

Copy link
Copy Markdown

Residual coverage gap: the no-SRQ path (per-QP receives, repost retry, rx_remaining drain, and the new EXCEEDS_LIMIT path) still has no test or CI job, since contrib/ibmock cannot create RC QPs.

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c Outdated
Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c
@svc-ucx

svc-ucx commented Sep 15, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests roce on worker 2) · commit c6545c04

TL;DR: The only failing test in the whole 8726-test run is rcx/test_ucp_fault_tolerance.probe_gated_recovery/6 (rc_x/AM), which fails on its own racy EXPECT_TRUE(probe_armed) check — the aux probe state it polls for is transient and can be armed and cleared inside a single worker progress iteration, so the test can never observe it. This is a flaky test introduced with the probe-gated recovery feature, not a regression from PR #11933 (which touches uct/ib/rc/verbs SRQ handling, while the failing variant is rc_x/rc_mlx5).

Full analysis

Summary: gtest job "roce on worker 2" failed with make: *** [Makefile:4713: test] Error 1; 8725 tests passed, 1 failed: rcx/test_ucp_fault_tolerance.probe_gated_recovery/6, where GetParam() = rc_x/AM.

Root cause: probe_gated_recovery polls for a transient internal state instead of a sticky one. Its predicate (test_ucp_fault_tolerance.cc:922-939) samples ep->ext->recovery_arg->probe[lane].comp.func != NULL between short_progress_loop() calls, then asserts probe_armed at line 941. In the implementation, ucp_ep_recovery_arm_probe() sets probe->comp.func (src/ucp/core/ucp_ep.c:2083) and, when uct_ep_check() on the aux (ud_mlx5) EP completes synchronously (status != UCS_INPROGRESS, ucp_ep.c:2089-2094), the same call chain continues to ucp_wireup_ep_connect_to_ep_v2() and immediately calls ucp_ep_recovery_reset_probe() which sets comp.func = NULL (ucp_ep.c:2183-2188). When all failed lanes recover, ucp_ep_recovery_arg_free() nulls recovery_arg (ucp_ep.c:2409). So arm→reset (or full recovery) can happen entirely within one progress call, leaving probe_armed == false and firing "RC p2p lane recovery completed without arming an aux probe". Note the same test_am_with_injected_failure(FAILURE_SIDE_TARGET, TEST_OP_AM) helper and the same "wait until failed lanes == 0" loop are exercised by target_failure/6 on the identical variant and passed in this run — the only delta is the probe_armed observation, which confirms the racy sampling is what failed.

Implicated commit: db208ee — "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)", Evgeny Leksikov (added both the probe arm/reset logic and this test)

File: test/gtest/ucp/test_ucp_fault_tolerance.cc:941 (predicate at :922-939); implementation: src/ucp/core/ucp_ep.c:2083-2094 and :2183-2188, :2409

Suggested fix: Stop sampling transient state. Preferred: intercept the probe deterministically like the neighbouring teardown_with_outstanding_probe test does — install a ucs::mock on ops.ep_check via mock_recovery_probe() with a wrapper that sets a probe_armed flag and forwards/returns UCS_INPROGRESS, then assert on that flag. Alternatively add a sticky counter (e.g. unsigned probes_armed) to ucp_ep_recovery_arg_t, increment it in ucp_ep_recovery_arm_probe() (never cleared by ucp_ep_recovery_reset_probe()), expose it as an EP/worker counter that survives ucp_ep_recovery_arg_free(), and check that instead of comp.func. Until fixed, this failure should not block PR #11933; if it recurs, mark the test as flaky/disabled per the precedent of commit 13ae935.

Related: PR #11563 (feature + test), PR #11933 (the PR under test, unrelated rc_verbs SRQ change), commit 13ae935 "TEST/GTEST: Disable failing fault-tolerance test (#11397)" — prior flakiness in the same suite

@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

Comment thread src/uct/ib/rc/verbs/rc_verbs_iface.c
Comment thread src/uct/ib/rc/verbs/rc_verbs_ep.c
@GuangguanWang

Copy link
Copy Markdown
Author

Hi, @brminich @Artemy-Mellanox, Could you take a look at this PR when you have a chance?
This PR introduce NON-SRQ path for rc_verbs, for devices that do not have SRQ capability.

@brminich

Copy link
Copy Markdown
Contributor

@GuangguanWang thanks for the contribution!
Do you have a CLA signed?

@GuangguanWang

Copy link
Copy Markdown
Author

@GuangguanWang thanks for the contribution! Do you have a CLA signed?

@brminich Yes, I have signed the CLA and already received the counter-signed agreement.

@brminich

Copy link
Copy Markdown
Contributor

@GuangguanWang can you pls resolve the conflict

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
The RC interface constructor takes uct_ib_iface_init_attr_t, so an RC
transport cannot pass an attribute which only the RC layer understands. Add
uct_rc_iface_init_attr_t, which wraps it, and pass that instead. No attribute
is added yet, so the behavior is unchanged.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
uct_rc_iface_fill_attr() always sets max_recv_wr to zero, because every RC
transport attaches its QPs to an SRQ. Take the length as an argument, so that
a QP can be created without an SRQ. All the callers pass zero, so the
behavior is unchanged.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
The device port check finds out whether a device supports SRQ by comparing
IBV_DEV_ATTR(dev, max_srq) with zero. Wrap the comparison in a helper, so
that the transports can perform the same check.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
An endpoint is added to iface->ep_list when uct_rc_ep_t is initialized, but
it is removed outside of the uct_rc_ep_t cleanup: by rc_verbs and rc_mlx5
through uct_rc_ep_cleanup_qp(), and by gga_mlx5 on its own. If a transport
fails to initialize an endpoint, it is freed while it is still on ep_list,
and uct_rc_iface_flush() then calls uct_ep_flush() on freed memory.

Remove the endpoint in the uct_rc_ep_t cleanup, which runs both when the
endpoint is destroyed and when a transport fails to initialize it.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
A receive descriptor is taken from the interface pool for every receive WR
posted to the SRQ, but a completion which reports an error returns without
releasing it, while the SRQ counts the WR as available and the next post
takes another descriptor from the pool. Every failed receive therefore leaks
a descriptor.

Release the descriptor before skipping the completion.
rc_verbs currently rejects devices whose max_srq is zero. Allow these
devices and create every QP with RC_VERBS_RX_MAX_WR receive work requests.
RC_VERBS_SRQ_ENABLE controls whether SRQ is required, disabled, or selected
according to the device capability.

For QPs without SRQ, post and refill receive work requests per endpoint and
use the QP number in each receive completion to update the corresponding
receive queue. Limit the flow control window to RC_VERBS_RX_MAX_WR, and warn
when the sum of the receive queue capacities exceeds the RX CQ capacity.

IBV_EVENT_QP_LAST_WQE_REACHED is not generated for QPs without SRQ. Keep a
destroyed QP until completions for all its posted receive work requests have
been polled, then release the QP and restore its RX CQ capacity.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
The LAST_WQE_REACHED event is generated only for a QP which is attached to an
SRQ, so the tests which wait for it never complete otherwise. Skip them when
the interface works without an SRQ.

Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

@GuangguanWang

Copy link
Copy Markdown
Author

@GuangguanWang can you pls resolve the conflict

@brminich Done, the code has been rebased to the master branch.

@@ -429,15 +535,42 @@ unsigned uct_rc_verbs_iface_post_recv_always(uct_rc_verbs_iface_t *iface, unsign
return 0;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

iface_attr.max_num_eps still advertises the configured value even when max_cqe caps num_eps below it (e.g. small max_cqe, or a larger RX_MAX_WR), so ep_create later fails with EXCEEDS_LIMIT instead of letting the upper layer pick another transport. can we also lower self->super.super.super.config.max_num_eps to rx_cq_len / rx_max_wr?

@svc-ucx

svc-ucx commented Sep 29, 2026

Copy link
Copy Markdown

🤖 CI Triage Agent — UCX PR (Tests althca on worker 2) · commit ac194c17

TL;DR: The gtest cma/test_uct_perf.envelope/0 crashed with SIGSEGV inside ucx_perf_allocator_register() because the global perftest allocator registry is mutated without any locking/once-guard while two perf threads call ucx_perf_global_init() concurrently — one thread reads a NULL array slot whose counter was already incremented. Fix: serialize ucx_perf_global_init() with ucs_init_once/pthread_once and protect register/unregister with a mutex (and store the slot before bumping the counter).

Full analysis

Summary: cma/test_uct_perf.envelope/0 segfaulted (Caught signal 11 ... address not mapped to object at address (nil)), killing the gtest run (make: *** [test] Segmentation fault, task exit code 2) in the "althca on worker 2" job.

Root cause: Backtrace from the log:

0 ucx_perf_allocator_register()  src/tools/perf/lib/libperf_int.h:121
1 ucx_perf_global_init()         src/tools/perf/lib/libperf_memory.c:415
2 ucx_perf_run()                 src/tools/perf/lib/libperf.c:2239
3 test_perf::test_func()         test/gtest/common/test_perf.cc:194
4 start_thread()

The faulting line is if (!strcmp(name, ucx_perf_allocators[i]->name)) (libperf_int.h:121), and the fault address is (nil) — i.e. ucx_perf_allocators[i] was NULL while ucx_perf_num_allocators already counted it.

The registry is a plain unsynchronized global (libperf.c:87-88):

const ucx_perf_allocator_t *ucx_perf_allocators[UCX_PERF_ALLOCATOR_MAX];
unsigned ucx_perf_num_allocators;

ucx_perf_global_init() (libperf_memory.c:384-423) has no ucs_init_once/pthread_once guard and is called unconditionally on every ucx_perf_run() (libperf.c:2239). The gtest test_perf harness starts one thread per RTE peer, each running ucx_perf_run() (test_perf.cc:185-196; note the crashing thread is tagged p-1, so p-0 was running in parallel). Both threads therefore execute ucx_perf_allocator_register() simultaneously, where the publish is:

ucx_perf_allocators[ucx_perf_num_allocators++] = allocator;   /* libperf_int.h:132 */

There is no ordering between the counter increment and the pointer store (and no mutual exclusion at all), so the peer thread's loop at line 120-121 can observe num_allocators == 1 with slot 0 still NULL → NULL dereference. UCS_MODULE_FRAMEWORK_LOAD at line 422 (cuda/rocm allocators registering themselves) widens the same window. This is a latent race, not a functional break from PR #11933 — expect it to be intermittent/flaky across jobs.

Implicated commit: e917453 — "TOOLS/PERF: Add cuda-async memory allocator (#11566)", Thomas Vegas (introduced the dynamic ucx_perf_allocators[] registry and the inline ucx_perf_allocator_register() used at the crash site); PR #11933 / commit ac194c1 only exposed it by scheduling.

File: src/tools/perf/lib/libperf_int.h:121 (publish bug at libperf_int.h:132; unguarded caller at src/tools/perf/lib/libperf_memory.c:415 and src/tools/perf/lib/libperf.c:2239)

Suggested fix:

  1. Make ucx_perf_global_init() idempotent and single-threaded, e.g. wrap its body in static ucs_init_once_t init_once = UCS_INIT_ONCE_INITIALIZER; UCS_INIT_ONCE(&init_once) { ... } (or pthread_once), so module loading and the built-in host/rdma registrations happen exactly once.
  2. Protect ucx_perf_allocator_register() / ucx_perf_allocator_unregister() / ucx_perf_allocator_by_name() with a static mutex, since modules can also register from other contexts.
  3. Fix the publish order regardless of locking: store the pointer first, then increment the count, e.g.
    ucx_perf_allocators[ucx_perf_num_allocators] = allocator;
    ucs_memory_cpu_store_fence();
    ++ucx_perf_num_allocators;
    and/or add a defensive if (ucx_perf_allocators[i] == NULL) continue; in the lookup loops (libperf_int.h:120, libperf.c:97).
  4. Since this is intermittent, re-run the job to unblock PR UCT/IB/RC/VERBS: Support QPs without SRQ #11933, but track the race separately.

Related: PR #11566 (introduced the allocator registry); no existing issue found for this crash signature — worth opening one.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants