UCT/IB/RC/VERBS: Support QPs without SRQ - #11933
GuangguanWang wants to merge 8 commits into
Conversation
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The "AUTHORS file check" step failed because PR #11933's author, Guangguan Wang guangguan.wang@linux.alibaba.com, is not listed in the Full analysisSummary: Azure Pipelines codestyle job "AUTHORS file update check" → step "AUTHORS file check" exited with code 1 because Root cause: Not an infrastructure or timing problem. The check computes the PR's commit range ( Confirmed by reading the repo: Implicated commit: [REDACTED:Hex High Entropy String] (Guangguan Wang) — the PR head commit, which lacks the AUTHORS entry. No regression in the CI tooling. File: AUTHORS:43 (insertion point); check logic in contrib/authors_update.sh:36-46 Suggested fix: In PR #11933, run the same command locally and commit the result: Equivalently, manually insert Related: none
|
|
Overall: REQUEST_CHANGES (one blocker). pls split the Note: QP drain now depends on the RX progress callback being registered (the SRQ path used the worker Coverage gap: no test exercises the no-SRQ path with more endpoints than the RX CQ can hold, i.e. the actual overflow behind the new warning; |
1e869fb to
faa3c55
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: |
|
🤖 CI Triage Agent — [incomplete: stop_reason=max_tokens] |
faa3c55 to
bd524c3
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap (beyond what was already noted): no test runs the existing rc_verbs suite with |
bd524c3 to
91e0f1f
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The only failure in the 8725-test run is the newly added Full analysisSummary: Root cause:
Recovery is driven by progress, and the helper already runs many AM sends/flushes/progress loops after the failure injection. If recovery completes during those loops, Implicated commit: db208ee — "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)", Evgeny Leksikov (added both the probe logic and this test) File: test/gtest/ucp/test_ucp_fault_tolerance.cc:922-943 (probe polling + Suggested fix: Make the observation deterministic rather than sampled:
Related: #11563 (introduced the probe gate and this test); PR under test: #11933. No existing issue found for |
91e0f1f to
6c6653e
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: beyond what was already noted, nothing exercises |
6c6653e to
5774977
Compare
|
🤖 Starting review — findings will be posted here when done. |
5774977 to
71b9961
Compare
|
🤖 Starting review — findings will be posted here when done. |
71b9961 to
36f56d0
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Residual coverage gap: the no-SRQ path (per-QP receives, repost retry, |
36f56d0 to
c6545c0
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 CI Triage Agent — TL;DR: The only failing test in the whole 8726-test run is Full analysisSummary: gtest job "roce on worker 2" failed with Root cause: Implicated commit: db208ee — "UCP/FT: probe-gated lane recovery via aux uct_ep_check (#11563)", Evgeny Leksikov (added both the probe arm/reset logic and this test) File: test/gtest/ucp/test_ucp_fault_tolerance.cc:941 (predicate at :922-939); implementation: src/ucp/core/ucp_ep.c:2083-2094 and :2183-2188, :2409 Suggested fix: Stop sampling transient state. Preferred: intercept the probe deterministically like the neighbouring Related: PR #11563 (feature + test), PR #11933 (the PR under test, unrelated |
c6545c0 to
101149d
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
Hi, @brminich @Artemy-Mellanox, Could you take a look at this PR when you have a chance? |
|
@GuangguanWang thanks for the contribution! |
@brminich Yes, I have signed the CLA and already received the counter-signed agreement. |
|
@GuangguanWang can you pls resolve the conflict |
Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
The RC interface constructor takes uct_ib_iface_init_attr_t, so an RC transport cannot pass an attribute which only the RC layer understands. Add uct_rc_iface_init_attr_t, which wraps it, and pass that instead. No attribute is added yet, so the behavior is unchanged. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
uct_rc_iface_fill_attr() always sets max_recv_wr to zero, because every RC transport attaches its QPs to an SRQ. Take the length as an argument, so that a QP can be created without an SRQ. All the callers pass zero, so the behavior is unchanged. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
The device port check finds out whether a device supports SRQ by comparing IBV_DEV_ATTR(dev, max_srq) with zero. Wrap the comparison in a helper, so that the transports can perform the same check. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
An endpoint is added to iface->ep_list when uct_rc_ep_t is initialized, but it is removed outside of the uct_rc_ep_t cleanup: by rc_verbs and rc_mlx5 through uct_rc_ep_cleanup_qp(), and by gga_mlx5 on its own. If a transport fails to initialize an endpoint, it is freed while it is still on ep_list, and uct_rc_iface_flush() then calls uct_ep_flush() on freed memory. Remove the endpoint in the uct_rc_ep_t cleanup, which runs both when the endpoint is destroyed and when a transport fails to initialize it. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
A receive descriptor is taken from the interface pool for every receive WR posted to the SRQ, but a completion which reports an error returns without releasing it, while the SRQ counts the WR as available and the next post takes another descriptor from the pool. Every failed receive therefore leaks a descriptor. Release the descriptor before skipping the completion.
rc_verbs currently rejects devices whose max_srq is zero. Allow these devices and create every QP with RC_VERBS_RX_MAX_WR receive work requests. RC_VERBS_SRQ_ENABLE controls whether SRQ is required, disabled, or selected according to the device capability. For QPs without SRQ, post and refill receive work requests per endpoint and use the QP number in each receive completion to update the corresponding receive queue. Limit the flow control window to RC_VERBS_RX_MAX_WR, and warn when the sum of the receive queue capacities exceeds the RX CQ capacity. IBV_EVENT_QP_LAST_WQE_REACHED is not generated for QPs without SRQ. Keep a destroyed QP until completions for all its posted receive work requests have been polled, then release the QP and restore its RX CQ capacity. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
The LAST_WQE_REACHED event is generated only for a QP which is attached to an SRQ, so the tests which wait for it never complete otherwise. Skip them when the interface works without an SRQ. Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
101149d to
ac194c1
Compare
|
🤖 Starting review — findings will be posted here when done. |
@brminich Done, the code has been rebased to the master branch. |
| @@ -429,15 +535,42 @@ unsigned uct_rc_verbs_iface_post_recv_always(uct_rc_verbs_iface_t *iface, unsign | |||
| return 0; | |||
There was a problem hiding this comment.
iface_attr.max_num_eps still advertises the configured value even when max_cqe caps num_eps below it (e.g. small max_cqe, or a larger RX_MAX_WR), so ep_create later fails with EXCEEDS_LIMIT instead of letting the upper layer pick another transport. can we also lower self->super.super.super.config.max_num_eps to rx_cq_len / rx_max_wr?
|
🤖 CI Triage Agent — TL;DR: The gtest Full analysisSummary: Root cause: Backtrace from the log: The faulting line is The registry is a plain unsynchronized global ( const ucx_perf_allocator_t *ucx_perf_allocators[UCX_PERF_ALLOCATOR_MAX];
unsigned ucx_perf_num_allocators;
ucx_perf_allocators[ucx_perf_num_allocators++] = allocator; /* libperf_int.h:132 */There is no ordering between the counter increment and the pointer store (and no mutual exclusion at all), so the peer thread's loop at line 120-121 can observe Implicated commit: e917453 — "TOOLS/PERF: Add cuda-async memory allocator (#11566)", Thomas Vegas (introduced the dynamic File: src/tools/perf/lib/libperf_int.h:121 (publish bug at libperf_int.h:132; unguarded caller at src/tools/perf/lib/libperf_memory.c:415 and src/tools/perf/lib/libperf.c:2239) Suggested fix:
Related: PR #11566 (introduced the allocator registry); no existing issue found for this crash signature — worth opening one. |
What?
Enable rc_verbs on devices without SRQ by posting receive WRs to each QP.
Add configuration for selecting SRQ usage and setting the receive WR count per
QP. Update flow control, RX CQ accounting and QP cleanup for this mode.
Two pre-existing bugs are fixed as separate commits, because the new mode
reaches both of them. They change existing behavior of the SRQ path:
it was removed outside of the uct_rc_ep_t cleanup, so an endpoint whose
initialization failed was freed while still on the list. The removal is moved
to the uct_rc_ep_t cleanup and dropped from rc_verbs, rc_mlx5 and gga_mlx5.
posted to the SRQ, but a completion which reported an error skipped the
release of the descriptor while the SRQ counted the WR as available, so every
failed receive leaked one descriptor.
Why?
rc_verbs currently requires every device to support SRQ during transport
discovery. Some devices, such as eRDMA, support RC QPs but do not support SRQ,
so rc_verbs is not exposed on them. Supporting QPs without SRQ enables a wider
range of RDMA devices to use UCX through rc_verbs.
How?
IBV_EVENT_QP_LAST_WQE_REACHED is not generated for a QP without an SRQ.
of one more QP, which makes max_num_eps an enforced limit.
The gtest coverage of this mode is a follow-up PR #11938.