UCT/API: Clarify capability requirements for purge and error handler by jeynmann · Pull Request #11849 · openucx/ucx

jeynmann · 2026-08-28T04:58:51Z

What?
Clarify the API requirements for error-handler in-progress handling and outstanding operation purge.

Why?
Ensure that UCS_INPROGRESS and outstanding purge are used only by interfaces supporting UCT_IFACE_FLAG_V2_QUERY_TOKEN.

How?
Restrict returning UCS_INPROGRESS to interfaces supporting UCT_IFACE_FLAG_V2_QUERY_TOKEN.
Restrict uct_ep_outstanding_purge() to interfaces supporting UCT_IFACE_FLAG_V2_QUERY_TOKEN.
Require the RX token and callback bits in the purge field mask.

svc-nvidia-pr-review · 2026-08-28T04:59:03Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T05:00:10Z

-                                                               support token query, and
-                                                               @ref uct_ep_outstanding_purge
-                                                               is supported. */
+#define UCT_IFACE_FLAG_V2_QUERY_TOKEN            UCS_BIT(2)  /**< @ref uct_iface_query_v2


why remove the uct_ep_outstanding_purge support statement here? rc_mlx5 still sets ep_outstanding_purge under the same DEVX condition that sets this flag, so callers currently rely on this flag to detect purge support. if purge is now advertised differently, pls point to the new flag.

svc-nvidia-pr-review · 2026-08-28T05:00:10Z

+                                                                  and @ref uct_ep_query
+                                                                  support token query. */
+        /* Error handling capabilities */
+#define UCT_IFACE_FLAG_V2_ERR_HANDLER_INPROGRESS UCS_BIT(3)  /**< Supports error handler


this flag isn't set by any transport or checked anywhere, and there's no test. is the producer/consumer coming in a follow-up? pls add the transport that reports it (or a test) so the capability is actually observable.

svc-nvidia-pr-review · 2026-08-28T05:00:11Z

Residual coverage gap: No test change accompanies the new flag, and the existing query_token_support test does not assert anything about ERR_HANDLER_INPROGRESS or the purge-support relationship, so the doc/behavior change is not validated by CI.

svc-nvidia-pr-review · 2026-08-28T05:17:41Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T05:19:31Z

                                                 ucs_status_t status)
 {
-    if (UCS_STATUS_IS_ERR(err_handler_status)) {
+    if (err_handler_status != UCS_OK) {


The condition changed from UCS_STATUS_IS_ERR(err_handler_status) (i.e. < 0) to err_handler_status != UCS_OK. Since UCS_INPROGRESS == 1 (positive), the old code treated an error handler returning UCS_INPROGRESS as non-fatal, while the new code logs UCS_LOG_LEVEL_FATAL for it. This is the opposite of what the newly added UCT_IFACE_FLAG_V2_ERR_HANDLER_INPROGRESS flag documents ("Supports error handler returning UCS_INPROGRESS"). Impact: a transport/error-handler that returns UCS_INPROGRESS — the exact case the new flag is meant to advertise as valid — would now produce a FATAL failure log. The change also does not consult the new capability flag when deciding fatality, so the flag and the code are inconsistent.

svc-nvidia-pr-review · 2026-08-28T05:19:31Z

+                                                                  and @ref uct_ep_query
+                                                                  support token query. */
+        /* Error handling capabilities */
+#define UCT_IFACE_FLAG_V2_ERR_HANDLER_INPROGRESS UCS_BIT(3)  /**< Supports error handler


UCT_IFACE_FLAG_V2_ERR_HANDLER_INPROGRESS is defined but never set by any transport, never read by any consumer, and does not gate the uct_base_iface_failure_log_level change. As is, it has no effect. Either a transport should advertise it and uct_base_iface_failure_log_level should key its UCS_INPROGRESS handling off it, or the flag should be dropped from this PR.

svc-nvidia-pr-review · 2026-08-28T05:19:31Z

-                                                               support token query, and
-                                                               @ref uct_ep_outstanding_purge
-                                                               is supported. */
+#define UCT_IFACE_FLAG_V2_QUERY_TOKEN            UCS_BIT(2)  /**< @ref uct_iface_query_v2


The UCT_IFACE_FLAG_V2_QUERY_TOKEN doc previously stated that uct_ep_outstanding_purge is supported when the flag is set. The PR removes that sentence. rc_mlx5 still both sets UCT_IFACE_FLAG_V2_QUERY_TOKEN and provides ep_outstanding_purge (uct_ib_mlx5_ext_ep_outstanding_purge), so if any caller relied on that documented relationship to gate purge usage, this removes the contract without a replacement. Worth confirming this is intentional and that no consumer keyed purge support off QUERY_TOKEN.

svc-nvidia-pr-review · 2026-08-28T05:19:33Z

Residual coverage gap: no test exercises UCS_INPROGRESS handling or the new flag.

svc-nvidia-pr-review · 2026-08-28T07:39:19Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T07:41:25Z

            status = uct_rc_mlx5_iface_query_rx_token(tl_iface, iface_attr);
            if (status != UCS_OK) {
                return status;
            }


is this intended to disable the feature, or only hide the advertisement? the token query and outstanding-purge paths are still fully implemented in uct_rc_mlx5_iface_query_v2 / uct_rc_mlx5_ep_query_tx_token, so the capability stays reachable but undiscoverable. if the goal is to disable it, should the token-query code be removed/gated too?

svc-nvidia-pr-review · 2026-08-28T07:41:25Z

 */
        /* PUT capabilities */
 #define UCT_IFACE_FLAG_V2_PUT_SGL_ZCOPY       UCS_BIT(0)  /**< Zero-copy SGL put */
        /* GET capabilities */


The reworded comment for UCT_IFACE_FLAG_V2_QUERY_TOKEN still lists uct_ep_query token query and uct_ep_outstanding_purge as gated by this flag. Since no transport now sets the flag, and the mechanisms remain active, this documentation may mislead users about what the flag actually guards.

svc-ucx · 2026-08-28T07:43:53Z

🤖 CI Triage Agent — UCX PR (Build build_aarch64 ctyunos2507_aarch64) · commit 599adf73

TL;DR: The build_docs step failed because a doxygen @ref in src/uct/api/uct_def.h:578 points at UCT_IFACE_FLAG_V2_QUERY_TOKEN, a symbol that no longer exists after PR #11849's commit "UCT/API: rm in progress flag and revert query" — doxygen runs with warnings-as-errors, so the unresolved reference aborts the build. Fix by dropping the stale @ref (or referencing the capability flag that actually exists/is added by this PR).

Full analysis

Summary: build_aarch64 / build_docs failed at make V=1 docs — doxygen aborted on an unresolved \ref in the UCT public API headers.

Root cause: Log line: /__w/29/s/src/uct/api/uct_def.h:578: error: unable to resolve reference to 'UCT_IFACE_FLAG_V2_QUERY_TOKEN' for \ref command (warning treated as error, aborting now) → make: *** [Makefile:1308: docs/doxygen-doc/ucx.tag] Error 1. The doc block for uct_error_handler_t says the user may return UCS_INPROGRESS only if the interface supports @ref UCT_IFACE_FLAG_V2_QUERY_TOKEN, but that identifier is not defined anywhere in the tree at this commit — the PR head commit removed the in-progress/token-query flag while leaving the doc reference behind. Additionally, doxygen's INPUT list (visible in the log) only parses src/uct/api/uct.h and uct_def.h non-recursively, so src/uct/api/v2/uct_v2.h symbols can never be resolved by @ref from these headers even if the flag were re-added there. Since ucxdox treats warnings as errors, this is a hard build failure (not test flakiness).

Implicated commit: [REDACTED:Hex High Entropy String] — Zihao Zhao, "UCT/API: rm in progress flag and revert query" (2026-08-28)

File: src/uct/api/uct_def.h:578

Suggested fix: In src/uct/api/uct_def.h, remove the dangling reference. Either:

Replace @ref UCT_IFACE_FLAG_V2_QUERY_TOKEN with plain/inline-code text (e.g. "... only if the interface reports the error-handler in-progress capability"), or
If the PR is meant to expose a capability bit, define and document it in src/uct/api/uct.h next to the other UCT_IFACE_FLAG_ERRHANDLE_* flags (e.g. UCT_IFACE_FLAG_ERRHANDLE_INPROGRESS) and @ref that instead — uct.h is in doxygen's INPUT so it will resolve.

Avoid @ref-ing anything under src/uct/api/v2/ from these headers; that directory is not in the doxygen INPUT set. Verify locally with ./contrib/configure-release --with-docs-only && make docs.

Related: PR #11849 (this PR), PR #11823 (prior "rc support in progress error handler"), PR #11277 / #11668 (earlier users of the token-query flag)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 9ab53ced-f74f-4c93-b8c5-4a01ad0dc5bc in the triage console for the audit trail.

jeynmann · 2026-08-28T07:49:57Z

+                                                                  and @ref uct_ep_query
+                                                                  support token query. */
+        /* Error handling capabilities */
+#define UCT_IFACE_FLAG_V2_ERR_HANDLER_INPROGRESS UCS_BIT(3)  /**< Supports error handler


@evgeny-leksikov suggested:

we dont need extra UCT_IFACE_FLAG_V2_ERR_HANDLER_INPROGRESS, this is consequence of tokens + outstanding purge support

jeynmann · 2026-08-28T07:51:06Z

-                                                               support token query, and
-                                                               @ref uct_ep_outstanding_purge
-                                                               is supported. */
+#define UCT_IFACE_FLAG_V2_QUERY_TOKEN            UCS_BIT(2)  /**< @ref uct_iface_query_v2


@evgeny-leksikov suggested:

All three functionalities can be covered in single flag. Just need to mention it in doc

svc-nvidia-pr-review · 2026-08-28T08:19:12Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T08:21:01Z

🤖 Review complete — no issues found.

jeynmann · 2026-08-28T08:46:51Z

+                                                               @ref uct_iface_query_v2 and
+                                                               @ref uct_ep_query,
+                                                               outstanding purge through
+                                                               @ref uct_ep_outstanding_purge, and


Remove 'an error handler...'.

svc-nvidia-pr-review · 2026-08-28T09:21:57Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T09:22:43Z

🤖 Review complete — no issues found.

svc-nvidia-pr-review · 2026-08-28T11:21:07Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T11:22:09Z

🤖 Review complete — no issues found.

evgeny-leksikov · 2026-08-28T12:39:26Z

+ * This routine should be called only after the error handler for @a ep returns
+ * @ref UCS_INPROGRESS. It can be used only if the interface supports
+ * @c UCT_IFACE_FLAG_V2_QUERY_TOKEN.
 * @ref uct_ep_outstanding_purge_params_t::cb is invoked once for each
 * undelivered outstanding operation, in the original endpoint posting order.


Suggested change

* This routine should be called only after the error handler for @a ep returns

* @ref UCS_INPROGRESS. It can be used only if the interface supports

* @c UCT_IFACE_FLAG_V2_QUERY_TOKEN.

* @ref uct_ep_outstanding_purge_params_t::cb is invoked once for each

* undelivered outstanding operation, in the original endpoint posting order.

* @ref uct_ep_outstanding_purge_params_t::cb is invoked once for each

* undelivered outstanding operation, in the original endpoint posting order.

* @note This routine should be called only after the error handler for @a ep returns

* @ref UCS_INPROGRESS. It can be used only if the interface supports

* @c UCT_IFACE_FLAG_V2_QUERY_TOKEN.

svc-nvidia-pr-review · 2026-08-28T13:55:42Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T13:57:05Z

🤖 Review complete — no issues found.

svc-ucx · 2026-08-28T14:00:30Z

🤖 CI Triage Agent — UCX PR (Build build_aarch64 ctyunos2507_aarch64) · commit 02b879e2

TL;DR: The build_docs step of the aarch64 build failed because a new doxygen @ref uct_ep_outstanding_purge in src/uct/api/uct_def.h:570 points at a symbol doxygen doesn't know about, and doxygen is configured to treat warnings as errors. Replace the @ref with an existing documented symbol (e.g. uct_ep_pending_purge / uct_ep_flush with UCT_FLUSH_FLAG_CANCEL) or drop the @ref prefix.

Full analysis

Summary: make V=1 docs (first sub-test build_docs of the build_aarch64 ctyunos2507_aarch64 job) aborted with make: *** [Makefile:1308: docs/doxygen-doc/ucx.tag] Error 1.

Root cause: Not an environment or timing issue — a hard documentation error:

/__w/28/s/src/uct/api/uct_def.h:570: error: unable to resolve reference to
'uct_ep_outstanding_purge' for \ref command (warning treated as error, aborting now)

Commit 02b879e2 added this note to the uct_error_handler_t doc block:

 * @note If the callback returns @ref UCS_INPROGRESS, user should purge
 *       outstanding operations with @ref uct_ep_outstanding_purge.

uct_ep_outstanding_purge is not declared/documented in any of the headers in doxygen's INPUT set (src/ucp/api, src/uct/api, selected src/ucs/*) — the public purge routine documented in src/uct/api/uct.h is uct_ep_pending_purge, and outstanding-operation cancellation is expressed via uct_ep_flush() with UCT_FLUSH_FLAG_CANCEL. (The uct_ep_outstanding_purge naming appears only in the not-merged PR #11580 / the in-progress error-handler series.) Since docs/doxygen/ucxdox enables warnings-as-errors, the unresolved \ref aborts doxygen immediately.

Note the failure is platform-independent — the aarch64/ctyunos2507 leg just happens to be the one reported; every leg running build_docs with doxygen ≥ 1.8.11 will fail identically. Also note commit 53ed756e on the same branch is literally titled "UCT/API: fix doc build", and the follow-up "address comments" commit re-introduced the broken reference.

Implicated commit: [REDACTED:Hex High Entropy String] — "UCT/API: address comments", Zihao Zhao (regressing the earlier fix in 53ed756e)

File: src/uct/api/uct_def.h:569-570

Suggested fix: Point the reference at a symbol that exists in the doxygen-visible API, e.g.:

 * @note If the callback returns @ref UCS_INPROGRESS, the user is responsible for
 *       cancelling the outstanding operations on the endpoint (see
 *       @ref uct_ep_pending_purge and @ref uct_ep_flush with
 *       @ref UCT_FLUSH_FLAG_CANCEL).

If you deliberately want to mention a routine that is not yet part of the public documented API, write it without @ref (plain text or @c uct_ep_outstanding_purge). Before pushing, reproduce locally with ./contrib/configure-release --with-docs-only && make docs to confirm doxygen is clean.

Related: PR #11849 (this PR), PR #11829 (UCT/IB/MLX5: support in progress error handler), PR #11580 (UCT: Add outstanding operation extraction support — where the uct_ep_outstanding_purge name originates, not merged)

🛡️ This comment had 1 potential secret(s) redacted (Hex High Entropy String). See request_id 334933ce-4a41-4dd9-9cbe-156e233f54dc in the triage console for the audit trail.

svc-nvidia-pr-review · 2026-08-28T14:05:46Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-28T14:06:47Z

+ *       and cancel all possible outstanding operations prior closing a UCT
+ *       endpoint.
+ *
+ * @note If the callback returns @ref UCS_INPROGRESS, user should purge


minor: use @ref uct_ep_outstanding_purge here to match the @ref style used everywhere else in these headers and generate the doc link.

@jeynmann pls address

svc-nvidia-pr-review · 2026-08-28T14:06:47Z

 * undelivered outstanding operation, in the original endpoint posting order.
+ *
+ * @note This routine should be called only after the error handler for @a ep
+ *       returns @ref UCS_INPROGRESS. It can be used only if the interface


minor: use @ref UCT_IFACE_FLAG_V2_QUERY_TOKEN instead of @c for consistency with the rest of the file.

svc-nvidia-pr-review · 2026-08-31T11:19:55Z

🤖 Starting review — findings will be posted here when done.

svc-nvidia-pr-review · 2026-08-31T11:20:41Z

🤖 Review complete — no issues found.

UCT/API: Add error handler in-progress capability

018a20d

svc-nvidia-pr-review reviewed Aug 28, 2026

View reviewed changes

UCT/BASE: change default log level

089a465

svc-nvidia-pr-review reviewed Aug 28, 2026

View reviewed changes

UCT/API: rm in progress flag and revert query

599adf7

svc-nvidia-pr-review reviewed Aug 28, 2026

View reviewed changes

jeynmann commented Aug 28, 2026

View reviewed changes

jeynmann changed the title ~~UCT/API: Add error handler in-progress capability~~ UCT/API: Add ERR_HANDLER_INPROGRESS capability Aug 28, 2026

tvegas1 added the API label Aug 28, 2026

svc-ucx mentioned this pull request Aug 28, 2026

UCT/IB/MLX5: support in progress error handler #11829

Merged

UCT/API: fix doc build

53ed756

jeynmann commented Aug 28, 2026

View reviewed changes

JeynmannZ added 2 commits August 28, 2026 17:12

UCT/BASE: cleanup

49e9095

UCT/API: fix comments

341f7ad

jeynmann changed the title ~~UCT/API: Add ERR_HANDLER_INPROGRESS capability~~ UCT/API: Clarify capability requirements for purge and error callback Aug 28, 2026

jeynmann changed the title ~~UCT/API: Clarify capability requirements for purge and error callback~~ UCT/API: Clarify capability requirements for purge and error handler Aug 28, 2026

UCT/API: update comments

2c1b660

evgeny-leksikov reviewed Aug 28, 2026

View reviewed changes

UCT/API: address comments

02b879e

UCT/API: fix doc

4f96e98

svc-nvidia-pr-review reviewed Aug 28, 2026

View reviewed changes

UCT/API: address comments

945b93f

evgeny-leksikov approved these changes Aug 31, 2026

View reviewed changes

evgeny-leksikov requested a review from roiedanino August 31, 2026 13:27

roiedanino approved these changes Aug 31, 2026

View reviewed changes

roiedanino enabled auto-merge (squash) August 31, 2026 13:53

roiedanino merged commit c1eea35 into openucx:master Aug 31, 2026
162 checks passed

jeynmann deleted the failover_uct_err_handler_inprogress_cap branch September 1, 2026 01:30

This was referenced Sep 9, 2026

UCT/IB/MLX5: Purge outstanding AM short operations #11857

Merged

UCT/IB/MLX5: cleanup plugin code from iface query #11951

Open

Conversation

jeynmann commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-ucx commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

jeynmann Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

svc-ucx commented Aug 28, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 28, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

evgeny-leksikov Aug 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

svc-nvidia-pr-review commented Aug 31, 2026

Uh oh!

svc-nvidia-pr-review commented Aug 31, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

jeynmann commented Aug 28, 2026 •

edited

Loading

jeynmann Aug 28, 2026 •

edited

Loading

evgeny-leksikov Aug 31, 2026 •

edited

Loading