Skip to content

UCT/GDAKI: Revert - Check for HCA reachability like rc_mlx5 - v1.23.x - #11839

Closed
ofirfarjun7 wants to merge 1 commit into
openucx:v1.23.xfrom
ofirfarjun7:topic/revert-gdaki-reachability-1.23.x
Closed

ofirfarjun7 wants to merge 1 commit into
openucx:v1.23.xfrom
ofirfarjun7:topic/revert-gdaki-reachability-1.23.x

Conversation

@ofirfarjun7

Copy link
Copy Markdown
Contributor

What?

Revert #11801

Why?

Reachability check of ib can be wrong for gda

@ofirfarjun7 ofirfarjun7 changed the title UCT/GDAKI: Revert - Check for HCA reachability like rc_mlx5 (#11781) UCT/GDAKI: Revert - Check for HCA reachability like rc_mlx5 - v1.23.x Aug 26, 2026
@svc-nvidia-pr-review

Copy link
Copy Markdown

🤖 Starting review — findings will be posted here when done.

.ep_invalidate = (uct_ep_invalidate_func_t)ucs_empty_function_return_unsupported,
.ep_connect_to_ep_v2 = uct_rc_gdaki_ep_connect_to_ep_v2,
.iface_is_reachable_v2 = uct_ib_iface_is_reachable_v2,
.iface_is_reachable_v2 = (uct_iface_is_reachable_v2_func_t)ucs_empty_function_return_one_int,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why switch gda from uct_ib_iface_is_reachable_v2 to always-reachable? gda still uses IB device addresses, and the old callback rejected peers on a different link layer/subnet/pkey and enforced device-scope reachability (used by ucp wireup select). With this stub uct_iface_is_reachable_v2() reports any remote address as reachable, so a gda lane can be selected toward an unreachable peer and fail only at connect. pls add a comment explaining why dropping the check is safe here.

@svc-nvidia-pr-review

Copy link
Copy Markdown

test/gtest/uct/test_uct_iface.cc

this removes the only reachability coverage for gda. The is_reachable test expects some corrupted addresses to be unreachable, which can't hold once the callback always returns 1. is gda reachability / ep_is_connected device matching covered elsewhere? if not, pls keep a gda case instead of removing it.

@svc-ucx

svc-ucx commented Aug 27, 2026

Copy link
Copy Markdown

🤖 CI Triage AgentUCX PR (AddressSanitizer BlueField on worker 1) · commit 2c66bd38

TL;DR: All 8627 gtests passed; the job failed only because LeakSanitizer reported a 131-byte "indirect leak" from glibc's __dlerror() buffer materialized by the pointless (void)dlerror() call in ucs_sys_get_lib_info() (src/ucs/sys/lib.c:18) during ucs_profile_cleanup. This is unrelated to the gdaki-reachability revert in PR #11839 — fix by removing that dlerror() call and/or adding a leak:__dlerror entry to contrib/lsan.supp.

Full analysis

Summary: make test in the AddressSanitizer/BlueField job exited with error 1 after the gtest run completed successfully, due to a LeakSanitizer report (SUMMARY: AddressSanitizer: 131 byte(s) leaked in 1 allocation(s)).

Root cause: The test run itself is clean — [ PASSED ] 8627 tests, no failures, no timeouts, continuous log activity to the end (largest gap ≈7 s, so no hang). The only error is at process teardown:

Indirect leak of 131 byte(s) in 1 object(s) allocated from:
  #1 __vasprintf_internal libio/vasprintf.c:71
  #2 ___asprintf stdio-common/asprintf.c:31
  #3 __dlerror dlfcn/dlerror.c:74
  #4 ucs_sys_get_lib_info  src/ucs/sys/lib.c:18
  #5 ucs_sys_get_lib_path  src/ucs/sys/lib.c:32
  #6 ucs_profile_write     src/ucs/profile/profile.c:322
  #7 ucs_profile_dump / #8 ucs_profile_cleanup
  #9 scoped_profile::~scoped_profile()  test/gtest/ucs/test_profile.cc:48
 #10 test_profile.perf_overhead          test/gtest/ucs/test_profile.cc:482

ucs_sys_get_lib_info() calls (void)dlerror() before dladdr(). That call is dead code — the result is discarded and dladdr() failure is detected via its return value — but it has a side effect: if any earlier dlopen()/dlsym() on the same thread left a pending error (very common on this platform, where optional plugins/CUDA/DOCA/GGA libraries are probed; note Suppressions used: 1 24 dlsym in the log), glibc asprintf()s the message into thread-local last_result storage. That buffer is only freed by the next dlerror() call or thread exit, so LSan sees it as leaked at exit().

contrib/lsan.supp suppresses dlopen and dlsym frames but has no entry for __dlerror, so this escapes suppression and fails the build. Because it depends on (a) a preceding failed dlopen/dlsym and (b) gtest's randomized test order putting test_profile.perf_overhead in the right place, it is intermittent — a flaky infrastructure failure, not a regression from this PR (the PR only reverts the GDAKI HCA-reachability check; src/ucs/sys/lib.c has not been touched since 2022 and nothing in the profile/dlerror path is related).

Implicated commit: Not a regression from 2c66bd38 (PR #11839). Pre-existing code: bca0bf5b / original 125542b1 (Leonid Genkin) for src/ucs/sys/lib.c; exposure increased by 49a0d4c1 "UCS/SYS: Added support for dynamically loaded external modules/plugins (#11206)" (Roie Danino), which added the dlopen/dlsym plugin probing and the corresponding lsan.supp entries.

File: src/ucs/sys/lib.c:18 (and contrib/lsan.supp:1-7)

Suggested fix:

  1. Remove the no-op (void)dlerror(); at src/ucs/sys/lib.c:18 — it is never read and only forces glibc to allocate/format a stale pending error string:
    ucs_status_t ucs_sys_get_lib_info(Dl_info *dl_info)
    {
        if (dladdr(ucs_sys_get_lib_info, dl_info) == 0) {
            return UCS_ERR_NO_MEMORY;
        }
        return UCS_OK;
    }
  2. Belt-and-braces: add leak:__dlerror (or leak:dlerror) to contrib/lsan.supp alongside the existing dlopen/dlsym entries, since any other dlerror() consumer can hit the same glibc artifact.
  3. Meanwhile, this build should simply be re-run — the failure does not indicate a problem with the revert in PR UCT/GDAKI: Revert - Check for HCA reachability like rc_mlx5 - v1.23.x #11839.

Related: PR #11839 (this build), sibling reverts #11838 / #11819; suppression-list history: #11206 (added dlopen/dlsym suppressions), commit 311fa48b (added ibv_alloc_pd suppression for BF/GGA), f675ed50 (added the BlueField ASAN job).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants