#11547 concentrates multi-rank GPU RMA traffic onto a subset of NICs (4 of 8), causing severe oversubscription and multi-× bandwidth loss

[bosilca]

While working with #11677 and #11637 I noticed a significant performance drop on my MoE tester. The drop was so significant that it could not have come from the SGL lists, so I tracked it down to 38c99d4

Environment

Node: 8× H100 + 8× NIC (HGX/DGX-class), NDR IB; one GPU per rank via CUDA_VISIBLE_DEVICES; mpirun, 8 ranks/node, inside a container
UCX: current master (includes 38c99d4 "UCP/PROTO: Distribute GPU-NIC matching across equal-bw lanes (#11547)")
NVML verified present/functional in the container: nvidia-smi -L and a direct dlopen("libnvidia-ml.so.1") + nvmlInit_v2 + nvmlDeviceGetCount_v2 probe both report 8 GPUs

Symptom Per-rank large-message GPU RMA bandwidth drops multi-× vs. before #11547. The application sends balanced data to all peers, yet UCX funnels all traffic through only 4 of 8 HCAs; the other 4 are ~200× idle.

Per-NIC port_xmit_data deltas, same workload, default (broken) vs. per-rank UCX_NET_DEVICES pinning:

HCA default (broken) pinned (fixed)
mlx5_0 231.5 GB 133.5 GB
mlx5_3 1.05 GB 133.5 GB
mlx5_4 249.0 GB 137.6 GB
mlx5_5 0.90 GB 134.5 GB
mlx5_6 224.4 GB 136.6 GB
mlx5_9 1.04 GB 128.4 GB
mlx5_10 237.5 GB 123.3 GB
mlx5_11 0.92 GB 127.4 GB

In the default case exactly 4 HCAs carry traffic (2 ranks each) and 4 are idle. Reverting 38c99d4, or pinning UCX_NET_DEVICES=:1 per rank, balances all 8 HCAs and restores full bandwidth.

Root cause ucp_proto_multi_find_max_avail_bw_lane() (src/ucp/proto/proto_multi.c) distributes the selected lane among equal-bandwidth NICs:

seed = req_sys_dev_ord % num_max_bw_devs; // GPU BDF ordinal
selected_sys_dev = sys_devs[seed]; // tied NICs, sorted by bus id
NVML enumeration (uct_cuda_enum_gpus) works here, so req_sys_dev_ord is a correct global GPU ordinal (0..7) — ordinals are not collapsing. Even so, the distribution maps the 8 ranks onto only 4 distinct NICs: the tied max-bw set resolves to num_max_bw_devs == 4 (the same 4 NICs for every GPU), so gpu_ord % 4 collides GPUs {0,4}, {1,5}, {2,6}, {3,7} onto shared HCAs and leaves the other 4 unused → 2:1 oversubscription (worse per-rank under contention).

Reproduction / evidence

nvidia-smi -L and NVML probe → 8 GPUs (rules out NVML/container GPU-hiding).
Per-NIC port_xmit_data deltas above (4 saturated / 4 idle by default; all 8 balanced when pinned).
Reverting 38c99d4 → all 8 HCAs used, full bandwidth.
UCX_NET_DEVICES=:1 per rank → all 8 HCAs used, full bandwidth.

Suggested fixes

• Guarantee distinct NIC assignment for distinct requesting devices when enough tied NICs exist; gpu_ord % num_max_bw_devs collides ranks whenever num_max_bw_devs < ranks_sharing_the_tied_set. Prefer the requesting GPU's topologically-affine NIC before falling back to round-robin.
• Investigate why the tied max-bw set resolves to a 4-NIC subset rather than all 8 on a symmetric topology — the distance→bandwidth fp8 quantization may be under-counting distinct devices.
• Provide a config opt-out to restore the previous per-rank affine selection.

[guy-ealey-morag]

Hi @bosilca, can you please provide trace level logs? UCX_LOG_LEVEL=trace

[guy-ealey-morag]

A possible fix can be found here: #11750
Please let me know when you will be able to test it