UCT/CUDA: Set Blackwell RTX IPC bandwidth - #11865
Conversation
|
🤖 Starting review — findings will be posted here when done. |
| UCT_CUDA_BASE_GEN_H100 = 9, | ||
| UCT_CUDA_BASE_GEN_B100 = 10 | ||
| UCT_CUDA_BASE_GEN_B100 = 10, | ||
| UCT_CUDA_BASE_GEN_BRTX = 12 |
There was a problem hiding this comment.
minor: pls add a short comment that 12 is the compute-capability major for sm_120 (consumer Blackwell) and 11 is intentionally skipped, since these enum values are matched directly against the CC major version.
|
The 800000.0 * UCS_MBYTE value for the NVLink-present BRTX case mirrors the B100 non-NVSwitch estimate. This is a data/tuning choice and not a blocker. Worth confirming the bandwidth values were validated on actual Blackwell RTX hardware (with and without NVLink). Non-blocking (pre-existing): the num_nvlinks static cache does not memoize the failure (goto err returns 0 without setting num_nvlinks), so each call re-probes NVML on failure. This is unchanged by the code move and out of scope for this PR. No test is added. The behavior is a hardware-generation-specific bandwidth estimate requiring sm_120 hardware to exercise, so automated coverage is not practical. |
2ec0371 to
2314cd9
Compare
|
🤖 Starting review — findings will be posted here when done. |
| UCT_CUDA_BASE_GEN_H100 = 9, | ||
| UCT_CUDA_BASE_GEN_B100 = 10 | ||
| UCT_CUDA_BASE_GEN_B100 = 10, | ||
| UCT_CUDA_BASE_GEN_BRTX = 12 |
There was a problem hiding this comment.
minor: can we add a short comment that these values are the CUDA compute-capability major version? The jump from 10 to 12 (skipping 11) looks like a typo otherwise, though it correctly matches sm_120 for Blackwell RTX.
|
minor: why move Test coverage: No test is added. The change is a hardware-generation-specific bandwidth constant plus a code move; exercising it requires sm_120 (Blackwell RTX) hardware, so automated coverage is not practical. The |
2314cd9 to
cf06cf6
Compare
|
🤖 Starting review — findings will be posted here when done. |
|
🤖 Review complete — no issues found. |
WXD/IBM ONLY — DO NOT UPSTREAM OR OPEN AS AN UPSTREAM PR. The measured AWS configuration has lived in the test launcher. A consumer building Velox through Presto's dependency tree therefore gets different reader, cache, exchange and batching behavior from the benchmarked system. Make the validated compressed baseline the branch default: KvikIO with AsyncDataCache and registered host slabs, the main async GPU allocator, the measured operator/batch settings, and adaptive compression permitted on CUDA IPC endpoints. Supply missing UCX and KvikIO environment defaults before their initialization, without overriding explicit settings, and log the effective profile at startup. Use mixed SRD/IPC with one rendezvous rail and the measured fragment/cache bounds. Select KvikIO's MULTI_POLL backend, 128 concurrent request slots per process, four reactors, PER_CHUNK dispatch, 32-MiB tasks, a 16-thread worker pool and strict direct receive. The request limit is not a thread count. Pin the CentOS adapter installer to CUDA 13.2 and the published WXD UCX 1.22 revision containing openucx/ucx#11865 and its GDA build prerequisites. Keep the raw/PULL IPC, exportable exchange-pool, private-async access and adaptive flow-control experiments out. This does not resolve the raw IPC stall or SF3K memory failures, and does not claim an IPC speedup. Document the dedicated-host assumptions, large memory budgets, reduced fault tolerance and deployment settings that remain external to the profile. Test the defaults, reader/property overrides, every startup environment default, backend overrides, thread-count alias, independent exchange/S3 enablement, and installer failure propagation. Log the effective backend, request window, task size and thread settings so an unset launcher environment does not hide the profile actually used by the worker.
4 participants
What?
Set the CUDA IPC bandwidth estimate for RTX Pro 6000 Blackwell (CUDA capability 12.0, no NVLink support) GPUs to 50GB/s (conservative PCIe 5.0 bandwidth).
Why?
The previous fallback estimate of 6911MB/s significantly underestimated PCIe device-to-device bandwidth on Blackwell RTX GPUs. This can cause protocol selection to prefer a GPU-staging pipeline instead of CUDA IPC put zcopy, reducing observed bandwidth substantially.
Below are before and after on a g7e.12xlarge instance:
Before (UCX_TLS=all)
Before (UCX_TLS=^srd)
After (UCX_MAX_RNDV_RAILS=2)
After (UCX_MAX_RNDV_RAILS=1)
How?
Identify Blackwell RTX GPUs by CUDA compute capability major version 12.