Repository navigation
Description
What happened?
l8k discover (built from main at 6c54e0f) cannot complete on GB300 nodes at a cloud partner. The bootstrap nic-configuration-daemon crash-loops on every GPU node, no NicDevice objects are published, and no cluster-config.yaml is written.
Each GPU node has four ConnectX-8 functions and one BlueField-3 (MT43244 BlueField-3 integrated ConnectX-7 network controller). The provider operates the BlueField-3, and host access to it is disabled.
l8k discover --kubeconfig "$KUBECONFIG" \
--fabric infiniband --deployment-type rdma_shared \
--save-cluster-config ./cluster-config.yaml
Output (repeated progress lines removed):
Restricting discovery daemon to 8 Ready schedulable node(s)
Bootstrapping NIC configuration daemon in namespace "nvidia-k8s-launch-kit"
✓ Found 8 ready pod(s) in namespace "nvidia-k8s-launch-kit"
Discovered devices on 0/1 node(s)...
Bootstrap daemon on a GPU node (nic-configuration-operator-daemon:network-operator-v26.7.1). The four ConnectX-8 functions are read successfully, then:
INFO devicediscovery/devicediscovery.go:103 Found Mellanox device {"address": "0016:01:00.0", "type": "MT43244 BlueField-3 integrated ConnectX-7 network controller"}
ERROR devicediscovery/utils.go:369 IsZeroTrust(): Failed to run mlxprivhost {"error": "exit status 1"}
ERROR devicediscovery/devicediscovery.go:121 Failed to get device's zero-trust (host restriction) status. Proceed on premise of non-zero-trust {"address": "0016:01:00.0", "error": "exit status 1"}
INFO nvconfig/utils.go:89 mlxconfig -d /dev/fwctl/fwctl4 query INTERNAL_CPU_OFFLOAD_ENGINE
ERROR nvconfig/utils.go:101 queryMLXConfig(): Failed to run mlxconfig {"output": "-E- NVQC access register is not supported. Try a different type of access to the device. rc=(262)\n", "error": "exit status 3"}
ERROR devicediscovery/devicediscovery.go:154 Failed to get BlueField device's operation mode {"address": "0016:01:00.0", "error": "exit status 3"}
ERROR nic-configuration-daemon/main.go:131 failed to discover NIC devices {"error": "exit status 3"}
Problems
- The restricted-BlueField exclusion does not catch this device. fix: exclude restricted BlueFields from discovery #246 and fix(discovery): report host-restricted BlueField exclusions #341 exclude a BlueField whose trust level reads as
restricted. Here the trust query itself fails, and the docs say an unrecognized trust query is treated as non-restricted, so the node stays in the wait set and the daemon aborts on the operation-mode query. - One unreadable device hides the readable ones. The daemon exits before publishing anything, so the four ConnectX-8 functions it already read are lost.
- The failure is not diagnosed. The pods are counted once as ready (
Found 8 ready pod(s)), then crash-loop. Discovery reports onlyDiscovered devices on 0/1 node(s)...until the 10 minuteNicDevicewait expires. Nothing says the daemon is crash-looping or why. - The expected set was 1 node on a cluster with 4 GPU nodes that each carry ConnectX-8 NICs. I did not establish why.
- Bootstrap daemon RBAC on nodes without NICs. On CPU nodes the daemon loops on
configmaps is forbidden: User "system:serviceaccount:nvidia-k8s-launch-kit:k8s-launch-kit-nic-config-daemon" cannot list resource "configmaps" in API group "" at the cluster scope.
What I expected
Discovery should complete with the ConnectX-8 inventory and report the BlueField-3 as excluded, the same way it reports a host-restricted BlueField today. At cloud partners a BlueField the host cannot query is the normal case.
Possible fixes:
- Treat a failed trust query on a BlueField as host-restricted in the sysfs probe, and exclude it.
- Set
SKIP_DEVICE_ON_DISCOVERY_ERROR=trueon the bootstrap daemon. Skip device on discovery errors when enabled Mellanox/nic-configuration-operator#407 added it for exactly the operation-mode lookup failure, and it is in thenetwork-operator-v26.7.0and later tags. The bootstrap DaemonSet sets onlyNODE_NAME,NAMESPACEandLOG_LEVEL. - When bootstrap pods restart during the
NicDevicewait, fail early and print the last daemon log line.
The daemon-side default is tracked in Mellanox/nic-configuration-operator#601.
Environment
l8kbuilt frommainat6c54e0f- Bootstrap daemon image:
nvcr.io/nvidia/mellanox/nic-configuration-operator-daemon:network-operator-v26.7.1 - Kubernetes 1.36.6, arm64, kernel
6.17.0-1032-nvidia-64k - 4 GPU nodes (GB300) and 4 CPU nodes; per GPU node 4 x ConnectX-8 functions (InfiniBand, firmware
40.50.1002) and 1 x BlueField-3 - I interrupted the run at about 10 minutes, shortly before the
NicDevicewait would have timed out.
Activity
almaslennikov commentedon Nov 20, 2026
Verified the reported Launch Kit revision and
network-operator-v26.7.1daemon source. Draft #409 fixes the ConfigMap cache RBAC and exposes failed eligibility probes and later daemon failures while preserving recovery. It does not yet restore readable-peer inventory after a BlueField startup error. #246/#341 cover confirmed restriction, whereas a failed trust query remains unknown; enabling skip-on-error alone can also omit failed ConnectX NICs or preserve stale CRs.Could you share the discovery debug output from a run with
--keep-namespace --log-level debug, together with each GPU daemon pod's node name, restart count, current container state and last termination state? The code drops nodes whose exec probe fails before verifying a usable NIC; this evidence is needed to explain the four-GPU-node to one-publisher reduction. The trust-query details already requested in Mellanox/nic-configuration-operator#601 remain necessary before changing restriction detection.