Skip to content

l8k discover cannot complete when a BlueField-3 denies host NV access and the trust query fails; bootstrap daemon crash-loops #408

Open

Description

@atif1996

What happened?

l8k discover (built from main at 6c54e0f) cannot complete on GB300 nodes at a cloud partner. The bootstrap nic-configuration-daemon crash-loops on every GPU node, no NicDevice objects are published, and no cluster-config.yaml is written.

Each GPU node has four ConnectX-8 functions and one BlueField-3 (MT43244 BlueField-3 integrated ConnectX-7 network controller). The provider operates the BlueField-3, and host access to it is disabled.

l8k discover --kubeconfig "$KUBECONFIG" \
  --fabric infiniband --deployment-type rdma_shared \
  --save-cluster-config ./cluster-config.yaml

Output (repeated progress lines removed):

Restricting discovery daemon to 8 Ready schedulable node(s)
Bootstrapping NIC configuration daemon in namespace "nvidia-k8s-launch-kit"
✓ Found 8 ready pod(s) in namespace "nvidia-k8s-launch-kit"
  Discovered devices on 0/1 node(s)...

Bootstrap daemon on a GPU node (nic-configuration-operator-daemon:network-operator-v26.7.1). The four ConnectX-8 functions are read successfully, then:

INFO   devicediscovery/devicediscovery.go:103  Found Mellanox device  {"address": "0016:01:00.0", "type": "MT43244 BlueField-3 integrated ConnectX-7 network controller"}
ERROR  devicediscovery/utils.go:369  IsZeroTrust(): Failed to run mlxprivhost  {"error": "exit status 1"}
ERROR  devicediscovery/devicediscovery.go:121  Failed to get device's zero-trust (host restriction) status. Proceed on premise of non-zero-trust  {"address": "0016:01:00.0", "error": "exit status 1"}
INFO   nvconfig/utils.go:89  mlxconfig -d /dev/fwctl/fwctl4 query INTERNAL_CPU_OFFLOAD_ENGINE
ERROR  nvconfig/utils.go:101  queryMLXConfig(): Failed to run mlxconfig  {"output": "-E- NVQC access register is not supported. Try a different type of access to the device. rc=(262)\n", "error": "exit status 3"}
ERROR  devicediscovery/devicediscovery.go:154  Failed to get BlueField device's operation mode  {"address": "0016:01:00.0", "error": "exit status 3"}
ERROR  nic-configuration-daemon/main.go:131  failed to discover NIC devices  {"error": "exit status 3"}

Problems

  1. The restricted-BlueField exclusion does not catch this device. fix: exclude restricted BlueFields from discovery #246 and fix(discovery): report host-restricted BlueField exclusions #341 exclude a BlueField whose trust level reads as restricted. Here the trust query itself fails, and the docs say an unrecognized trust query is treated as non-restricted, so the node stays in the wait set and the daemon aborts on the operation-mode query.
  2. One unreadable device hides the readable ones. The daemon exits before publishing anything, so the four ConnectX-8 functions it already read are lost.
  3. The failure is not diagnosed. The pods are counted once as ready (Found 8 ready pod(s)), then crash-loop. Discovery reports only Discovered devices on 0/1 node(s)... until the 10 minute NicDevice wait expires. Nothing says the daemon is crash-looping or why.
  4. The expected set was 1 node on a cluster with 4 GPU nodes that each carry ConnectX-8 NICs. I did not establish why.
  5. Bootstrap daemon RBAC on nodes without NICs. On CPU nodes the daemon loops on configmaps is forbidden: User "system:serviceaccount:nvidia-k8s-launch-kit:k8s-launch-kit-nic-config-daemon" cannot list resource "configmaps" in API group "" at the cluster scope.

What I expected

Discovery should complete with the ConnectX-8 inventory and report the BlueField-3 as excluded, the same way it reports a host-restricted BlueField today. At cloud partners a BlueField the host cannot query is the normal case.

Possible fixes:

  • Treat a failed trust query on a BlueField as host-restricted in the sysfs probe, and exclude it.
  • Set SKIP_DEVICE_ON_DISCOVERY_ERROR=true on the bootstrap daemon. Skip device on discovery errors when enabled Mellanox/nic-configuration-operator#407 added it for exactly the operation-mode lookup failure, and it is in the network-operator-v26.7.0 and later tags. The bootstrap DaemonSet sets only NODE_NAME, NAMESPACE and LOG_LEVEL.
  • When bootstrap pods restart during the NicDevice wait, fail early and print the last daemon log line.

The daemon-side default is tracked in Mellanox/nic-configuration-operator#601.

Environment

  • l8k built from main at 6c54e0f
  • Bootstrap daemon image: nvcr.io/nvidia/mellanox/nic-configuration-operator-daemon:network-operator-v26.7.1
  • Kubernetes 1.36.6, arm64, kernel 6.17.0-1032-nvidia-64k
  • 4 GPU nodes (GB300) and 4 CPU nodes; per GPU node 4 x ConnectX-8 functions (InfiniBand, firmware 40.50.1002) and 1 x BlueField-3
  • I interrupted the run at about 10 minutes, shortly before the NicDevice wait would have timed out.

Activity

  1. almaslennikov commented on Nov 20, 2026

    @almaslennikov
    Collaborator

    Verified the reported Launch Kit revision and network-operator-v26.7.1 daemon source. Draft #409 fixes the ConfigMap cache RBAC and exposes failed eligibility probes and later daemon failures while preserving recovery. It does not yet restore readable-peer inventory after a BlueField startup error. #246/#341 cover confirmed restriction, whereas a failed trust query remains unknown; enabling skip-on-error alone can also omit failed ConnectX NICs or preserve stale CRs.

    Could you share the discovery debug output from a run with --keep-namespace --log-level debug, together with each GPU daemon pod's node name, restart count, current container state and last termination state? The code drops nodes whose exec probe fails before verifying a usable NIC; this evidence is needed to explain the four-GPU-node to one-publisher reduction. The trust-query details already requested in Mellanox/nic-configuration-operator#601 remain necessary before changing restriction detection.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions