Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 20 additions & 2 deletions docs/user/discovery.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,8 +27,12 @@ Before waiting for `NicDevice` resources, Launch Kit checks BlueField trust
mode with `mlxprivhost`. A restricted BlueField is excluded from the wait set.
If the same node has another non-restricted NVIDIA NIC, that node remains in
the wait set and its published devices are discovered normally. A failed or
unrecognized trust query is treated as non-restricted, matching NIC
Configuration Operator behavior.
unrecognized trust query leaves the device's restriction status unknown and
keeps it eligible, matching NIC Configuration Operator behavior. A later
device-query error can still abort daemon startup; readable peers are not
guaranteed to publish. Launch Kit does not enable
`SKIP_DEVICE_ON_DISCOVERY_ERROR`: that daemon option can omit any failed NIC,
and its skipped-device list is not exposed through `NicDevice` status.

## Run discovery

Expand Down Expand Up @@ -74,6 +78,20 @@ If a worker is missing, check its readiness, image pull, discovery pod,

The bootstrap namespace is fixed. `--network-operator-namespace` is accepted for compatibility but ignored by discovery.

The temporary daemon needs cluster-wide ConfigMap `get`, `list`, and `watch`
permissions for its informer cache, including on nodes without NICs. Its cache
filters ConfigMaps by label; Kubernetes RBAC grants these read permissions
across namespaces. Bootstrap grants no ConfigMap write permissions.

Pod readiness is a startup snapshot, not proof of device publication. During
the `NicDevice` wait, discovery reports failing bootstrap containers on missing
publishers and keeps waiting for recovery. Repeated unchanged failures are
reported once. Progress counts only expected publishers; timeout errors name
the last missing nodes. Eligibility probe failures identify the unverified
node/pod; they can reduce the wait set and leave inventory incomplete. Retain the
namespace with `--keep-namespace` and inspect current and previous daemon logs
as described in [Troubleshooting](troubleshooting.md#discovery-failures).

Discovery reports each confirmed host-restricted BlueField PCI function with its
node and PCI address in normal CLI output, before deleting the bootstrap namespace.
The NIC-probe summary logs `restrictedDevicesSkipped`; this counts PCI functions,
Expand Down
16 changes: 16 additions & 0 deletions docs/user/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,6 +91,7 @@ kubectl get events -n nvidia-k8s-launch-kit \
--sort-by=.metadata.creationTimestamp
kubectl describe pod -n nvidia-k8s-launch-kit <pod-name>
kubectl logs -n nvidia-k8s-launch-kit <pod-name>
kubectl logs -n nvidia-k8s-launch-kit <pod-name> --previous
```

Common causes include an unavailable daemon image, a missing image pull secret,
Expand All @@ -100,6 +101,21 @@ sysfs, or a cluster where every detected BlueField is in zero-trust
NIC Configuration Operator does not publish `NicDevice` resources for them.
NFD labels are not required for discovery scheduling.

An initially Ready daemon can subsequently crash before publishing devices.
Discovery reports container failure reasons for pending publishers while
continuing to wait; a restart alone is not a permanent failure. If an eligibility
probe cannot execute, its node/pod warning explains why that node was excluded
from the wait set. Check these warnings when the expected node count is smaller
than the hardware inventory. Pod-status inspection failures leave health unknown.

A failed `mlxprivhost` query does not establish host restriction. If it is
followed by a failing BlueField operation-mode query, collect both command
outputs and the daemon's MFT and device firmware versions. Bootstrap does not
enable automatic skipping of discovery errors, which could also omit failed
ConnectX NICs without a completeness signal. ConfigMap `list/watch` denials
indicate bootstrap RBAC incompatible with the daemon's cluster-wide cache;
use a Launch Kit version with those read permissions.

## Namespace Mismatch

Generation, deployment, and validation must agree on the Network Operator namespace:
Expand Down
6 changes: 6 additions & 0 deletions openspec/specs/discovery-inventory/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,12 @@ A discovery extension MUST identify eligible sources and distinguish a missing o
- **THEN** normal CLI output identifies their node and PCI addresses and reports a PCI-function count before namespace teardown; those devices remain excluded from saved inventory.
- **AND** a later PCI scan failure retains an already-verified publisher and warns that exclusion reporting is incomplete; read-only eligibility probes retain their existing failure policy.

#### Scenario: Bootstrap publisher fails after pod readiness

- **WHEN** a bootstrap publisher has not published a `NicDevice` and its container enters a failure state after the initial readiness snapshot
- **THEN** discovery refreshes bootstrap pod status and reports the affected node/pod and failure reason without treating a transient restart as permanent failure; unchanged failures are not repeated each poll.
- **AND** failed eligibility probes identify unverified excluded nodes and warn of incomplete inventory; probe failure is not evidence of absent NICs or host restriction. Read-only waiters do not acquire bootstrap status inspection.

#### Scenario: Read-only publisher eligibility is unknown by default

- **WHEN** a Ready existing-daemon node has no `NicDevice` and discovery cannot inspect its NIC eligibility
Expand Down
120 changes: 120 additions & 0 deletions pkg/networkoperatorplugin/discovery/daemonhealth.go
Original file line number Diff line number Diff line change
@@ -0,0 +1,120 @@
// Copyright 2026 NVIDIA CORPORATION & AFFILIATES
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//
// SPDX-License-Identifier: Apache-2.0

package discovery

import (
"context"
"fmt"
"slices"
"strings"
"time"

"github.com/nvidia/k8s-launch-kit/pkg/nicconfigdaemon"
"github.com/nvidia/k8s-launch-kit/pkg/ui"
corev1 "k8s.io/api/core/v1"
"sigs.k8s.io/controller-runtime/pkg/client"
)

// newBootstrapDaemonObserver reports failure evidence only for nodes still
// missing NicDevices. Pod readiness was a snapshot before the sysfs probes;
// later restarts are diagnostics, not proof that a publisher cannot recover.
func newBootstrapDaemonObserver(c client.Client, output ui.Output) func(context.Context, []string) {
previous := map[string]string{}
lastInspectionError := ""
showedHint := false
return func(ctx context.Context, pendingNodes []string) {
if len(pendingNodes) == 0 {
return
}
inspectionCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
pods := &corev1.PodList{}
if err := c.List(inspectionCtx, pods, client.InNamespace(nicconfigdaemon.Namespace)); err != nil {
if message := err.Error(); message != lastInspectionError {
output.Warning("Could not inspect discovery daemon health while waiting for NicDevice resources: %v; daemon health is unverified", err)
lastInspectionError = message
}
return
}
lastInspectionError = ""
pending := make(map[string]bool, len(pendingNodes))
for _, node := range pendingNodes {
pending[node] = true
}
slices.SortFunc(pods.Items, func(a, b corev1.Pod) int { return strings.Compare(a.Name, b.Name) })
current := map[string]string{}
for _, pod := range pods.Items {
if !pending[pod.Spec.NodeName] || !bootstrapDaemonPod(pod) {
continue
}
statuses := append([]corev1.ContainerStatus{}, pod.Status.InitContainerStatuses...)
statuses = append(statuses, pod.Status.ContainerStatuses...)
for _, status := range statuses {
failure := daemonContainerFailure(status)
if failure == "" {
continue
}
key := string(pod.UID) + "/" + pod.Name + "/" + status.Name
current[key] = failure
if previous[key] == failure {
continue
}
output.Warning("Discovery daemon on %s, pod %s, container %s: %s (restarts: %d); NicDevice publication is still pending",
pod.Spec.NodeName, pod.Name, status.Name, failure, status.RestartCount)
if !showedHint {
output.Warning("Inspect daemon logs with kubectl logs -n %s %s -c %s (add --previous for the prior container); re-run discovery with --keep-namespace to retain bootstrap resources",
nicconfigdaemon.Namespace, pod.Name, status.Name)
showedHint = true
}
}
}
previous = current
}
}

func bootstrapDaemonPod(pod corev1.Pod) bool {
if pod.Namespace != nicconfigdaemon.Namespace {
return false
}
for _, owner := range pod.OwnerReferences {
if owner.Kind == "DaemonSet" && owner.Name == nicconfigdaemon.DaemonSetName {
return true
}
}
return false
}

func daemonContainerFailure(status corev1.ContainerStatus) string {
termination := status.State.Terminated
if waiting := status.State.Waiting; waiting != nil && stuckWaitingReasons[waiting.Reason] {
failure := "waiting: " + waiting.Reason
if last := status.LastTerminationState.Terminated; last != nil {
failure += fmt.Sprintf("; last termination: %s (exit code %d)", last.Reason, last.ExitCode)
}
return failure
}
if termination != nil {
if termination.ExitCode == 0 {
return ""
}
return fmt.Sprintf("terminated: %s (exit code %d)", termination.Reason, termination.ExitCode)
}
if last := status.LastTerminationState.Terminated; status.RestartCount > 0 && last != nil && last.ExitCode != 0 {
return fmt.Sprintf("restarted; last termination: %s (exit code %d)", last.Reason, last.ExitCode)
}
return ""
}
Loading