diff --git a/docs/user/discovery.md b/docs/user/discovery.md index f19c85f7..220241d7 100644 --- a/docs/user/discovery.md +++ b/docs/user/discovery.md @@ -27,8 +27,12 @@ Before waiting for `NicDevice` resources, Launch Kit checks BlueField trust mode with `mlxprivhost`. A restricted BlueField is excluded from the wait set. If the same node has another non-restricted NVIDIA NIC, that node remains in the wait set and its published devices are discovered normally. A failed or -unrecognized trust query is treated as non-restricted, matching NIC -Configuration Operator behavior. +unrecognized trust query leaves the device's restriction status unknown and +keeps it eligible, matching NIC Configuration Operator behavior. A later +device-query error can still abort daemon startup; readable peers are not +guaranteed to publish. Launch Kit does not enable +`SKIP_DEVICE_ON_DISCOVERY_ERROR`: that daemon option can omit any failed NIC, +and its skipped-device list is not exposed through `NicDevice` status. ## Run discovery @@ -74,6 +78,20 @@ If a worker is missing, check its readiness, image pull, discovery pod, The bootstrap namespace is fixed. `--network-operator-namespace` is accepted for compatibility but ignored by discovery. +The temporary daemon needs cluster-wide ConfigMap `get`, `list`, and `watch` +permissions for its informer cache, including on nodes without NICs. Its cache +filters ConfigMaps by label; Kubernetes RBAC grants these read permissions +across namespaces. Bootstrap grants no ConfigMap write permissions. + +Pod readiness is a startup snapshot, not proof of device publication. During +the `NicDevice` wait, discovery reports failing bootstrap containers on missing +publishers and keeps waiting for recovery. Repeated unchanged failures are +reported once. Progress counts only expected publishers; timeout errors name +the last missing nodes. Eligibility probe failures identify the unverified +node/pod; they can reduce the wait set and leave inventory incomplete. Retain the +namespace with `--keep-namespace` and inspect current and previous daemon logs +as described in [Troubleshooting](troubleshooting.md#discovery-failures). + Discovery reports each confirmed host-restricted BlueField PCI function with its node and PCI address in normal CLI output, before deleting the bootstrap namespace. The NIC-probe summary logs `restrictedDevicesSkipped`; this counts PCI functions, diff --git a/docs/user/troubleshooting.md b/docs/user/troubleshooting.md index 6471bf1f..634b0a4e 100644 --- a/docs/user/troubleshooting.md +++ b/docs/user/troubleshooting.md @@ -91,6 +91,7 @@ kubectl get events -n nvidia-k8s-launch-kit \ --sort-by=.metadata.creationTimestamp kubectl describe pod -n nvidia-k8s-launch-kit kubectl logs -n nvidia-k8s-launch-kit +kubectl logs -n nvidia-k8s-launch-kit --previous ``` Common causes include an unavailable daemon image, a missing image pull secret, @@ -100,6 +101,21 @@ sysfs, or a cluster where every detected BlueField is in zero-trust NIC Configuration Operator does not publish `NicDevice` resources for them. NFD labels are not required for discovery scheduling. +An initially Ready daemon can subsequently crash before publishing devices. +Discovery reports container failure reasons for pending publishers while +continuing to wait; a restart alone is not a permanent failure. If an eligibility +probe cannot execute, its node/pod warning explains why that node was excluded +from the wait set. Check these warnings when the expected node count is smaller +than the hardware inventory. Pod-status inspection failures leave health unknown. + +A failed `mlxprivhost` query does not establish host restriction. If it is +followed by a failing BlueField operation-mode query, collect both command +outputs and the daemon's MFT and device firmware versions. Bootstrap does not +enable automatic skipping of discovery errors, which could also omit failed +ConnectX NICs without a completeness signal. ConfigMap `list/watch` denials +indicate bootstrap RBAC incompatible with the daemon's cluster-wide cache; +use a Launch Kit version with those read permissions. + ## Namespace Mismatch Generation, deployment, and validation must agree on the Network Operator namespace: diff --git a/openspec/specs/discovery-inventory/spec.md b/openspec/specs/discovery-inventory/spec.md index 23f7260f..c96fa770 100644 --- a/openspec/specs/discovery-inventory/spec.md +++ b/openspec/specs/discovery-inventory/spec.md @@ -40,6 +40,12 @@ A discovery extension MUST identify eligible sources and distinguish a missing o - **THEN** normal CLI output identifies their node and PCI addresses and reports a PCI-function count before namespace teardown; those devices remain excluded from saved inventory. - **AND** a later PCI scan failure retains an already-verified publisher and warns that exclusion reporting is incomplete; read-only eligibility probes retain their existing failure policy. +#### Scenario: Bootstrap publisher fails after pod readiness + +- **WHEN** a bootstrap publisher has not published a `NicDevice` and its container enters a failure state after the initial readiness snapshot +- **THEN** discovery refreshes bootstrap pod status and reports the affected node/pod and failure reason without treating a transient restart as permanent failure; unchanged failures are not repeated each poll. +- **AND** failed eligibility probes identify unverified excluded nodes and warn of incomplete inventory; probe failure is not evidence of absent NICs or host restriction. Read-only waiters do not acquire bootstrap status inspection. + #### Scenario: Read-only publisher eligibility is unknown by default - **WHEN** a Ready existing-daemon node has no `NicDevice` and discovery cannot inspect its NIC eligibility diff --git a/pkg/networkoperatorplugin/discovery/daemonhealth.go b/pkg/networkoperatorplugin/discovery/daemonhealth.go new file mode 100644 index 00000000..107d3f3c --- /dev/null +++ b/pkg/networkoperatorplugin/discovery/daemonhealth.go @@ -0,0 +1,120 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "context" + "fmt" + "slices" + "strings" + "time" + + "github.com/nvidia/k8s-launch-kit/pkg/nicconfigdaemon" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + corev1 "k8s.io/api/core/v1" + "sigs.k8s.io/controller-runtime/pkg/client" +) + +// newBootstrapDaemonObserver reports failure evidence only for nodes still +// missing NicDevices. Pod readiness was a snapshot before the sysfs probes; +// later restarts are diagnostics, not proof that a publisher cannot recover. +func newBootstrapDaemonObserver(c client.Client, output ui.Output) func(context.Context, []string) { + previous := map[string]string{} + lastInspectionError := "" + showedHint := false + return func(ctx context.Context, pendingNodes []string) { + if len(pendingNodes) == 0 { + return + } + inspectionCtx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + pods := &corev1.PodList{} + if err := c.List(inspectionCtx, pods, client.InNamespace(nicconfigdaemon.Namespace)); err != nil { + if message := err.Error(); message != lastInspectionError { + output.Warning("Could not inspect discovery daemon health while waiting for NicDevice resources: %v; daemon health is unverified", err) + lastInspectionError = message + } + return + } + lastInspectionError = "" + pending := make(map[string]bool, len(pendingNodes)) + for _, node := range pendingNodes { + pending[node] = true + } + slices.SortFunc(pods.Items, func(a, b corev1.Pod) int { return strings.Compare(a.Name, b.Name) }) + current := map[string]string{} + for _, pod := range pods.Items { + if !pending[pod.Spec.NodeName] || !bootstrapDaemonPod(pod) { + continue + } + statuses := append([]corev1.ContainerStatus{}, pod.Status.InitContainerStatuses...) + statuses = append(statuses, pod.Status.ContainerStatuses...) + for _, status := range statuses { + failure := daemonContainerFailure(status) + if failure == "" { + continue + } + key := string(pod.UID) + "/" + pod.Name + "/" + status.Name + current[key] = failure + if previous[key] == failure { + continue + } + output.Warning("Discovery daemon on %s, pod %s, container %s: %s (restarts: %d); NicDevice publication is still pending", + pod.Spec.NodeName, pod.Name, status.Name, failure, status.RestartCount) + if !showedHint { + output.Warning("Inspect daemon logs with kubectl logs -n %s %s -c %s (add --previous for the prior container); re-run discovery with --keep-namespace to retain bootstrap resources", + nicconfigdaemon.Namespace, pod.Name, status.Name) + showedHint = true + } + } + } + previous = current + } +} + +func bootstrapDaemonPod(pod corev1.Pod) bool { + if pod.Namespace != nicconfigdaemon.Namespace { + return false + } + for _, owner := range pod.OwnerReferences { + if owner.Kind == "DaemonSet" && owner.Name == nicconfigdaemon.DaemonSetName { + return true + } + } + return false +} + +func daemonContainerFailure(status corev1.ContainerStatus) string { + termination := status.State.Terminated + if waiting := status.State.Waiting; waiting != nil && stuckWaitingReasons[waiting.Reason] { + failure := "waiting: " + waiting.Reason + if last := status.LastTerminationState.Terminated; last != nil { + failure += fmt.Sprintf("; last termination: %s (exit code %d)", last.Reason, last.ExitCode) + } + return failure + } + if termination != nil { + if termination.ExitCode == 0 { + return "" + } + return fmt.Sprintf("terminated: %s (exit code %d)", termination.Reason, termination.ExitCode) + } + if last := status.LastTerminationState.Terminated; status.RestartCount > 0 && last != nil && last.ExitCode != 0 { + return fmt.Sprintf("restarted; last termination: %s (exit code %d)", last.Reason, last.ExitCode) + } + return "" +} diff --git a/pkg/networkoperatorplugin/discovery/daemonhealth_test.go b/pkg/networkoperatorplugin/discovery/daemonhealth_test.go new file mode 100644 index 00000000..ac441b63 --- /dev/null +++ b/pkg/networkoperatorplugin/discovery/daemonhealth_test.go @@ -0,0 +1,297 @@ +// Copyright 2026 NVIDIA CORPORATION & AFFILIATES +// +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// +// SPDX-License-Identifier: Apache-2.0 + +package discovery + +import ( + "bytes" + "context" + "errors" + "strings" + "testing" + "time" + + nicop "github.com/Mellanox/nic-configuration-operator/api/v1alpha1" + "github.com/nvidia/k8s-launch-kit/pkg/nicconfigdaemon" + "github.com/nvidia/k8s-launch-kit/pkg/ui" + "github.com/stretchr/testify/assert" + "github.com/stretchr/testify/require" + corev1 "k8s.io/api/core/v1" + apierrors "k8s.io/apimachinery/pkg/api/errors" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/apimachinery/pkg/runtime" + "k8s.io/client-go/rest" + "sigs.k8s.io/controller-runtime/pkg/client" + "sigs.k8s.io/controller-runtime/pkg/client/fake" + "sigs.k8s.io/controller-runtime/pkg/client/interceptor" +) + +func bootstrapHealthPod(name, node string, status corev1.ContainerStatus) *corev1.Pod { + return &corev1.Pod{ + ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: nicconfigdaemon.Namespace, + OwnerReferences: []metav1.OwnerReference{{Kind: "DaemonSet", Name: nicconfigdaemon.DaemonSetName}}}, + Spec: corev1.PodSpec{NodeName: node, Containers: []corev1.Container{{Name: "daemon"}}}, + Status: corev1.PodStatus{ContainerStatuses: []corev1.ContainerStatus{status}}, + } +} + +func crashLoopDaemonStatus() corev1.ContainerStatus { + return corev1.ContainerStatus{Name: "daemon", RestartCount: 2, + State: corev1.ContainerState{Waiting: &corev1.ContainerStateWaiting{Reason: "CrashLoopBackOff"}}, + LastTerminationState: corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{Reason: "Error", ExitCode: 3}}, + } +} + +func TestBootstrapDaemonObserverRefreshesFailuresAndIgnoresOtherPods(t *testing.T) { + scheme := runtime.NewScheme() + require.NoError(t, corev1.AddToScheme(scheme)) + ready := corev1.ContainerStatus{Name: "daemon", Ready: true, State: corev1.ContainerState{Running: &corev1.ContainerStateRunning{}}} + pod := bootstrapHealthPod("daemon-missing", "missing", ready) + pod.Status.InitContainerStatuses = []corev1.ContainerStatus{{Name: "init", State: corev1.ContainerState{ + Terminated: &corev1.ContainerStateTerminated{Reason: "Completed", ExitCode: 0}, + }}} + healthyPeer := bootstrapHealthPod("daemon-published", "published", crashLoopDaemonStatus()) + foreign := bootstrapHealthPod("foreign", "missing", crashLoopDaemonStatus()) + foreign.OwnerReferences[0].Name = "another-daemon" + otherNamespace := bootstrapHealthPod("other-namespace", "missing", crashLoopDaemonStatus()) + otherNamespace.Namespace = "other" + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(pod, healthyPeer, foreign, otherNamespace).Build() + var messages bytes.Buffer + observe := newBootstrapDaemonObserver(c, ui.NewWithWriter(&messages)) + observe(context.Background(), []string{"missing"}) + assert.Empty(t, messages.String()) + + pod.Status.ContainerStatuses = []corev1.ContainerStatus{crashLoopDaemonStatus()} + require.NoError(t, c.Status().Update(context.Background(), pod)) + observe(context.Background(), []string{"missing"}) + observe(context.Background(), []string{"missing"}) + assert.Equal(t, 1, strings.Count(messages.String(), "waiting: CrashLoopBackOff")) + assert.Contains(t, messages.String(), "on missing, pod daemon-missing, container daemon") + assert.Contains(t, messages.String(), "last termination: Error (exit code 3)") + assert.Contains(t, messages.String(), "--previous") + assert.Contains(t, messages.String(), "--keep-namespace") + assert.NotContains(t, messages.String(), "daemon-published") + assert.NotContains(t, messages.String(), "foreign") + assert.NotContains(t, messages.String(), "other-namespace") + + recovered := crashLoopDaemonStatus() + recovered.Ready = true + recovered.State = corev1.ContainerState{Running: &corev1.ContainerStateRunning{}} + pod.Status.ContainerStatuses = []corev1.ContainerStatus{recovered} + require.NoError(t, c.Status().Update(context.Background(), pod)) + observe(context.Background(), []string{"missing"}) + assert.Contains(t, messages.String(), "restarted; last termination: Error (exit code 3)") + pod.Status.ContainerStatuses = []corev1.ContainerStatus{crashLoopDaemonStatus()} + pod.Status.ContainerStatuses[0].RestartCount = recovered.RestartCount + 1 + require.NoError(t, c.Status().Update(context.Background(), pod)) + observe(context.Background(), []string{"missing"}) + assert.Equal(t, 2, strings.Count(messages.String(), "waiting: CrashLoopBackOff"), "a later recurrence after recovery is a new failure state") +} + +func TestBootstrapDaemonWaitAllowsRecoveryAndReportsOnlyPendingPublishers(t *testing.T) { + scheme := runtime.NewScheme() + require.NoError(t, corev1.AddToScheme(scheme)) + require.NoError(t, nicop.AddToScheme(scheme)) + deviceLists, podLists := 0, 0 + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects( + bootstrapHealthPod("daemon-missing", "missing", crashLoopDaemonStatus()), + bootstrapHealthPod("daemon-published", "published", crashLoopDaemonStatus()), + ).WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, c client.WithWatch, list client.ObjectList, opts ...client.ListOption) error { + switch out := list.(type) { + case *nicop.NicDeviceList: + deviceLists++ + out.Items = []nicop.NicDevice{{Status: nicop.NicDeviceStatus{Node: "published"}}, {Status: nicop.NicDeviceStatus{Node: "unrelated"}}} + if deviceLists == 3 { + out.Items = append(out.Items, nicop.NicDevice{Status: nicop.NicDeviceStatus{Node: "missing"}}) + } + return nil + case *corev1.PodList: + podLists++ + } + return c.List(ctx, list, opts...) + }}).Build() + var messages bytes.Buffer + ctx := ui.WithOutput(context.Background(), ui.NewWithWriter(&messages)) + err := waitNicDevicesDiscoveredWithObserver(ctx, c, []string{"published", "missing"}, time.Millisecond, newBootstrapDaemonObserver(c, ui.FromContext(ctx))) + require.NoError(t, err) + assert.Equal(t, 3, deviceLists) + assert.Equal(t, 2, podLists) + assert.Equal(t, 1, strings.Count(messages.String(), "waiting: CrashLoopBackOff")) + assert.Contains(t, messages.String(), "Discovered devices on 1/2 node(s)") + assert.NotContains(t, messages.String(), "pod daemon-published") +} + +func TestBootstrapDaemonObserverReportsReplacementPodWithSameName(t *testing.T) { + scheme := runtime.NewScheme() + require.NoError(t, corev1.AddToScheme(scheme)) + pod := bootstrapHealthPod("daemon", "missing", crashLoopDaemonStatus()) + pod.UID = "original" + c := fake.NewClientBuilder().WithScheme(scheme).WithObjects(pod).Build() + var messages bytes.Buffer + observe := newBootstrapDaemonObserver(c, ui.NewWithWriter(&messages)) + observe(context.Background(), []string{"missing"}) + require.NoError(t, c.Delete(context.Background(), pod)) + pod.ResourceVersion = "" + pod.UID = "replacement" + require.NoError(t, c.Create(context.Background(), pod)) + observe(context.Background(), []string{"missing"}) + assert.Equal(t, 2, strings.Count(messages.String(), "waiting: CrashLoopBackOff"), "a replacement pod is a new source even when its failure and name match") +} + +func TestBootstrapDaemonObserverBoundsInspectionAndHonorsCancellation(t *testing.T) { + for _, parentHasDeadline := range []bool{false, true} { + t.Run(map[bool]string{false: "five second bound", true: "shorter parent deadline"}[parentHasDeadline], func(t *testing.T) { + parentCtx, cancel := context.WithCancel(context.Background()) + if parentHasDeadline { + cancel() + parentCtx, cancel = context.WithTimeout(context.Background(), time.Second) + } + defer cancel() + started := time.Now() + inspections := 0 + c := fake.NewClientBuilder().WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, _ client.WithWatch, _ client.ObjectList, _ ...client.ListOption) error { + inspections++ + deadline, ok := ctx.Deadline() + require.True(t, ok, "health inspection must have its own bound") + if parentHasDeadline { + parentDeadline, _ := parentCtx.Deadline() + assert.Equal(t, parentDeadline, deadline) + } else { + assert.WithinDuration(t, started.Add(5*time.Second), deadline, 100*time.Millisecond) + } + cancel() + <-ctx.Done() + return ctx.Err() + }}).Build() + var messages bytes.Buffer + newBootstrapDaemonObserver(c, ui.NewWithWriter(&messages))(parentCtx, []string{"missing"}) + assert.Equal(t, 1, inspections) + assert.Contains(t, messages.String(), "context canceled") + assert.Contains(t, messages.String(), "daemon health is unverified") + }) + } +} + +func TestBootstrapDaemonInspectionErrorDoesNotMaskDiscoveryError(t *testing.T) { + scheme := runtime.NewScheme() + require.NoError(t, corev1.AddToScheme(scheme)) + require.NoError(t, nicop.AddToScheme(scheme)) + primary := errors.New("cannot list NicDevice resources") + deviceLists := 0 + c := fake.NewClientBuilder().WithScheme(scheme).WithInterceptorFuncs(interceptor.Funcs{List: func(_ context.Context, _ client.WithWatch, list client.ObjectList, _ ...client.ListOption) error { + if _, ok := list.(*corev1.PodList); ok { + return errors.New("pod health forbidden") + } + deviceLists++ + if deviceLists == 3 { + return primary + } + return nil + }}).Build() + var messages bytes.Buffer + ctx := ui.WithOutput(context.Background(), ui.NewWithWriter(&messages)) + err := waitNicDevicesDiscoveredWithObserver(ctx, c, []string{"missing"}, time.Millisecond, newBootstrapDaemonObserver(c, ui.FromContext(ctx))) + require.ErrorIs(t, err, primary) + assert.Equal(t, 1, strings.Count(messages.String(), "pod health forbidden")) + assert.Contains(t, messages.String(), "daemon health is unverified") + assert.NotContains(t, messages.String(), "Found") +} + +func TestNicDeviceWaitWithoutObserverDoesNotInspectPods(t *testing.T) { + scheme := runtime.NewScheme() + require.NoError(t, nicop.AddToScheme(scheme)) + deviceLists, podLists := 0, 0 + c := fake.NewClientBuilder().WithScheme(scheme).WithInterceptorFuncs(interceptor.Funcs{List: func(_ context.Context, _ client.WithWatch, list client.ObjectList, _ ...client.ListOption) error { + if out, ok := list.(*nicop.NicDeviceList); ok { + deviceLists++ + if deviceLists == 2 { + out.Items = []nicop.NicDevice{{Status: nicop.NicDeviceStatus{Node: "worker"}}} + } + return nil + } + podLists++ + return errors.New("unexpected pod inspection") + }}).Build() + require.NoError(t, waitNicDevicesDiscoveredWithInterval(context.Background(), c, []string{"worker"}, time.Millisecond)) + assert.Equal(t, 2, deviceLists) + assert.Zero(t, podLists) +} + +func TestNicDeviceWaitTimeoutNamesMissingPublishers(t *testing.T) { + for _, retryError := range []bool{false, true} { + t.Run(map[bool]string{false: "successful list", true: "retryable list error"}[retryError], func(t *testing.T) { + ctx, cancel := context.WithCancel(context.Background()) + defer cancel() + listErr := apierrors.NewServiceUnavailable("API restarting") + c := &sequenceListClient{ + Client: fake.NewClientBuilder().Build(), + items: []nicop.NicDevice{{Status: nicop.NicDeviceStatus{Node: "published"}}}, + afterCall: func(int) { cancel() }, + } + if retryError { + c.persistentErr = listErr + } + err := waitNicDevicesDiscoveredWithInterval(ctx, c, []string{"worker-b", "published", "worker-a"}, time.Millisecond) + require.Error(t, err) + if retryError { + assert.ErrorIs(t, err, listErr) + assert.Contains(t, err.Error(), "missing: published, worker-a, worker-b") + } else { + assert.Contains(t, err.Error(), "missing: worker-a, worker-b") + assert.NotContains(t, err.Error(), "missing: published") + } + }) + } +} + +func TestDaemonContainerFailureReportsCurrentTermination(t *testing.T) { + status := corev1.ContainerStatus{Name: "daemon", State: corev1.ContainerState{ + Terminated: &corev1.ContainerStateTerminated{Reason: "OOMKilled", ExitCode: 137}, + }} + assert.Equal(t, "terminated: OOMKilled (exit code 137)", daemonContainerFailure(status)) + status = crashLoopDaemonStatus() + status.State = corev1.ContainerState{Running: &corev1.ContainerStateRunning{}} + assert.Equal(t, "restarted; last termination: Error (exit code 3)", daemonContainerFailure(status)) + status.Name = "completed-init" + status.State = corev1.ContainerState{Terminated: &corev1.ContainerStateTerminated{Reason: "Completed", ExitCode: 0}} + assert.Empty(t, daemonContainerFailure(status), "successful init completion is not a current failure, even after an earlier retry") +} + +func TestFilterNodesWithNICsReportsUnverifiedNodes(t *testing.T) { + for _, missingPod := range []bool{false, true} { + t.Run(map[bool]string{false: "probe fails", true: "missing pod"}[missingPod], func(t *testing.T) { + var messages bytes.Buffer + pods := []corev1.Pod{{ObjectMeta: metav1.ObjectMeta{Name: "daemon"}, Spec: corev1.PodSpec{NodeName: "worker", Containers: []corev1.Container{{Name: "daemon"}}}}} + if missingPod { + pods = nil + } + withNICs, restricted := filterNodesWithNICs(context.Background(), nil, nicconfigdaemon.Namespace, []string{"worker"}, pods, ui.NewWithWriter(&messages), + func(context.Context, *rest.Config, string, string, string, []string) (string, error) { + return "", errors.New("container is restarting") + }) + assert.Empty(t, withNICs) + assert.Empty(t, restricted) + assert.Contains(t, messages.String(), "worker") + assert.Contains(t, messages.String(), "unverified node") + assert.Contains(t, messages.String(), "inventory may be incomplete") + assert.NotContains(t, messages.String(), "host-restricted") + if !missingPod { + assert.Contains(t, messages.String(), "daemon pod daemon") + assert.Contains(t, messages.String(), "container is restarting") + } + }) + } +} diff --git a/pkg/networkoperatorplugin/discovery/discover.go b/pkg/networkoperatorplugin/discovery/discover.go index 019885b4..f980fc42 100644 --- a/pkg/networkoperatorplugin/discovery/discover.go +++ b/pkg/networkoperatorplugin/discovery/discover.go @@ -269,7 +269,8 @@ func DiscoverClusterConfig(ctx context.Context, c client.Client, restConfig *res // in another namespace can host the CRs the discovery daemon reconciles // (the daemon's controller queries cluster-wide by node name), so // restricting to nicconfigdaemon.Namespace would miss them. - if err := waitNicDevicesDiscovered(ctx, c, expectedNodes); err != nil { + if err := waitNicDevicesDiscoveredWithObserver(ctx, c, expectedNodes, 10*time.Second, + newBootstrapDaemonObserver(c, uiOutput)); err != nil { return err } @@ -1006,6 +1007,14 @@ func waitNicDevicesDiscovered(parentCtx context.Context, c client.Client, expect } func waitNicDevicesDiscoveredWithInterval(parentCtx context.Context, c client.Client, expectedNodes []string, pollInterval time.Duration) error { + return waitNicDevicesDiscoveredWithObserver(parentCtx, c, expectedNodes, pollInterval, nil) +} + +// waitNicDevicesDiscoveredWithObserver optionally diagnoses pending bootstrap +// publishers. Other callers retain their polling behavior without pod inspection. +func waitNicDevicesDiscoveredWithObserver(parentCtx context.Context, c client.Client, expectedNodes []string, pollInterval time.Duration, + observePending func(context.Context, []string), +) error { uiOutput := ui.FromContext(parentCtx) progress := uiOutput.StartProgress(fmt.Sprintf("Discovering network devices on %d node(s) (timeout: 10 min)", len(expectedNodes))) @@ -1021,6 +1030,11 @@ func waitNicDevicesDiscoveredWithInterval(parentCtx context.Context, c client.Cl for _, n := range expectedNodes { expectedSet[n] = true } + missingNodes := make([]string, 0, len(expectedSet)) + for node := range expectedSet { + missingNodes = append(missingNodes, node) + } + slices.Sort(missingNodes) ticker := time.NewTicker(pollInterval) defer ticker.Stop() @@ -1044,29 +1058,32 @@ func waitNicDevicesDiscoveredWithInterval(parentCtx context.Context, c client.Cl } } - allFound := true + missingNodes = missingNodes[:0] for node := range expectedSet { if !discoveredNodes[node] { - allFound = false - break + missingNodes = append(missingNodes, node) } } + slices.Sort(missingNodes) - if allFound && len(discoveredNodes) > 0 { + if len(missingNodes) == 0 && len(discoveredNodes) > 0 { progress.Success(fmt.Sprintf("Found %d device(s) on %d node(s)", len(list.Items), len(discoveredNodes))) return nil } + if observePending != nil { + observePending(ctx, missingNodes) + } - progress.Update(fmt.Sprintf("Discovered devices on %d/%d node(s)...", len(discoveredNodes), len(expectedSet))) + progress.Update(fmt.Sprintf("Discovered devices on %d/%d node(s)...", len(expectedSet)-len(missingNodes), len(expectedSet))) } select { case <-ctx.Done(): progress.Fail("Timeout waiting for devices") if lastRetryableListErr != nil { - return fmt.Errorf("timeout waiting for NicDevice resources from all expected nodes: %w", lastRetryableListErr) + return fmt.Errorf("timeout waiting for NicDevice resources from all expected nodes (missing: %s): %w", strings.Join(missingNodes, ", "), lastRetryableListErr) } - return fmt.Errorf("timeout waiting for NicDevice resources from all expected nodes") + return fmt.Errorf("timeout waiting for NicDevice resources from all expected nodes (missing: %s)", strings.Join(missingNodes, ", ")) case <-ticker.C: } } @@ -1990,6 +2007,7 @@ func filterNodesWithNICs(ctx context.Context, restConfig *rest.Config, pod := findDaemonPod([]string{node}, dsPods) if pod == nil { log.Log.V(1).Info("No daemon pod on node; excluding from NIC wait set", "node", node) + uiOutput.Warning("No discovery daemon pod is available on %s; excluding this unverified node from the NIC wait set (inventory may be incomplete)", node) continue } containerName := "" @@ -2006,6 +2024,7 @@ func filterNodesWithNICs(ctx context.Context, restConfig *rest.Config, if probeResult != mellanoxNICProbePresent { log.Log.Error(err, "failed to probe node for NVIDIA NIC; excluding from wait set", "node", node, "pod", pod.Name) + uiOutput.Warning("Could not inspect NVIDIA NICs on %s using daemon pod %s: %v; excluding this unverified node from the NIC wait set (inventory may be incomplete)", node, pod.Name, err) continue } // A usable NIC was already verified. A later scan failure must not diff --git a/pkg/nicconfigdaemon/assets/daemon.yaml.tmpl b/pkg/nicconfigdaemon/assets/daemon.yaml.tmpl index ecee8e60..0163b359 100644 --- a/pkg/nicconfigdaemon/assets/daemon.yaml.tmpl +++ b/pkg/nicconfigdaemon/assets/daemon.yaml.tmpl @@ -85,6 +85,10 @@ rules: - configmaps verbs: - get + # The daemon watches Spectrum-X profile ConfigMaps through a cluster-wide, + # label-filtered cache in addition to reading its firmware map directly. + - list + - watch - apiGroups: - "" resources: diff --git a/pkg/nicconfigdaemon/bootstrap_test.go b/pkg/nicconfigdaemon/bootstrap_test.go index 12b363cb..ad65984f 100644 --- a/pkg/nicconfigdaemon/bootstrap_test.go +++ b/pkg/nicconfigdaemon/bootstrap_test.go @@ -250,6 +250,31 @@ func TestEnsure_AppliesDaemonSetWithExpectedImage(t *testing.T) { assert.Equal(t, corev1.HostPathDirectory, *etcVolume.HostPath.Type) } +func TestEnsure_GrantsReadOnlyConfigMapCacheAccess(t *testing.T) { + c := newFakeClient(t) + require.NoError(t, Ensure(context.Background(), c, Options{ + Repository: testRepo, + Version: testVersion, + })) + + role := &rbacv1.ClusterRole{} + require.NoError(t, c.Get(context.Background(), types.NamespacedName{Name: ClusterRoleName}, role)) + + var configMapRules []rbacv1.PolicyRule + for _, rule := range role.Rules { + for _, resource := range rule.Resources { + if resource == "configmaps" { + configMapRules = append(configMapRules, rule) + } + } + } + require.Len(t, configMapRules, 1) + assert.Equal(t, []string{""}, configMapRules[0].APIGroups) + assert.Equal(t, []string{"configmaps"}, configMapRules[0].Resources) + assert.ElementsMatch(t, []string{"get", "list", "watch"}, configMapRules[0].Verbs) + assert.Empty(t, configMapRules[0].ResourceNames, "the daemon cache lists ConfigMaps across namespaces") +} + func TestEnsure_AppliesDaemonSetWithNodeNameAffinity(t *testing.T) { c := newFakeClient(t) diff --git a/skills/k8s-launch-kit-discover/SKILL.md b/skills/k8s-launch-kit-discover/SKILL.md index a3bda7b7..3f394834 100644 --- a/skills/k8s-launch-kit-discover/SKILL.md +++ b/skills/k8s-launch-kit-discover/SKILL.md @@ -448,9 +448,19 @@ Do not add GPU counts or resource maps under `clusterConfig`. its deadline first, discovery returns that context error. It aborts if **no** pod ever becomes Ready. Common causes of a total failure: image tag missing in your registry, pull-secret missing. +- Readiness does not prove device publication. During the subsequent `NicDevice` + wait, bootstrap discovery refreshes pending publishers' pod failure states and + warns without aborting transient recovery. Eligibility-probe failures identify + unverified excluded nodes; inspect these when the wait-set count is too small. + Use `--keep-namespace` and current/`--previous` pod logs for diagnosis. +- Bootstrap ConfigMap access is cluster-wide `get/list/watch`, as required by + the daemon's informer cache, with no ConfigMap write verbs. It does not set + `SKIP_DEVICE_ON_DISCOVERY_ERROR`; failed device queries can still abort daemon + startup. A failed trust query is unknown, not confirmed host restriction. - If discovery reports "no nodes with an NVIDIA NIC (PCI vendor 15b3) were found", - the daemon ran but no node's sysfs exposed a `0x15b3` device (no Mellanox/NVIDIA - NICs, or `/sys` not mounted/readable in the pod). + no probe verified a usable NVIDIA NIC. Check per-node warnings before + concluding that hardware is absent: `/sys` may be unreadable or pod exec may + have failed during a restart. - If discovery reports "no discoverable NVIDIA NICs were found", every detected BlueField is in zero-trust (`restricted`) mode. Those devices cannot be configured from the host and intentionally do not publish `NicDevice` resources. diff --git a/skills/k8s-launch-kit-discover/references/discovery-internals.md b/skills/k8s-launch-kit-discover/references/discovery-internals.md index 08f758b2..0ad3f8af 100644 --- a/skills/k8s-launch-kit-discover/references/discovery-internals.md +++ b/skills/k8s-launch-kit-discover/references/discovery-internals.md @@ -36,6 +36,14 @@ The upstream nic-configuration-operator Helm chart uses cluster-scoped names `k8s-launch-kit-nic-config-daemon` for all three so it can coexist with a Network Operator install without collision. +### ConfigMap cache permissions + +The daemon registers a label-filtered, cluster-wide ConfigMap informer for its +Spectrum-X controller even on nodes without NICs. The bootstrap ClusterRole +therefore grants `get/list/watch` on ConfigMaps, with no write verbs. Kubernetes +RBAC cannot restrict these reads to the cache's label filter. The empty firmware +ConfigMap alone does not satisfy informer synchronization without these verbs. + ### Why CRDs persist Cleanup deletes the namespace and the cluster RBAC but **not** the CRDs. If the @@ -93,6 +101,21 @@ If the scan fails after verifying a usable NIC, the node remains eligible and a warning marks exclusion reporting as incomplete. Read-only publisher eligibility probes retain their early-exit behavior and do not collect this full inventory. +Probe failures before a usable NIC is verified identify the excluded node/pod +and warn that eligibility is unverified and inventory may be incomplete. During +bootstrap's `NicDevice` wait, pod status is refreshed for missing publishers; +container failure reasons are reported once per unchanged state while recovery +remains possible. Read-only waiters do not acquire this bootstrap observer. +Keep the namespace and inspect current and `--previous` logs for the command +error; pod status alone cannot establish host restriction. + +Bootstrap deliberately leaves `SKIP_DEVICE_ON_DISCOVERY_ERROR` unset. The +daemon can omit firmware/PSID failures on ConnectX as well as BlueField mode +errors, and its in-process skipped-device list is not published in CR status. +It may preserve an omitted device's existing CR in another namespace. Enabling +the option without completeness/freshness evidence could let the node-level +wait succeed on incomplete or stale topology. + ### Leftover pre-clean Teardown runs in a `defer`, so a crashed/killed run can leave the namespace diff --git a/skills/k8s-launch-kit-troubleshoot/SKILL.md b/skills/k8s-launch-kit-troubleshoot/SKILL.md index 76529a41..441e3053 100644 --- a/skills/k8s-launch-kit-troubleshoot/SKILL.md +++ b/skills/k8s-launch-kit-troubleshoot/SKILL.md @@ -68,6 +68,8 @@ kubectl get pods -A -o wide | grep -E 'ContainerCreating|Init' | Phase 0 Helm chart download returns HTTP 401 or an image-pull-Secret error | The configured Secret is missing from the operator namespace, unreadable by the kubeconfig, or has no compatible Docker auth entry | Verify the Secret in `networkOperator.namespace`; for NGC, its `.dockerconfigjson` must contain `nvcr.io` credentials | | `l8k discover` daemon pods stuck (ImagePullBackOff / Pending) | Bad image tag, missing pull secret, or no Ready schedulable nodes | Re-run with `--keep-namespace` then `kubectl describe pod -n nvidia-k8s-launch-kit`. Fix `networkOperator.componentVersion`, pass `--image-pull-secrets`, or restore node readiness. NFD is not required. | | `l8k discover` waits for a node that has only a BlueField | The BlueField is in zero-trust (`restricted`) mode and NIC Configuration Operator will not publish a `NicDevice` | Use a Launch Kit version that excludes restricted BlueFields from the wait set; confirm the mode with `mlxprivhost -d q` in the daemon pod. | +| `l8k discover` found Ready pods but waits for `NicDevice` resources | A daemon can crash after the readiness snapshot; a failed BlueField mode query can prevent readable peers from publishing | Inspect the pending-publisher status warnings; retain the namespace with `--keep-namespace` and collect current and `--previous` logs. A failed trust query is not proof of restriction. | +| Bootstrap logs deny cluster-wide ConfigMap `list/watch` | The daemon's informer cache needs more than a direct ConfigMap `get` | Use a Launch Kit version granting bootstrap ConfigMap `get/list/watch`; no ConfigMap write permission is needed. | | `l8k validate` / `deploy` can't find Network Operator pods | Operator namespace mismatch | Verify `--network-operator-namespace` matches actual namespace (does NOT apply to `l8k discover` — it ignores the flag and uses its own `nvidia-k8s-launch-kit` namespace) | | IPPool not allocating | NV-IPAM subnet exhausted or misconfigured | Check `ippools` CR status, verify CIDR ranges | | `--for requires --node-selector` | `--for` was passed without `--node-selector` | Add `--node-selector key=val,…`. The synthesized clusterConfig has no live worker-node list; the selector identifies target nodes at apply time. | diff --git a/skills/k8s-launch-kit-troubleshoot/references/common-failures.md b/skills/k8s-launch-kit-troubleshoot/references/common-failures.md index 27c82c7a..201ce2ef 100644 --- a/skills/k8s-launch-kit-troubleshoot/references/common-failures.md +++ b/skills/k8s-launch-kit-troubleshoot/references/common-failures.md @@ -206,3 +206,16 @@ therefore excluded from `NicDevice` discovery. 5. NFD and `feature.node.kubernetes.io/pci-15b3.present` are not prerequisites for discovery. `--node-selector` is saved for deployment-time selection and does not constrain the discovery daemon or its `NicDevice` wait set. + +If pods initially appear Ready but then crash while discovery awaits +`NicDevice` publication, inspect the pending-publisher container warnings and +both current and `kubectl logs --previous` output. Eligibility-probe failures +can exclude unverified GPU nodes and explain a smaller wait set. Neither an +exec failure nor a failed `mlxprivhost` query proves absence or restriction. +Collect the trust query's stdout/stderr, exit status, MFT version and device +firmware version before diagnosing an access restriction. + +Bootstrap leaves `SKIP_DEVICE_ON_DISCOVERY_ERROR` unset because it can omit +ConnectX NICs too and provides no CR completeness signal. For ConfigMap +`list/watch` denials, use a Launch Kit version granting the read permissions +required by the daemon's cluster-wide cache, including on NIC-less nodes.