Skip to content

LOCAL-K3S-NVIDIA-RUNBOOK.md

Latest commit

 

History

History
232 lines (171 loc) · 7.05 KB

LOCAL-K3S-NVIDIA-RUNBOOK.md

File metadata and controls

232 lines (171 loc) · 7.05 KB
Raw

Local k3s NVIDIA Runbook

This runbook deploys llm-observability-stack on a local Xubuntu 24 host after k3s-nvidia-edge has prepared k3s, the NVIDIA runtime, GPU Operator, device plugin, RuntimeClass, and DCGM exporter. This chart installs Ollama, Open WebUI, Prometheus/Grafana, OpenTelemetry Collector, and LLM observability resources.

1. Enter the Project

cd /media/waqasm86/External1/Waqas-Projects/Project-Edge-Computing-LLM/llm-observability-stack

2. Verify Host Prerequisites

nvidia-smi
helm version
kubectl version --client
kubectl get nodes -o wide
kubectl get storageclass

For k3s, local-path storage should exist. The GPU must be visible on the host before Kubernetes can schedule Ollama with nvidia.com/gpu.

3. Verify NVIDIA Runtime and Device Plugin

kubectl get runtimeclass
kubectl get pods -A | grep -Ei 'nvidia|device-plugin|dcgm'
kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" gpu="}{.status.allocatable.nvidia\.com/gpu}{"\n"}{end}'

If RuntimeClass/nvidia or nvidia.com/gpu is missing, install or repair the base layer first:

edge install infra --yes
edge validate infra

4. Verify the Local GGUF Model

The GeForce 940M profile expects:

ls -lh /media/waqasm86/External1/Waqas-Projects/repos-llamatelemetry/llamatelemetry-xubuntu24/models/qwen-1.8b-chat-q4_K_M.gguf
sha256sum /media/waqasm86/External1/Waqas-Projects/repos-llamatelemetry/llamatelemetry-xubuntu24/models/qwen-1.8b-chat-q4_K_M.gguf

The expected SHA-256 is ef0125bcc77278420b64229fc29e235435c517f67ab7aa8546a9f5f7be644cef. Other profiles may retain their Gemma defaults.

The chart mounts that host directory read-only into Ollama at /models/gguf, and the Ollama PVC is annotated with helm.sh/resource-policy: keep.

5. Build and Import Local Images

The ollama-gateway and edge-toolbox images are local project images. Build and import them into k3s containerd:

./hack/build-local-image.sh ollama-gateway 0.2.0 . ollama-gateway/Dockerfile
./hack/import-local-image-to-k3s.sh ollama-gateway 0.2.0

./hack/build-local-image.sh edge-toolbox 0.2.0 . edge-toolbox/Dockerfile
./hack/import-local-image-to-k3s.sh edge-toolbox 0.2.0

Verify:

sudo k3s ctr images ls | grep -E 'ollama-gateway|edge-toolbox'

The images that must be built locally before enabling ollamaGateway and edgeToolbox are:

ollama-gateway:0.2.0
edge-toolbox:0.2.0

The platform images rendered by the enterprise profile are:

docker.io/grafana/grafana:13.0.2
docker.io/library/busybox:1.38.0
ghcr.io/jkroepke/kube-webhook-certgen:1.8.3
ghcr.io/open-telemetry/opentelemetry-collector-releases/opentelemetry-collector-k8s:0.153.0
ghcr.io/open-webui/open-webui:0.8.10
ollama/ollama:0.17.7
quay.io/kiwigrid/k8s-sidecar:2.7.3
quay.io/prometheus-operator/prometheus-operator:v0.91.0
quay.io/prometheus/alertmanager:v0.33.0
quay.io/prometheus/node-exporter:v1.11.1-distroless
quay.io/prometheus/prometheus:v3.12.0-distroless
redis:7.4.2-alpine3.21
registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.19.1

6. Validate Helm Rendering

helm dependency update .
helm lint .
helm template llm-observability-stack . -f values.enterprise-pilot-k3s.yaml >/tmp/llm-observability-enterprise.yaml
grep -n 'opentelemetry-collector' /tmp/llm-observability-enterprise.yaml | head
grep -n 'helm.sh/resource-policy: keep' /tmp/llm-observability-enterprise.yaml

7. Install the Stack

Use the bootstrap helper to apply Prometheus Operator CRDs first, then install the umbrella chart:

./hack/bootstrap-enterprise-pilot-k3s.sh

For a lighter first install without local app images:

./hack/bootstrap-enterprise-pilot-k3s.sh \
  --set ollamaGateway.enabled=false \
  --set edgeToolbox.enabled=false

After importing local images, enable the workloads with:

helm upgrade --install llm-observability-stack . \
  -n llm-observability --create-namespace \
  -f values.enterprise-pilot-k3s.yaml \
  --set kube-prometheus-stack.crds.enabled=false

8. Watch Rollout

kubectl get pods,svc,pvc -n llm-observability -o wide
kubectl rollout status deploy/ollama -n llm-observability --timeout=300s
kubectl rollout status deploy/opentelemetry-collector -n llm-observability --timeout=180s
kubectl rollout status deploy/ollama-gateway -n llm-observability --timeout=180s

9. Verify Ollama and the Local Model

kubectl exec -n llm-observability deploy/ollama -- ls -lh /models/gguf
kubectl exec -n llm-observability deploy/ollama -- ollama list
kubectl exec -n llm-observability deploy/ollama -- ollama ps

The NVIDIA profiles render Ollama with:

OLLAMA_KEEP_ALIVE=-1
ollama.ollama.models.run:
  - qwen-1-8b-chat-q4-k-m-local

At container startup, the chart creates the local GGUF-backed model and then runs a small warmup prompt so Ollama loads the model into its GPU-enabled runner. OLLAMA_KEEP_ALIVE=-1 keeps that loaded runner resident instead of unloading it after an idle timeout.

On the GeForce 940M profile, the GPU has 1 GiB of VRAM, so a GGUF model may not fit entirely in GPU memory. This is expected: Ollama should still use CUDA and offload as many layers as fit, with the remainder on CPU. Verify the split with:

kubectl logs -n llm-observability deploy/ollama --tail=200 | grep -Ei 'CUDA|offload|model weights|gpu memory'
kubectl exec -n llm-observability deploy/ollama -- ollama ps
kubectl exec -n llm-observability deploy/ollama -- nvidia-smi

Port-forward Ollama:

kubectl port-forward -n llm-observability svc/ollama 11434:11434

From another terminal:

curl -s http://127.0.0.1:11434/api/tags | jq
curl -s http://127.0.0.1:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen-1-8b-chat-q4-k-m-local","prompt":"Reply with one short sentence.","stream":false}' | jq

10. Verify OpenTelemetry Collector

kubectl get deploy,svc,configmap -n llm-observability | grep opentelemetry-collector
kubectl logs -n llm-observability deploy/opentelemetry-collector --tail=100
kubectl port-forward -n llm-observability svc/opentelemetry-collector 4317:4317 4318:4318 8888:8888
curl -s http://127.0.0.1:8888/metrics | head

ollama-gateway sends OTLP to http://opentelemetry-collector:4317 when OpenTelemetry is enabled.

11. Access Open WebUI and Grafana

kubectl port-forward -n llm-observability svc/open-webui 8080:8080
kubectl port-forward -n llm-observability svc/llm-observability-stack-grafana 3000:80

Open:

http://127.0.0.1:8080
http://127.0.0.1:3000

Grafana local credentials are pinned by this chart:

username: admin
password: admin

12. Run Tests

go test ./...
./hack/validate-local-stack.sh
./hack/validate-local-stack.sh --strict-gpu

13. Uninstall Without Deleting Models

helm uninstall llm-observability-stack -n llm-observability
kubectl get pvc -n llm-observability

Do not delete the Ollama PVC or the host GGUF directory unless you intentionally want to remove model state. The GGUF hostPath mount is read-only, and Helm keeps the Ollama PVC.