bug(kubernetes): sandbox provisioning times out — boundary cannot attach to supervisor (control connect rejected after admission + policy enrichment) #5098

bug(kubernetes): sandbox provisioning times out — boundary cannot attach to supervisor (control connect rejected after admission + policy enrichment)

OpenShell version: 0.1.6 (helm chart oci://ghcr.io/nvidia/openshell/helm-chart, gateway/supervisor/sandbox-runtime images all 0.1.6)
Compute driver: Kubernetes (Agent Sandbox controller v1.0.3; also reproduced with v1.0.9)
Cluster: single-node k3s v1.36.5+k3s1 (linux/amd64), Cilium (NetworkPolicy enforcement verified), Let's Encrypt-signed external endpoint via Traefik TLS passthrough
Topology: tested with both the default per-sandbox supervisor pods and supervisor_topology = "gateway" (shared supervisor Deployment)
Auth: mTLS user auth (agent CA, no OIDC); cert-manager PKI per the chart's certManager.enabled path, plus the built-in PKI job path (both render clean)

Summary

On this cluster, sandbox provisioning reliably times out at 300 s with
ConfigurationPending: Waiting for effective configuration validation before workload activation.
The workload boundary never establishes its routed control connection to the supervisor and retries
every 30 s:

WARN openshell_sandbox::boundary_server::linux: Runtime-initiated control cannot reach the supervisor; retrying

The failure occurs after the supervisor has admitted the assigned instance and enriched the policy
— i.e., at the boundary↔supervisor control attach. On the shared topology the supervisor then
withdraws the instance; on the default (dedicated supervisor pod) topology the supervisor pod
crashes with:

Error: × discover workload image policy: boundary unavailable: boundary idempotent
│ control request timed out while waiting for remote boundary boot

(the supervisor container restarts, re-runs discovery, hits the idempotently-bound boundary again,
crash-loops until the 300 s timeout; the workload pod is left SchedulingGated forever — the bootstrap
scheduling gate is never released on the failure path, which also leaks pods + workspace PVCs).

What we verified working (everything below the attach)

• Gateway: Connected, kubernetes driver active, Sandbox resources created; admission + policy
  enrichment succeed (OCSF CONFIG:ENRICHED) on both topologies
• Policy DNS mapping published (Policy DNS mapped <host> resolved=… synthetic=198.18.x ports=…)
• Sandbox policy Effective (v2), credentials stored and resolved — plaintext egress credential
  resolution verified end-to-end (echo endpoint received the resolved value)
• Network reachability verified from the workload pod while the control attach was failing:
  TCP connect supervisor-svc:5500 and TCP connect supervisor-podIP:5500 both succeed
  (python create_connection, no broker interference); netpol allows workload→supervisor:5500
  (os-shared-control-* policy present) and supervisor ingress from workloads
• Both provider and sandbox-policy endpoints materialize; NET:OPEN ALLOWED + HTTP:GET ALLOWED
  appear for allowed plaintext flows in the same pod
• Control channel for kubectl exec (ssh relay) establishes fine

What we could not determine

The routed control attach fails with the boundary logging only the message and no error= field
(the tracing event is tracing::warn!(%error, "Runtime-initiated control cannot reach the supervisor; retrying")
— the field is not rendered by the boundary's log format, so the actual reason — DNS resolve,
route-unknown reply, TLS auth — is invisible). Evidence from the supervisor side: it admits the instance
and registers the route, but the workload's negotiated connection is closed without a usable reply.
Observed cadence: dial → connection accepted → closed within the negotiation → 2 s backoff doubling to
a 30 s retry cadence.

Environment details

• Sandbox image: Debian slim variant (python:3.13-slim-based, merged-/usr); also reproduced with
  nvcr.io/nvidia/base/ubuntu:24.04
• Gateway gatewayConfig.openshell.drivers.kubernetes.supervisor_topology = gateway (shared) and
  unset (per-sandbox); enforcement: enforce and audit both exercised
• Providers attached: two (source-control + inference), credentials stored in the gateway store,
  GetSandboxProviderEnvironment returns 200
• Cluster DNS works (external + internal names resolve from pods); no TLS-interception between pods

Additional questions

1. Is the per-sandbox dedicated supervisor pod topology (supervisor_topology unset) expected to
   work on 0.1.6, or is gateway the only supported shared mode? (The docs only document gateway.)
2. The boundary idempotent refusal on re-attach after a supervisor restart leaves the sandbox
   permanently unprovisionable. Should the boundary accept a re-attach carrying the same wire policy?
3. Related observability gap: the boundary's dial-failure %error field is not rendered, which hides
   the actual cause (cf. feat(observability): surface Kubernetes scheduling status while a sandbox pod is Pending/unscheduled #4746).

Repro artifacts available

Full gateway.toml, rendered policy, OCSF event timeline, kubectl describe of workload +
supervisor pods, tcpdump-style connect matrix, and the values used. Happy to run dev-channel builds
(0.0.0-dev) against this cluster.