atenet-egress ext-proc has no OTLP wiring - all egress metrics and spans are silently dropped

[git286]

Summary

The egress gateway's ext-proc container is deployed without the shared OTLP exporter settings: unlike every other control-plane component, the atenet-egress Deployment does not consume the ate-otel-config ConfigMap via envFrom. OTEL_EXPORTER_OTLP_ENDPOINT is therefore unset, the OTel SDK falls back to localhost:4317 (where nothing listens), and every egress metric and trace span is silently dropped — while the SDK logs an export-failure warning every ~20 s.

Observed (live GKE install, substrate rev 0d85e85)

Ext-proc logs, repeating forever (~210 warnings per 2 h — the single largest warn/error source in the whole cluster):

"OpenTelemetry SDK error" err="traces export: exporter export timeout: rpc error: code = Unavailable
  desc = ... dial tcp [::1]:4317: connect: cannot assign requested address"
"OpenTelemetry SDK error" err="failed to upload metrics: exporter export timeout: ..."

Consequences verified in the managed-Prometheus / Cloud Trace data:

• target_info in GMP lists ateapi, atecontroller, atelet, atenet-router, and ateom workers — no entry from the egress deployment.
• The registry-listed ext_proc instruments (docs/metrics/registry/metrics.yaml) do not exist in GMP for the egress gateway: no per-actor CONNECT/authz metrics at all.
• Egress spans are absent from every distributed trace (suspend/resume traces jump from atelet straight to GCS).
• The container's local :9090/metrics endpoint serves the data fine (HTTP 200) — it is generated, then dropped at export.

Note this is not one of the documented telemetry gaps: docs/metrics/substrate.yaml blind_spots lists dns, the store, podcertcontroller, etc., but not egress.

Root cause

manifests/ate-install/atenet-egress.yaml (and the sdsmint variant) give the ext-proc container POD_NAME/POD_NAMESPACE env but no envFrom: ate-otel-config, no POD_UID, and no OTEL_RESOURCE_ATTRIBUTES — all of which the ingress router container has. (The explicitly empty --otlp-collector-address= in the args is a red herring: that flag only controls Envoy-side tracing; the ext-proc's own SDK reads $OTEL_EXPORTER_OTLP_ENDPOINT.) The ate-otel-config header comment even says it is "shared by every control plane component" — egress just isn't in the consumer list.

Fix

Add the same envFrom + POD_UID + OTEL_RESOURCE_ATTRIBUTES wiring the ingress router has, in both egress manifests. Change is ready on a fork branch and verified on a live install (SDK warnings drop to zero; target_info with k8s.deployment.name=atenet-egress appears in GMP within ~a minute of rollout):

https://github.com/git286/substrate/tree/fix-egress-otlp-export

Happy to send the PR.

One labeling quirk reviewers may want to weigh in on: because ingress and egress run the same binary, the egress telemetry lands under job="atenet-router" (service.name), distinguishable only by k8s.deployment.name. If that's considered confusing, a follow-up could set a distinct service.name for --mode=egress.

[git286]

cc Bowei Du (@bowei) haiyanmeng to help set priority