feat(observability): verify OTLP collectors with a configured CA and client certificate #5080

User Story

As an operator exporting OpenShell telemetry, I need every OTLP lane (gateway traces, OCSF logs, supervisor traces, and the agent trace relay) to reach a collector whose certificate is signed by a private CA, or that requires a client certificate, so that telemetry and the collector credentials sent with it travel over verified TLS.

Problem Statement

OTLP export from OpenShell trusts only the platform roots and cannot present a client certificate:

• gRPC export verifies https:// collectors against the system trust roots only, so a collector behind an internal CA fails unless SSL_CERT_FILE is changed for the whole gateway process.
• OTLP/HTTP export and the agent trace relay have no TLS at all and reject https:// endpoints, so the headers they send, including credentials, travel in cleartext.
• No exporter can present a client certificate, so collectors that require mutual TLS are out of reach.

Impact / Why This Matters

Collectors in production commonly sit behind an organization's CA and often require mutual TLS. Today the only workarounds are a translating collector sidecar that terminates TLS, or replacing the gateway's trust roots wholesale. Neither covers supervisors, which run in each sandbox and do not inherit the gateway's environment.

Proposed Design

Each [openshell.gateway.otlp] collector table accepts TLS files alongside endpoint, protocol, and headers, inherited with the rest of the collector by tables without their own endpoint:

• ca_file: PEM certificates that replace the platform roots when verifying the collector.
• cert_file and key_file: a PEM client certificate chain and key presented to the collector. Each requires the other.

TLS files require an https:// endpoint. The gateway reads them at startup and rejects unreadable, mismatched, or oversized files. Every exporter, over gRPC or OTLP/HTTP, verifies against the configured CA and presents the client certificate. Supervisors receive each lane's TLS material with their launch credentials, so no files need to be mounted into sandboxes. Standalone compute drivers take --otlp-ca-file, --otlp-cert-file, and --otlp-key-file for their own spans.

Suggested UX

[openshell.gateway.otlp]
endpoint  = "https://otel-collector.observability.svc:4317"
ca_file   = "/etc/openshell/otlp/ca.pem"
cert_file = "/etc/openshell/otlp/client.pem"
key_file  = "/etc/openshell/otlp/client-key.pem"

Acceptance Criteria

• Gateway, supervisor, and driver exports verify https:// collectors against ca_file over gRPC and OTLP/HTTP.
• Exports present the configured client certificate, and fail against a collector that requires one when none is configured.
• The agent trace relay exports to https:// collectors with the same settings.
• Invalid TLS configuration fails at startup with an error naming the key.
• Gateway configuration docs describe the keys, inheritance, and Helm mounting.

Alternatives Considered

• SSL_CERT_FILE on the gateway: replaces trust for every outbound connection, never reaches supervisors, and cannot supply a client certificate.
• A collector sidecar that terminates TLS: adds a component per deployment and leaves the hop to it unprotected.
• The OTEL_EXPORTER_OTLP_CERTIFICATE family of variables: environment-only, so it would not reach supervisors, and OpenShell deliberately keeps export settings in [openshell.gateway.otlp].

Agent Investigation

Related: #5067 (OTLP/HTTP and collector headers).