[Bug]: S3/GCS/R2 stores fail with a panic when no system CA store is present (official image ships none) instead of returning a config error

[KR-Ravindra]

Version

v1.6.7 (ghcr.io/tracemachina/nativelink:v1.6.7) and main at 76392b5. Cargo.lock: hyper-rustls 0.27.7, rustls-platform-verifier 0.6.2, rustls-native-certs 0.8.2.

Client

Buck2 (not client-specific; happens before any client connects)

OS / Architecture

Linux x86_64 (container)

Deployment method

Docker / Kubernetes (stock ghcr.io/tracemachina/nativelink image)

Storage backend

S3 (also applies to gcs and r2, which share TlsClient)

Configuration

{
  stores: [
    {
      name: "CAS_S3",
      experimental_cloud_object_store: {
        provider: "aws",
        region: "us-east-1",
        bucket: "some-bucket",
        key_prefix: "cas/",
      },
    },
  ],
  // ... servers using CAS_S3
}

Expected behavior

If no CA roots can be loaded, S3Store::new (and the GCS/R2/OCI constructors) should return a nativelink_error::Error that names the store and the remedy (mount a CA bundle and point SSL_CERT_FILE/SSL_CERT_DIR at it), the same way the ONTAP store already does via with_native_roots()?. Ideally CommonObjectSpec should also accept a root_certificates path like OntapS3Spec does, so the bundle can be configured in-file rather than only through an undocumented environment variable.

Actual behavior

The process aborts with a panic from a third-party expect during store construction, before serving anything.

TlsClient::new_with_http_support builds the connector with the panicking HttpsConnectorBuilder::new().with_platform_verifier():

nativelink/nativelink-store/src/common_s3_utils.rs

Lines 75 to 79 in 76392b5

	fn new_with_http_support(common: &CommonObjectSpec, allow_http: bool) -> Self {
	install_default_rustls_crypto_provider();
	let connector_with_roots = HttpsConnectorBuilder::new().with_platform_verifier();

In hyper-rustls 0.27.7 that method is self.try_with_platform_verifier().expect("failure to initialize platform verifier") (src/connector/builder.rs#L76-L79). On Linux rustls-platform-verifier loads roots with rustls-native-certs and returns Err(rustls::Error::General("No CA certificates were loaded from the system")) when the resulting store is empty (rustls-platform-verifier/src/verification/others.rs#L94-L110), so the expect fires.

The official image is built by nativelinkImageFor in flake.nix with copyToRoot containing only the nativelink /bin (pkgs.cacert appears only as a build-time buildInputs entry, not in the image):

nativelink/flake.nix

Lines 272 to 296 in 76392b5

	nativelinkImageFor = archPackages: arch:
	buildImage {
	inherit arch;
	name = "nativelink";
	copyToRoot = [
	(pkgs.buildEnv {
	name = "nativelink-buildEnv";
	paths = [archPackages];
	pathsToLink = ["/bin"];
	})
	];
	config = {
	Entrypoint = [(pkgs.lib.getExe' archPackages "nativelink")];
	Labels = {
	"org.opencontainers.image.description" = "An RBE compatible, high-performance cache and remote executor.";
	"org.opencontainers.image.documentation" = "https://github.com/TraceMachina/nativelink";
	"org.opencontainers.image.licenses" = "FSL-1.1-Apache-2.0";
	"org.opencontainers.image.revision" = "${self.rev or self.dirtyRev or "dirty"}";
	"org.opencontainers.image.source" = "https://github.com/TraceMachina/nativelink";
	"org.opencontainers.image.title" = "NativeLink";
	"org.opencontainers.image.vendor" = "Trace Machina, Inc.";
	};
	};
	};

So any experimental_cloud_object_store with provider: "aws" | "gcs" | "r2" panics on first start in the stock image unless the operator already knows to mount a CA bundle and set SSL_CERT_FILE. The same TlsClient::new is used by s3_store.rs#L112-L113, r2_store.rs#L49, oci_store.rs#L58 and ontap_s3_existence_cache_store.rs#L418, all of which are already inside Result-returning constructors.

Why this is a defect rather than "bring your own certs":

• Bundle Certificate Authorities #708 / Re-add cacerts to nativelink image #810 settled on not bundling CAs in the image, which is fine. But the failure mode should be a config error, not a panic. The ONTAP path already gets this right: ontap_s3_store.rs#L133-L139 uses ClientConfig::builder().with_native_roots()? and propagates the error.
• CommonObjectSpec (nativelink-config/src/stores.rs#L1322) has no root_certificates field, while OntapS3Spec has one (stores.rs#L861). For aws/gcs/r2 the only knob is the SSL_CERT_FILE/SSL_CERT_DIR env var honoured by rustls-native-certs, which is not mentioned anywhere in the repo (no hits for SSL_CERT_FILE in code, docs, or kubernetes/).
• web/apps/docs/content/docs/how-to/stores/s3-and-compatible.mdx#L123-L125 says "Without it, the system roots are used" (in the ONTAP section) without noting that the published image has no system roots, and the aws/gcs/r2 sections do not mention trust roots at all.

Logs / error output

Expected form of the panic (message from hyper-rustls expect, error is Debug-formatted):

thread '...' panicked at .../hyper-rustls-0.27.7/src/connector/builder.rs:...:
failure to initialize platform verifier: General("No CA certificates were loaded from the system")

Reproduce with the stock image and any config containing an aws store (credentials via env; the panic happens before credentials are used):

docker run --rm -v "$PWD/cas.json5:/cfg.json5" ghcr.io/tracemachina/nativelink:v1.6.7 /cfg.json5

Mounting a PEM bundle into the container and setting SSL_CERT_FILE=/path/to/ca-certificates.crt makes it start.

Additional context

Proposed fix (small):

1. nativelink-store/src/common_s3_utils.rs: replace with_platform_verifier() with try_with_platform_verifier() and make TlsClient::new / new_for_credentials / new_with_http_support return Result<Self, Error> (Code::InvalidArgument, message along the lines of "no CA certificates could be loaded for the TLS client; mount a CA bundle and set SSL_CERT_FILE, or set root_certificates"). Update the five call sites above plus nativelink-store/tests/common_s3_utils_test.rs#L49,L65 (all already in Result contexts). Roughly <80 lines; unit-testable by constructing TlsClient with SSL_CERT_FILE pointing at an empty file and SSL_CERT_DIR at an empty directory.
2. Optionally add root_certificates: Option<String> to CommonObjectSpec, mirroring OntapS3Spec, and build the connector with with_tls_config + with_root_certificates when set (the ONTAP store already has load_custom_certs to reuse).
3. Docs: one sentence in s3-and-compatible.mdx and the container/Kubernetes deploy pages stating that the image carries no CA store and showing the SSL_CERT_FILE mount.

Related: #441 (open; "we unintentionally depend on existing CAs during testtime and runtime" — this issue is the concrete user-facing failure mode for the S3-family stores), #708 / #810 (decision not to bundle CAs), #1785 (switched S3 stores to the platform verifier).

Found while operating this in production; happy to send a PR if maintainers agree with the analysis.