Skip to content

Commit 9a905c4

Browse files
authored
feat(auth): support GKE agent identity credential bundle for bound tokens and mTLS (#14595)
## Why GKE delivers agent identity credentials (Pod Certificates) as a single combined PEM file, holding the certificate chain and a PKCS#8 private key, at: ``` /var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem ``` The feature branch already handles Cloud Run, where a certificate config points into the well-known directory. On GKE there is no certificate config, so neither the transport nor the token side found the certificate. This is the Java counterpart of googleapis/google-cloud-python#18559. Background: go/gke-security-identity/features/agent_identity_diy.md and go/sdk-mds-bound-token. ## What changes When **no certificate configuration exists** (`GOOGLE_API_CERTIFICATE_CONFIG` is unset and there is no `~/.config/gcloud/certificate_config.json`), a readable, non-empty bundle now: - **Transport:** turns on mTLS automatically. - `MtlsUtils.getWorkloadCertPath` / `useMtlsClientCertificate` return the bundle path. - `DefaultMtlsProviderFactory.create()` builds its keystore from the bundle via a new package-private `X509Provider.createForTransport()`. - Every GAX transport (`EndpointContext`, gRPC, HTTP/JSON) goes through that factory. - **Tokens:** `AgentIdentityUtils` resolves the same bundle, so `ComputeEngineCredentials` requests tokens bound to the certificate the transport presents. - With `GOOGLE_API_USE_CLIENT_CERTIFICATE=true`, the bundle takes precedence over the Cloud Run well-known files for the same reason. **Kept unchanged:** - `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` and `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` still opt out. - An explicit or default certificate config (including ECP) always beats the bundle, on both sides. - **The bundle is never polled for.** GKE writes it before the container starts, and polling for a missing file would delay every workload that isn't an agent. - **Workload identity federation X.509 is unchanged.** The public `X509Provider` constructors don't fall back to the bundle. Only the transport factory does. - If the bundle disappears during the read (rotation), the transport reports `CertificateSourceUnavailableException` rather than a generic `IOException`. If it's briefly missing after a credential was cached, the token side falls back to the cached credential, but only while the bundle is still allowed (no certificate configuration has appeared). **New API:** a public static `MtlsUtils.hasCertificateConfiguration(...)` on an `@InternalApi` class. This is additive only. ## Differences from Python (intentional) - Python also checks `CLOUDSDK_CONTEXT_AWARE_CERTIFICATE_CONFIG_FILE_PATH`. Java has no equivalent env var. - Python temporarily keeps default **ID tokens** unbound on Cloud Run (when `GOOGLE_API_CERTIFICATE_CONFIG` is set). That gate is temporary and Python-only, so it isn't copied here. On GKE, both libraries bind ID tokens by default. #14559 adds the per-target `DISABLE_BOUND_ID_TOKEN` opt-out. ## Related - #14556 (HTTP/JSON mTLS fix, merged) enables mTLS over HTTP/JSON transports. - #14559 (`DISABLE_BOUND_ID_TOKEN`) is the per-target ID-token opt-out. ## Tests ### Unit tests - `MtlsUtilsTest`: - bundle present, absent, a directory, or empty; - `USE_CLIENT_CERTIFICATE=false`; - a default config without a workload section (ECP) blocks the bundle; - an explicit config takes precedence; - `hasCertificateConfiguration` / `getGkeCredentialBundlePath`. - `X509ProviderTest`: - transport provider with cert-first and key-first bundles; - an explicit config, or an override path, takes precedence; - unavailable when there is no bundle; - the public constructor ignores the bundle (WIF). - `AgentIdentityUtilsTest`: - implicit discovery with zero sleeps; - the cached credential stays bound; - opt-outs, a default gcloud config, and a non-agent cert all leave the token unbound; - an empty or absent bundle returns null without polling; - explicit true finds the bundle, never polls for it, and prefers it over the Cloud Run files; - fallback to the cache after the bundle is deleted, and no fallback once a certificate config has appeared (implicit and explicit paths). Full `oauth2_http` suite: 1089 tests, 0 failures. `fmt` is clean. ### Live test on GKE - **Cluster:** a GKE cluster with agent identity enabled. - **Pods:** the test pods get the agent identity credential bundle at `/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem`. - **Image:** built from this PR's jars merged with #14556, which is already on the base branch and is required for HTTP/JSON mTLS. It uses ADC with default settings and no client options. - **Details:** cluster setup, manifests and full logs are in the internal links below. **Checks:** - **GCS over raw HTTP:** read through `storage.mtls.googleapis.com` with the transport certificate. The same token on plain `storage.googleapis.com` gets 401, which proves the token is bound. - **GAX Resource Manager client:** `getProject` over gRPC and HTTP/JSON. - **ID tokens:** sent to an agent-to-agent receiver pod. It accepts only bound tokens over mTLS whose `cnf` matches the client certificate, and only unbound tokens over plain HTTP. | Scenario | Token | GAX resolved endpoint | GAX gRPC / HTTP-JSON | GCS mTLS / plain | ID token mTLS / plain | |---|---|---|---|---|---| | Defaults | Bound; `cnf` = bundle leaf = transport cert | `cloudresourcemanager.mtls.googleapis.com` | ✅ / ✅ | 200 / **401** | **200** / 401 | | `GOOGLE_API_ENABLE_RUNTIME_BOUND_TOKEN=false` | Unbound | mTLS endpoint | ✅ / ✅ | 200 / 200 | 401 / **200** | | `GOOGLE_API_USE_CLIENT_CERTIFICATE=false` | Unbound | plain endpoint | ✅ / ✅ | 200 / 200 | 401 / **200** | | `GOOGLE_API_USE_CLIENT_CERTIFICATE=true` | Bound (bundle) | mTLS endpoint | ✅ / ✅ | 200 / **401** | **200** / 401 | | Pod without agent identity | Unbound, no mTLS | plain endpoint | authenticated / authenticated | — / authenticated | n/a | **Timing:** the first token fetch took about 0.5–1s in every scenario. A pod without the bundle has no polling delay. **Internal links (Googlers only):** [setup](https://paste.googleplex.com/5635555912712192) · [manifests](https://paste.googleplex.com/6258309762514944) · [harness](https://paste.googleplex.com/4882197073494016) · [build/run](https://paste.googleplex.com/5942061774274560) · [results](https://paste.googleplex.com/5072605959290880) **Out of scope for this PR (tracked separately):** - **Leaf-only keystore:** `X509Provider`'s keystore holds only the leaf certificate, through google-http-client `SecurityUtils.createMtlsKeyStore`. Google APIs over `*.mtls.googleapis.com` are unaffected. Agent-to-agent mTLS against a peer that trusts only the SPIFFE root fails unless the caller presents the full chain.
1 parent 103734a commit 9a905c4

8 files changed

Lines changed: 793 additions & 22 deletions

File tree

‎google-auth-library-java/oauth2_http/java/com/google/auth/mtls/DefaultMtlsProviderFactory.java‎

Lines changed: 5 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -40,9 +40,10 @@ public class DefaultMtlsProviderFactory {
4040

4141
/**
4242
* Creates an instance of {@link MtlsProvider}. It first attempts to create an {@link
43-
* com.google.auth.mtls.X509Provider}. If the certificate source is unavailable, it falls back to
44-
* creating a {@link SecureConnectProvider}. If the secure connect provider also fails, it throws
45-
* a {@link com.google.auth.mtls.CertificateSourceUnavailableException}.
43+
* com.google.auth.mtls.X509Provider}, which also falls back to the GKE credential bundle when no
44+
* certificate configuration is present. If the certificate source is unavailable, it falls back
45+
* to creating a {@link SecureConnectProvider}. If the secure connect provider also fails, it
46+
* throws a {@link com.google.auth.mtls.CertificateSourceUnavailableException}.
4647
*
4748
* @return an instance of {@link MtlsProvider}.
4849
* @throws com.google.auth.mtls.CertificateSourceUnavailableException if neither provider can be
@@ -53,7 +54,7 @@ public static MtlsProvider create() throws IOException {
5354
// Note: The caller should handle CertificateSourceUnavailableException gracefully, since
5455
// it is an expected error case. All other IOExceptions are unexpected and should be surfaced
5556
// up the call stack.
56-
MtlsProvider mtlsProvider = new X509Provider();
57+
MtlsProvider mtlsProvider = X509Provider.createWithGkeCredentialBundleFallback();
5758
if (mtlsProvider.isAvailable()) {
5859
return mtlsProvider;
5960
}

‎google-auth-library-java/oauth2_http/java/com/google/auth/mtls/MtlsUtils.java‎

Lines changed: 83 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@
3333
import com.google.api.core.InternalApi;
3434
import com.google.auth.oauth2.EnvironmentProvider;
3535
import com.google.auth.oauth2.PropertyProvider;
36+
import com.google.common.annotations.VisibleForTesting;
3637
import com.google.common.base.Strings;
3738
import com.google.common.io.BaseEncoding;
3839
import java.io.File;
@@ -58,10 +59,78 @@ public class MtlsUtils {
5859
static final String WELL_KNOWN_CERTIFICATE_CONFIG_FILE = "certificate_config.json";
5960
static final String CLOUDSDK_CONFIG_DIRECTORY = "gcloud";
6061

62+
/**
63+
* Well-known path of the combined certificate chain and private key bundle that GKE delivers to
64+
* pods with Agent Identity (Pod Certificates). Used as a fallback client certificate source when
65+
* no certificate configuration file is configured or present.
66+
*/
67+
static final String GKE_CREDENTIAL_BUNDLE_PATH =
68+
"/var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem";
69+
70+
// Overridable via setGkeCredentialBundlePathForTesting so unit tests can point to a temp file.
71+
private static volatile String gkeCredentialBundlePath = GKE_CREDENTIAL_BUNDLE_PATH;
72+
6173
private MtlsUtils() {
6274
// Prevent instantiation for Utility class
6375
}
6476

77+
/**
78+
* Returns {@code true} if a certificate configuration is configured or present: either {@code
79+
* GOOGLE_API_CERTIFICATE_CONFIG} is set, or the well-known gcloud {@code certificate_config.json}
80+
* file exists. When this returns {@code true}, the GKE credential bundle fallback is not used, so
81+
* the certificate used for mTLS and the certificate bound to tokens always come from the same
82+
* source.
83+
*/
84+
public static boolean hasCertificateConfiguration(
85+
EnvironmentProvider envProvider, PropertyProvider propProvider) {
86+
if (!Strings.isNullOrEmpty(envProvider.getEnv(CERTIFICATE_CONFIGURATION_ENV_VARIABLE))) {
87+
return true;
88+
}
89+
try {
90+
return getWellKnownCertificateConfigFile(envProvider, propProvider).exists();
91+
} catch (IOException e) {
92+
// APPDATA missing on Windows, etc. No well-known configuration file.
93+
return false;
94+
}
95+
}
96+
97+
/**
98+
* Returns the GKE credential bundle path if no certificate configuration is configured or present
99+
* (see {@link #hasCertificateConfiguration}) and the bundle exists and is readable; otherwise
100+
* returns {@code null}.
101+
*/
102+
static @Nullable String getGkeCredentialBundlePath(
103+
EnvironmentProvider envProvider, PropertyProvider propProvider) {
104+
if (hasCertificateConfiguration(envProvider, propProvider)) {
105+
return null;
106+
}
107+
return findGkeCredentialBundle();
108+
}
109+
110+
/**
111+
* Returns {@code true} if the GKE credential bundle at {@code path} is a readable, non-empty
112+
* regular file.
113+
*/
114+
public static boolean isGkeCredentialBundleReady(String path) {
115+
File bundle = new File(path);
116+
return bundle.isFile() && bundle.canRead() && bundle.length() > 0;
117+
}
118+
119+
/**
120+
* Returns the GKE credential bundle path if it is a readable, non-empty regular file, otherwise
121+
* null.
122+
*/
123+
private static @Nullable String findGkeCredentialBundle() {
124+
String path = gkeCredentialBundlePath;
125+
return isGkeCredentialBundleReady(path) ? path : null;
126+
}
127+
128+
/** Overrides the GKE credential bundle path for testing; {@code null} restores the default. */
129+
@VisibleForTesting
130+
static void setGkeCredentialBundlePathForTesting(@Nullable String path) {
131+
gkeCredentialBundlePath = path != null ? path : GKE_CREDENTIAL_BUNDLE_PATH;
132+
}
133+
65134
/**
66135
* Returns if mutual TLS client certificate should be used. Returns true if valid workload
67136
* certificates are configured or if GOOGLE_API_USE_CLIENT_CERTIFICATE is explicitly set to true
@@ -89,16 +158,20 @@ public static boolean useMtlsClientCertificate(
89158
* <ol>
90159
* <li><b>Non-null {@link String} (Valid happy path):</b> A valid workload certificate
91160
* configuration was found and both the certificate and private key files exist and are
92-
* readable.
161+
* readable; or no certificate configuration is set or present and the GKE credential bundle
162+
* ({@code /var/run/secrets/workload-spiffe-credentials/
163+
* x509.credential-bundle.private-key.pem}) exists and is readable, in which case the bundle
164+
* path is returned.
93165
* <li><b>{@link IllegalStateException} (Invalid state - fail closed):</b> An explicit {@code
94166
* GOOGLE_API_CERTIFICATE_CONFIG} path or an existing default well-known certificate
95167
* configuration file is missing, unreadable, malformed, or references missing/unreadable
96168
* certificate or private key files. This is treated as an unrecoverable misconfiguration.
97169
* <li><b>{@code null} (Safe fallback / fail open):</b> Client certificates are explicitly
98170
* disabled via {@code GOOGLE_API_USE_CLIENT_CERTIFICATE=false}, no explicit configuration
99-
* is set and the default well-known configuration file does not exist on disk, or the
100-
* configuration specifies an non-workload source (e.g., ECP/PKCS11 without a {@code
101-
* workload} section). Callers can proceed without workload certificate file polling.
171+
* is set and neither the default well-known configuration file nor a readable GKE
172+
* credential bundle exists on disk, or the configuration specifies an non-workload source
173+
* (e.g., ECP/PKCS11 without a {@code workload} section). Callers can proceed without
174+
* workload certificate file polling.
102175
* </ol>
103176
*/
104177
public static @Nullable String getWorkloadCertPath(
@@ -175,9 +248,14 @@ public static boolean useMtlsClientCertificate(
175248
checkCertAndKeyFilesReadable(config, defaultConfigFile.getAbsolutePath(), true);
176249
return config.getCertPath();
177250
}
251+
// A default configuration file exists without a workload section (e.g. ECP); it takes
252+
// precedence over the GKE credential bundle.
253+
return null;
178254
}
179255

180-
return null;
256+
// 3. GKE credential bundle (Pod Certificates), used only when no certificate configuration
257+
// file is configured or present. The combined bundle serves as both certificate and key.
258+
return findGkeCredentialBundle();
181259
}
182260

183261
private static void checkCertAndKeyFilesReadable(

‎google-auth-library-java/oauth2_http/java/com/google/auth/mtls/X509Provider.java‎

Lines changed: 79 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
import com.google.auth.oauth2.SystemPropertyProvider;
3939
import java.io.File;
4040
import java.io.FileInputStream;
41+
import java.io.FileNotFoundException;
4142
import java.io.IOException;
4243
import java.io.InputStream;
4344
import java.io.SequenceInputStream;
@@ -54,7 +55,8 @@
5455
public class X509Provider implements MtlsProvider {
5556
private final EnvironmentProvider envProvider;
5657
private final PropertyProvider propProvider;
57-
private final String certConfigPathOverride;
58+
private final @Nullable String certConfigPathOverride;
59+
private final boolean allowGkeCredentialBundle;
5860

5961
/**
6062
* Creates an X509 provider with an override path for the certificate configuration, bypassing the
@@ -70,9 +72,39 @@ public X509Provider(
7072
EnvironmentProvider envProvider,
7173
PropertyProvider propProvider,
7274
String certConfigPathOverride) {
75+
this(envProvider, propProvider, certConfigPathOverride, false);
76+
}
77+
78+
/**
79+
* Creates an X509 provider that optionally falls back to the GKE credential bundle when no
80+
* certificate configuration is present.
81+
*
82+
* @param envProvider environment provider used for environment variables
83+
* @param propProvider property provider used for system properties
84+
* @param certConfigPathOverride the path to read the certificate configuration from.
85+
* @param allowGkeCredentialBundle whether to fall back to the GKE credential bundle when no
86+
* certificate configuration is present and {@code certConfigPathOverride} is null.
87+
*/
88+
X509Provider(
89+
EnvironmentProvider envProvider,
90+
PropertyProvider propProvider,
91+
@Nullable String certConfigPathOverride,
92+
boolean allowGkeCredentialBundle) {
7393
this.envProvider = envProvider;
7494
this.propProvider = propProvider;
7595
this.certConfigPathOverride = certConfigPathOverride;
96+
this.allowGkeCredentialBundle = allowGkeCredentialBundle;
97+
}
98+
99+
/**
100+
* Creates an X.509 provider that, in addition to the locations checked by {@link
101+
* #X509Provider()}, falls back to the GKE credential bundle when no certificate configuration is
102+
* present, so that the transport uses the same certificate that {@link
103+
* MtlsUtils#getWorkloadCertPath} reports for certificate-bound tokens.
104+
*/
105+
static X509Provider createWithGkeCredentialBundleFallback() {
106+
return new X509Provider(
107+
SystemEnvironmentProvider.getInstance(), SystemPropertyProvider.getInstance(), null, true);
76108
}
77109

78110
/**
@@ -105,6 +137,10 @@ public X509Provider() {
105137
* <li>The certificate config override path, if set.
106138
* <li>The path pointed to by the "GOOGLE_API_CERTIFICATE_CONFIG" environment variable
107139
* <li>The well known gcloud location for the certificate configuration file.
140+
* <li>If none of the above is configured or present, and this provider was created with {@link
141+
* #createWithGkeCredentialBundleFallback()}, the GKE credential bundle at {@code
142+
* /var/run/secrets/workload-spiffe-credentials/x509.credential-bundle.private-key.pem},
143+
* which contains both the certificate chain and the private key.
108144
* </ul>
109145
*
110146
* @return a KeyStore containing the X.509 certificate specified by the certificate configuration.
@@ -114,9 +150,48 @@ public X509Provider() {
114150
*/
115151
@Override
116152
public KeyStore getKeyStore() throws CertificateSourceUnavailableException, IOException {
117-
WorkloadCertificateConfiguration workloadCertConfig =
118-
MtlsUtils.getWorkloadCertificateConfiguration(
119-
envProvider, propProvider, certConfigPathOverride);
153+
WorkloadCertificateConfiguration workloadCertConfig;
154+
try {
155+
workloadCertConfig =
156+
MtlsUtils.getWorkloadCertificateConfiguration(
157+
envProvider, propProvider, certConfigPathOverride);
158+
} catch (CertificateSourceUnavailableException configUnavailable) {
159+
// Only fall back to the GKE credential bundle when created for the mTLS transport
160+
// (allowGkeCredentialBundle == true, rather than public X.509 WIF constructors) and when no
161+
// explicit certificate configuration path override was passed to the constructor.
162+
if (allowGkeCredentialBundle && certConfigPathOverride == null) {
163+
// getWorkloadCertificateConfiguration also throws CertificateSourceUnavailableException
164+
// when GOOGLE_API_CERTIFICATE_CONFIG is explicitly set or when a well-known
165+
// certificate_config.json exists with only a non-workload (e.g. ECP) configuration;
166+
// getGkeCredentialBundlePath checks hasCertificateConfiguration so we do not fall back to
167+
// the GKE bundle in those cases.
168+
String gkeCredentialBundlePath =
169+
MtlsUtils.getGkeCredentialBundlePath(envProvider, propProvider);
170+
if (gkeCredentialBundlePath != null) {
171+
try (InputStream bundleStream = new FileInputStream(new File(gkeCredentialBundlePath))) {
172+
return SecurityUtils.createMtlsKeyStore(bundleStream);
173+
} catch (FileNotFoundException e) {
174+
// The bundle was removed (e.g. mid-rotation) after the availability check.
175+
throw new CertificateSourceUnavailableException(
176+
"GKE credential bundle at "
177+
+ gkeCredentialBundlePath
178+
+ " was removed before it could be read (for example, during certificate"
179+
+ " rotation). Retry the request or verify that the GKE Pod Certificate volume"
180+
+ " is mounted.",
181+
e);
182+
} catch (Exception e) {
183+
throw new IOException(
184+
"X509Provider: Unable to load GKE credential bundle at "
185+
+ gkeCredentialBundlePath
186+
+ ". Verify that the file contains a valid PEM-encoded X.509 certificate chain"
187+
+ " and PKCS#8 private key, or set GOOGLE_API_USE_CLIENT_CERTIFICATE=false to"
188+
+ " disable mTLS.",
189+
e);
190+
}
191+
}
192+
}
193+
throw configUnavailable;
194+
}
120195

121196
// Read the certificate and private key file paths into streams.
122197
try (InputStream certStream = new FileInputStream(new File(workloadCertConfig.getCertPath()));

‎google-auth-library-java/oauth2_http/java/com/google/auth/oauth2/AgentIdentityCacheUtils.java‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -166,6 +166,12 @@ static final class CachedAgentIdentityInfo {
166166
*/
167167
final @Nullable CertInfo certInfo;
168168

169+
/**
170+
* Whether {@link #certMetadata} points to the well-known GKE credential bundle path ({@link
171+
* AgentIdentityUtils#isGkeCredentialBundle}).
172+
*/
173+
final boolean isGkeCredentialBundle;
174+
169175
/**
170176
* Constructs a cached snapshot of Agent Identity evaluation state and file metadata.
171177
*
@@ -190,6 +196,16 @@ static final class CachedAgentIdentityInfo {
190196
this.keyMetadata = keyMetadata;
191197
this.shouldRequestBoundToken = shouldRequestBoundToken;
192198
this.certInfo = certInfo;
199+
this.isGkeCredentialBundle =
200+
certMetadata != null && AgentIdentityUtils.isGkeCredentialBundle(certMetadata.getPath());
201+
}
202+
203+
/**
204+
* Returns {@code true} if this cached entry was discovered from the well-known directory (not a
205+
* certificate config file) and has both certificate and key metadata available for fallback.
206+
*/
207+
boolean canUseWellKnownFallback() {
208+
return configMetadata == null && certMetadata != null && keyMetadata != null;
193209
}
194210
}
195211

0 commit comments

Comments
 (0)