New Relic Python agent breaks SigV4 signing for `bedrock-agentcore:InvokeAgentRuntime` (13.x)

[alexcooper-economist]

New Relic Python agent breaks SigV4 signing for bedrock-agentcore:InvokeAgentRuntime (13.x)

Upgraded NR Python lib from 11.5 to 13.5 and it completely broke agentcore! Here is the indepth Claude (Opus 4.8) debugging/analysis - thanks!

Summary

With the Python agent active and connected (distributed tracing enabled), calls to
Amazon Bedrock AgentCore invoke_agent_runtime fail with:

An error occurred (AccessDeniedException) when calling the InvokeAgentRuntime operation:
The request signature we calculated does not match the signature you provided.
Check your AWS Secret Access Key and signing method.

Root cause: the agent injects its distributed-tracing headers (traceparent, tracestate,
newrelic) into the outbound HTTP request after botocore has already SigV4-signed it.
bedrock-agentcore:InvokeAgentRuntime is (as far as we can tell) the only AWS API whose
SigV4 SignedHeaders include traceparent — it is a modelled request header — so mutating
/ duplicating traceparent post-signing changes the canonical request the service reconstructs,
and signature verification fails.

Every other AWS API we use (bedrock-runtime, DynamoDB, OpenSearch, S3, …) does not sign
traceparent, so the injected header is harmless there. This is why only AgentCore calls fail
while all other AWS calls on the same process/credentials succeed.

Impact

• Regression appears when upgrading the agent 11.5.0 → 13.5.0 (all other deps, including
  boto3 1.43.87, held constant — see below).
• Fails deterministically only with a connected agent (monitor_mode = true), so it does not
  reproduce with the agent disabled/local dev.
• Not credential- or region-related: the same task/credentials sign every other AWS request fine.

Environment / libraries used to reproduce

Component	"Works"	"Fails"
newrelic	11.5.0	13.5.0
boto3	1.42.54	1.43.87
botocore	1.42.97	1.43.87
Python	3.14.3	3.14.3
Service	bedrock-agentcore (invoke_agent_runtime)	same
Distributed tracing	enabled, agent connected	enabled, agent connected

Isolation: we upgraded every dependency in our image except newrelic (kept boto3 at
1.43.87) and the failure disappeared. Reverting only newrelic 13.5.0 → 11.5.0 also fixes it.
botocore 1.42.97 vs 1.43.87 produce byte-identical SigV4 canonical requests for this call, so
the SDK is not involved.

The signed request

botocore signs the modelled traceparent header for this operation. Captured SignedHeaders
for invoke_agent_runtime:

SignedHeaders=host;traceparent;x-amz-date;x-amzn-bedrock-agentcore-runtime-session-id

The caller supplies traceParent (W3C format, e.g. 00-<32 hex>-<16 hex>-01) as an operation
parameter, which botocore serialises to the traceparent header and includes in the signature.

Where the agent injects (source references, 13.5.0)

newrelic/hooks/external_httplib.py — httplib_endheaders_wrapper (wraps
HTTPConnection.endheaders) adds NR headers to the already-built request just before send:

if not skip_headers and hasattr(tracer, "generate_request_headers"):
    outgoing_headers = tracer.generate_request_headers(transaction)
    for header_name, header_value in outgoing_headers:
        connection.putheader(header_name, header_value)

• newrelic/hooks/external_httplib2.py / external_urllib3.py route botocore's urllib3
  connections through the same endheaders instrumentation (sets _nr_library_info).
• newrelic/api/header_mixin.py → generate_request_headers → transaction.insert_distributed_trace_headers(...)
  emits traceparent / tracestate / newrelic when distributed tracing is enabled.

Because endheaders runs after botocore signs the request, the injected traceparent
is added on top of the one that was already signed → signature mismatch.

Relevant changes 11.5.0 → 13.5.0 (outbound-header path)

• newrelic/api/external_trace.py: ExternalTrace(CatHeaderMixin) → ExternalTrace(HeaderMixin).
• New newrelic/api/header_mixin.py replaces newrelic/api/cat_header_mixin.py;
  generate_request_headers dropped the legacy CAT branch and now emits W3C DT headers only.
• newrelic/hooks/external_httplib.py: the _nr_skip_headers detection narrowed from
  {"NEWRELIC", "X-NEWRELIC-ID", "X-NEWRELIC-TRANSACTION"} to {"NEWRELIC"}.

Steps to reproduce

1. Install newrelic==13.5.0, boto3==1.43.87 on Python 3.14.
2. Run under a connected agent with distributed tracing enabled
   (NEW_RELIC_DISTRIBUTED_TRACING_ENABLED=true, real license, monitor_mode=true), inside a
   web/background transaction.
3. Call:

   client = boto3.client("bedrock-agentcore", region_name="us-east-1")
   client.invoke_agent_runtime(
       agentRuntimeArn="arn:aws:bedrock-agentcore:us-east-1:<acct>:runtime/<id>/runtime-endpoint/PROD",
       payload=b'{"prompt": "hello"}',
       traceParent="00-" + "a"*32 + "-" + "b"*16 + "-01",
   )

4. Observe AccessDeniedException … signature does not match. With newrelic==11.5.0 (same
   boto3), the same call succeeds.

Note: the injection only manifests with a connected agent; a disabled/developer-mode agent
does not generate DT headers, so it will not reproduce.

Expected behaviour

The agent should not inject distributed-tracing headers into AWS SigV4-signed requests (it
already avoids breaking signing for other AWS services). For bedrock-agentcore, whose signed
headers include traceparent, injecting/overwriting traceparent after signing must be avoided.

[Newrelic-Boomi]

https://new-relic.atlassian.net/browse/NR-615001

[TimPansino]

What other tracing products are you using besides newrelic? Do you have OpenTelemetry installed by chance? It looks like the issue is caused by the traceparent header already existing before our instrumentation would inject it, so this particular issue is likely a conflict with the other tracing provider.

I've opened a PR to insert DT headers in botocore before signing actually occurs to prevent the current issue.

[alexcooper-economist]

Cool thanks.

We aren't adding any extra libs. We are setting it ourselves:

"traceParent": _format_traceparent(request.state.trace_id),

https://docs.aws.amazon.com/boto3/latest/reference/services/bedrock-agentcore/client/invoke_agent_runtime.html#

It's a param here:

response = client.invoke_agent_runtime(
    contentType='string',
    accept='string',
    mcpSessionId='string',
    runtimeSessionId='string',
    mcpProtocolVersion='string',
    mcpMethod='string',
    mcpName='string',
    runtimeUserId='string',
    traceId='string',
    traceParent='string',

So botocore serializes it to the traceparent header and signs it (SignedHeaders = host;traceparent;x-amz-date;x-amzn-bedrock-agentcore-runtime-session-id). The agent then injects/duplicates traceparent after signing, which breaks SigV4.

But NR shouldn't override it, as then AWS functionality will break. (we are trying to send traceid from NR FastAPI app to Bedrock Agentcore)