Agent re-parses the entire CA store on every harvest connection (Linux)

[junha6316]

Description

On Linux the agent parses the entire system CA store on every harvest connection rather than once per process. In a production Django + gevent application this accounted for 10.55% of on-CPU time and was the single largest self-time frame in the profile.

Two behaviors combine:

1. The agent deliberately closes its collector connection after each harvest (newrelic/core/application.py, comment: "Force close the socket connection which has been created for this harvest. New connection will be create automatically on the next harvest."). The next harvest therefore builds a new connection and a new SSLContext.

2. HttpClient.__init__ only sets ca_certs when the OS has no resolvable CA file (newrelic/common/agent_http.py). On typical Linux images get_default_verify_paths() does resolve, so ca_bundle_path stays None and ca_certs is never passed to the pool.

With no ca_certs, the vendored urllib3 takes its "load OS default certs" branch on every connect:

# newrelic/packages/urllib3/connection.py
default_ssl_context = False
if self.ssl_context is None:
    default_ssl_context = True
    self.ssl_context = create_urllib3_context(...)
...
if (
    not self.ca_certs
    and not self.ca_cert_dir
    and not self.ca_cert_data
    and default_ssl_context
    and hasattr(context, "load_default_certs")
):
    context.load_default_certs()

So the full CA store is re-read and re-parsed once per harvest connection.

Evidence

py-spy, 300 s at 99 Hz, --nonblocking, against a single gunicorn gevent worker serving real production traffic:

on-CPU samples (gevent hub idle excluded):  1431
SSL/TLS leaf frames:                         189  (13.21%)
  of which load_default_certs / set_default_verify_paths:
                                             151  (10.55%)

caller attribution for those 151 samples:
  100%   connect (newrelic/packages/urllib3/connection.py:417)
  chain: urlopen <- _make_request <- _validate_conn <- connect

All 151 samples are the agent. The workload runs the 5 s flexible harvest plus the 60 s default harvest, across two protocols (collector and OTLP, the latter active because application_logging.forwarding is enabled).

Why ca_bundle_path is not a workaround

Setting ca_bundle_path only moves the work from load_default_certs() to load_verify_locations(). Both parse a CA bundle, so the per-connection cost is unchanged:

load_default_certs()             median 3.351 ms
load_verify_locations(certifi)   median 3.431 ms

Suggested fix

Create the SSLContext once per HttpClient and pass it through the connection kwargs:

connection_kwargs["ssl_context"] = self._shared_ssl_context

HTTPSConnectionPool forwards **conn_kw to HTTPSConnection, and a supplied ssl_context makes default_ssl_context False, skipping the reload. Closing the socket per harvest continues to work; only the certificate parsing gets amortized.

We verified this by injecting a shared context into _connection_kwargs locally: creating five clients produced zero additional load_default_certs() calls, and verification settings were unchanged (verify_mode == CERT_REQUIRED, check_hostname is True, 128 CAs loaded).

Version

• Observed on 10.16.0.
• Still present on main (13.5.0). The block added to agent_http.py since 10.x is Windows-only; the POSIX path is unchanged.

Because ca_bundle_path stays None on Linux, Supportability/Python/Certificate/BundleRequired is never reported, so this path is likely invisible in your own telemetry.

Environment

• Python 3.11, Debian bookworm container
• Django 4.2 + DRF, gunicorn with gevent workers (monkey.patch_all())
• distributed_tracing.enabled = true, application_logging.forwarding.enabled = true

[Newrelic-Boomi]

https://new-relic.atlassian.net/browse/NR-609311