Skip to content

feat: thin lto for non consuming artifacts - #2396

Open
Aaalibaba42 wants to merge 1 commit into
mainfrom
jwiriath/APMSP-3950-thin-lto
Open

feat: thin lto for non consuming artifacts#2396
Aaalibaba42 wants to merge 1 commit into
mainfrom
jwiriath/APMSP-3950-thin-lto

Conversation

@Aaalibaba42

Copy link
Copy Markdown
Contributor

Fat LTO + codegen-units=1 makes the Generate profiling FFI step slow for no reason, this doesn't ship an artifact anyone consumes. Override to thin LTO / codegen-units=16 via CARGO_PROFILE_RELEASE_* env vars for this build only

@Aaalibaba42
Aaalibaba42 requested a review from a team as a code owner August 24, 2026 12:47

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ac022a4654

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment on lines +121 to +122
CARGO_PROFILE_RELEASE_LTO: "thin"
CARGO_PROFILE_RELEASE_CODEGEN_UNITS: "16"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Include the profile overrides in the Rust cache key

When the ffi job restores an existing main-branch cache, these variables are visible only to the later Generate step, so the preceding Swatinem/rust-cache action computes the same key as for the fat-LTO configuration. Cargo then invalidates and rebuilds the restored release artifacts, while the immutable exact-key cache prevents the newly built thin-LTO artifacts from replacing them; consequently subsequent runs can repeatedly rebuild the dependency graph and lose the intended speedup. Put these overrides at job scope or otherwise incorporate them into the cache key.

Useful? React with 👍 / 👎.

@datadog-datadog-prod-us1-2

datadog-datadog-prod-us1-2 Bot commented Aug 24, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 100.00%
Overall Coverage: 76.63% (-0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: ac022a4 | Docs | View more details | Give us feedback!

@pr-commenter

pr-commenter Bot commented Aug 24, 2026

Copy link
Copy Markdown

Benchmarks

Comparison

Benchmark execution time: 2026-08-24 13:21:12

Comparing candidate commit ac022a4 in PR branch jwiriath/APMSP-3950-thin-lto with baseline commit 9d6b2ff in branch main.

Found 1 performance improvements and 5 performance regressions! Performance is the same for 146 metrics, 0 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:flagevaluation_evp/payloads/stress/10flags_1000users_250fields

  • 🟥 execution_time [+4.363ms; +4.599ms] or [+5.144%; +5.423%]
  • 🟥 throughput [-608.276op/s; -576.570op/s] or [-5.158%; -4.890%]

scenario:msgpack_decoder::v05/low_sharing/10000

  • 🟥 execution_time [+1.095ms; +1.102ms] or [+6.301%; +6.338%]
  • 🟥 throughput [-34295.502op/s; -34089.816op/s] or [-5.962%; -5.926%]

scenario:normalization/normalize_trace/test_trace

  • 🟩 execution_time [-56.562ns; -51.553ns] or [-18.583%; -16.937%]

scenario:vec_map/as_deduped_map/needs_dedup_1_in_4/8

  • 🟥 execution_time [+19.825ns; +20.022ns] or [+5.050%; +5.100%]

Benchmark execution time: 2026-08-24 13:28:06

Comparing candidate commit ac022a4 in PR branch jwiriath/APMSP-3950-thin-lto with baseline commit 9d6b2ff in branch main.

Found 2 performance improvements and 2 performance regressions! Performance is the same for 164 metrics, 10 unstable metrics.

Explanation

This is an A/B test comparing a candidate commit's performance against that of a baseline commit. Performance changes are noted in the tables below as:

  • 🟩 = significantly better candidate vs. baseline
  • 🟥 = significantly worse candidate vs. baseline

We compute a confidence interval (CI) over the relative difference of means between metrics from the candidate and baseline commits, considering the baseline as the reference.

If the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD), the change is considered significant.

Feel free to reach out to #apm-benchmarking-platform on Slack if you have any questions.

More details about the CI and significant changes

You can imagine this CI as a range of values that is likely to contain the true difference of means between the candidate and baseline commits.

CIs of the difference of means are often centered around 0%, because often changes are not that big:

---------------------------------(------|---^--------)-------------------------------->
                              -0.6%    0%  0.3%     +1.2%
                                 |          |        |
         lower bound of the CI --'          |        |
sample mean (center of the CI) -------------'        |
         upper bound of the CI ----------------------'

As described above, a change is considered significant if the CI is entirely outside the configured SIGNIFICANT_IMPACT_THRESHOLD (or the deprecated UNCONFIDENCE_THRESHOLD).

For instance, for an execution time metric, this confidence interval indicates a significantly worse performance:

----------------------------------------|---------|---(---------^---------)---------->
                                       0%        1%  1.3%      2.2%      3.1%
                                                  |   |         |         |
       significant impact threshold --------------'   |         |         |
                      lower bound of CI --------------'         |         |
       sample mean (center of the CI) --------------------------'         |
                      upper bound of CI ----------------------------------'

scenario:datadog_sample_span/name_pattern_rule_not_matching/wall_time

  • 🟥 execution_time [+23.945ns; +24.028ns] or [+14.897%; +14.949%]

scenario:no_profiler/short_circuit/4096

  • 🟩 execution_time [-13.891ns; -13.761ns] or [-13.401%; -13.276%]

scenario:profiler_attached/fast_path_system/4096

  • 🟥 execution_time [+6.147ns; +6.310ns] or [+6.229%; +6.395%]

scenario:profiler_attached/slow_path_system/4096

  • 🟩 execution_time [-25.442ns; -25.350ns] or [-16.555%; -16.495%]

Candidate

Omitted due to size.

Baseline

Omitted due to size.

@dd-octo-sts

dd-octo-sts Bot commented Aug 24, 2026

Copy link
Copy Markdown
Contributor

Artifact Size Benchmark Report

aarch64-alpine-linux-musl
Artifact Baseline Commit Change
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.a 89.52 MB 89.52 MB 0% (0 B) 👌
/aarch64-alpine-linux-musl/lib/libdatadog_profiling.so 8.26 MB 8.26 MB 0% (0 B) 👌
aarch64-unknown-linux-gnu
Artifact Baseline Commit Change
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.a 100.70 MB 100.70 MB 0% (0 B) 👌
/aarch64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.06 MB 11.06 MB 0% (0 B) 👌
libdatadog-x64-windows
Artifact Baseline Commit Change
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.dll 26.64 MB 26.64 MB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.lib 94.96 KB 94.96 KB 0% (0 B) 👌
/libdatadog-x64-windows/debug/dynamic/datadog_profiling_ffi.pdb 180.73 MB 180.73 MB +0% (+8.00 KB) 👌
/libdatadog-x64-windows/debug/static/datadog_profiling_ffi.lib 772.85 MB 772.85 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.dll 8.73 MB 8.73 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.lib 94.96 KB 94.96 KB 0% (0 B) 👌
/libdatadog-x64-windows/release/dynamic/datadog_profiling_ffi.pdb 25.65 MB 25.65 MB 0% (0 B) 👌
/libdatadog-x64-windows/release/static/datadog_profiling_ffi.lib 51.09 MB 51.09 MB 0% (0 B) 👌
libdatadog-x86-windows
Artifact Baseline Commit Change
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.dll 23.22 MB 23.22 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.lib 96.45 KB 96.45 KB 0% (0 B) 👌
/libdatadog-x86-windows/debug/dynamic/datadog_profiling_ffi.pdb 185.66 MB 185.66 MB 0% (0 B) 👌
/libdatadog-x86-windows/debug/static/datadog_profiling_ffi.lib 759.09 MB 759.09 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.dll 6.74 MB 6.74 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.lib 96.45 KB 96.45 KB 0% (0 B) 👌
/libdatadog-x86-windows/release/dynamic/datadog_profiling_ffi.pdb 27.59 MB 27.59 MB 0% (0 B) 👌
/libdatadog-x86-windows/release/static/datadog_profiling_ffi.lib 48.62 MB 48.62 MB 0% (0 B) 👌
x86_64-alpine-linux-musl
Artifact Baseline Commit Change
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.a 79.78 MB 79.78 MB 0% (0 B) 👌
/x86_64-alpine-linux-musl/lib/libdatadog_profiling.so 9.16 MB 9.16 MB 0% (0 B) 👌
x86_64-unknown-linux-gnu
Artifact Baseline Commit Change
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.a 95.42 MB 95.42 MB 0% (0 B) 👌
/x86_64-unknown-linux-gnu/lib/libdatadog_profiling.so 11.17 MB 11.17 MB 0% (0 B) 👌

Comment on lines +121 to +122
CARGO_PROFILE_RELEASE_LTO: "thin"
CARGO_PROFILE_RELEASE_CODEGEN_UNITS: "16"

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of hardcoding these values in the workflow files we could setup a profile for the CI in the Cargo.toml the downside is that you would need to include some plumbering in the builder crate in order to pass the profile. That way we could reuse that profile in libdatadog, libddprof-build and also downstream projects that use the builder. Wdyt? It can be done in a subsequent PR if you want to test this approach first.
Also another advantage is that you can include the profile in the cache key rather than manually setting them depending on the job.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants