Skip to content

build: profile-guided optimization for natively built release wheels - #68

Merged
mrecachinas merged 2 commits into
perf/scan-poolfrom
build/pgo
Sep 23, 2026
Merged

mrecachinas merged 2 commits into
perf/scan-poolfrom
build/pgo

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

The Linux x86_64, macOS arm64 and Windows release wheels become profile-guided (PGO) builds:

  1. Build an instrumented wheel.
  2. Run scripts/pgo_train.py against it.
  3. Merge the profile with llvm-profdata.
  4. Rebuild every wheel with the profile.

How the pipeline works (in the reusable pgo-wheels.yml)

  • Releases call it on tags. Cross-built targets (Linux aarch64, macOS x86_64) keep plain builds, since the training can't run on their runner.
  • Pull requests run it only with the ci:wheels label. It builds and tests the PGO wheels, then compares them with a plain build on the same runner. The comparison uses scripts/pgo_bench.py (best of 5 alternating runs) and is written to the job summary. Nothing is published from PRs.
  • Toolchain: the manylinux container installs the same Rust version as the host, so llvm-profdata matches the compiler.
  • Interpreters: training uses each runner's default interpreter, so the set of published wheels is unchanged.

Training and flags

  • The training mix mirrors typical use: small single calls; str/bytes/bytearray/memoryview inputs; fixed-width and generic scans, both serial and pooled; pairwise and many-query batches; catalogs with and without an index; and error paths.
  • Profile-guided size optimization is disabled (-Cllvm-args=-pgso=false). Without that, paths the training never ran were size-optimized: pairwise_into at 8 B was 0.47× and set_algo 0.90×.

Measured on the full stack (Apple M4 Max, same toolchain for both builds, interleaved A/B):

Case Speedup
hamming_distance_string 16–64 chars 1.08–1.11×
hamming_distance_bytes 16/64 B 1.04–1.11×
within-distance calls, scans, pairwise, many-query neutral

The 1.2–1.6× PGO gains from the original analysis came mostly from inlining PyO3's argument glue, which #64 already removed. So this is a modest win on the hottest small calls.

⚠️ This PR carries the ci:wheels label, so its checks include the PGO build on all three platforms and x86 PGO-vs-plain timings (see the job summaries). Before the next release, check that the wheel set and timings look right.

x86 (this PR's CI, linux-x86_64, one run each): best/all over 1024×16 records are 1.17–1.23× faster with PGO, and 16384×64 scans are 1.04× faster. Small calls land at 0.94–1.05×, within single-run noise.

PyPy: the first PGO run failed compiling the PyPy 3.11 wheel (--find-interpreter includes PyPy). That was a real release blocker, now fixed in #57 (PyO3 0.29 on main) and #64 (the raw bindings). The matrix no longer fails fast, so one platform's failure doesn't hide the others' results.

Found while here: PyPI 3.0.0 ships only one CPython wheel for macOS (cp314) and one for Windows (cp312). Those jobs build for the runner's default interpreter, and this PR keeps that behavior. Adding --find-interpreter (with setup-python for 3.10–3.14) would publish wheels for every supported Python.

@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 03:50
@mrecachinas mrecachinas added the ci:wheels Build and benchmark PGO wheels on this pull request label Sep 23, 2026
@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison Results

✅ All benchmarks within noise

104 benchmarks compared, no significant changes detected.

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

mrecachinas and others added 2 commits September 23, 2026 07:54
The Linux x86_64, macOS arm64 and Windows wheels are now PGO builds:
an instrumented wheel runs scripts/pgo_train.py, llvm-profdata merges
the profile, and every wheel is rebuilt with it. The pipeline lives in
the reusable pgo-wheels.yml:

- The release calls it on tags. Cross-built targets (Linux aarch64,
  macOS x86_64) keep plain builds, since the training cannot run on
  their runner.
- On a pull request it runs only with the `ci:wheels` label. It builds
  and tests the PGO wheels, then compares them with a plain build on
  the same runner (scripts/pgo_bench.py, best of 5 alternating runs) in
  the job summary. Nothing is published from pull requests.
- The manylinux container installs the same toolchain version as the
  host, so llvm-profdata matches the compiler.
- Training uses the runner's default interpreter, so the set of
  published wheels is unchanged.

The training mix mirrors typical use: small single calls, str/bytes/
bytearray/memoryview inputs, fixed-width and generic scans (serial and
pooled), pairwise and many-query batches, catalogs with and without an
index, and error paths. Profile-guided size optimization is disabled
(-Cllvm-args=-pgso=false). With it, paths the training never ran
became size-optimized: pairwise_into at 8 B was 0.47x and set_algo
0.90x.

Measured on the full stack (Apple M4 Max, same toolchain for both
builds, interleaved A/B):
  hamming_distance_string/16..64   1.08-1.11x
  hamming_distance_bytes/16,64     1.04-1.11x
  check_*_within_dist, scans, pairwise, many-query: neutral
The ~1.2-1.6x PGO gains in the original analysis came mostly from
inlining PyO3's argument glue, which the raw FASTCALL layer already
removed. x86 numbers will come from the `ci:wheels` run.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
With fail-fast, the PyPy compile error on Linux cancelled the macOS job,
hiding whether it would have passed. Publishing already waits for every
platform, so letting the others finish costs nothing.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas merged commit 4bc6ffe into main Sep 23, 2026
35 checks passed
@mrecachinas
mrecachinas deleted the build/pgo branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci:wheels Build and benchmark PGO wheels on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant