Skip to content

docs: refresh README benchmarks for the optimized stack - #69

Merged
mrecachinas merged 2 commits into
build/pgofrom
docs/final-benchmarks
Sep 23, 2026
Merged

mrecachinas merged 2 commits into
build/pgofrom
docs/final-benchmarks

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Top of the stack. This PR re-measures every table in the README benchmark section on the final stack, and adds scripts/readme_benchmarks.py to regenerate the Python ones.

Changes

  • scripts/readme_benchmarks.py prints the Python tables as reStructuredText, ready to paste. Each value is the median of --runs passes, and each pass is the best of seven timeit repeats. It replaces .benchmarks/batch_measure.py, which the README cited but the repository doesn't contain.
  • Rust (Criterion) and Python tables are re-measured. The matrix tables are rebuilt with column widths that fit their headers; on main, docutils rejected one of them as malformed.
  • A new "Large catalogs" table covers single-query scans on the thread pool.
  • The Catalog index table is re-measured against the free function, which now scans on every core. It also notes that the planner scans linearly at radius 12.
  • The check_hexstrings_within_dist row uses the test suite's early-exit inputs, sliced once outside the timed call.
  • The x86 notes describe the SDE and Benchmark PR jobs. They add the AVX-512 block scanners' speedup over AVX2 (2.5–4.8×), measured when perf(x86): AVX-512 block scanners, VBMI hex parse, short-input POPCNT #63's Benchmark PR run landed on an Emerald Rapids host, and label the older AVX-512 table as predating them.
  • The batch-API interpretation is updated for the faster call path and for the serial prefix of early-exit batches.
  • .github/copilot-instructions.md explains how to regenerate the tables.

main → this stack, same script and machine (Apple M4 Max, CPython 3.14.7, local builds without PGO; main is the median of 3 runs, this stack of 7):

Case main Stack Speedup
hamming_distance_string, 3 chars 40.5 ns 30.7 ns 1.32×
hamming_distance_string, 1024 chars 116 ns 78.8 ns 1.47×
check_bytes_within_dist, 127 B 43.7 ns 34.3 ns 1.27×
first_within_dist 512×16, match at end 176 ns 128 ns 1.37×
first_within_dist 16384×64, match at end 19.7 µs 4.65 µs 4.23×
best_within_dist 16384×64, match at mid 11.9 µs 4.80 µs 2.49×
all_within_dist 16384×64 21.7 µs 4.64 µs 4.68×
all, 1024×16, random no-match 366 ns 239 ns 1.53×
pairwise into, 10,000×32 22.4 µs 5.88 µs 3.81×
best_many 100×1024×16 43.0 µs 7.87 µs 5.46×
dense all_into, 1024×16, all matching 1280 ns 968 ns 1.32×
first, 16 MiB, no match 262 µs 33.8 µs 7.74×
best, 64 MiB, no match (memory-bound) 356 µs 247 µs 1.44×
first_many 100×1024×16, permissive 442 ns 468 ns 0.94×
first_within_dist 512×16, match at record 0 40.1 ns 39.7 ns 1.0×

The last two rows are about even. Block scanners compare 16 records before an early match can end the scan, so a match on the very first record costs a whole block.

Documentation and a benchmark script only; no library changes.

Re-measure every table in the benchmark section on the final stack and
add scripts/readme_benchmarks.py, which regenerates the Python tables as
reStructuredText (median of seven runs, each the best of seven timeit
repeats). It replaces the .benchmarks/batch_measure.py the README cited,
which is not in the repository.

* Rust (Criterion) and Python tables re-measured; the malformed matrix
  tables are rebuilt with column widths that fit their headers.
* New "Large catalogs" table for single-query scans on the thread pool.
* Catalog index table re-measured against the now-parallel free
  function, noting the planner's linear choice at radius 12.
* The check_hexstrings_within_dist row uses the same early-exit inputs as
  the test suite's benchmark, sliced once outside the timed call.
* The x86 notes describe the SDE and Benchmark PR CI jobs; the AVX-512
  table is labelled as measured before the current block scanners.
* Batch-API interpretation updated for the faster per-call path and the
  serial prefix of early-exit batches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 11:59
@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison Results

✅ All benchmarks within noise

104 benchmarks compared, no significant changes detected.

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

The Benchmark PR run for #63 landed on an AVX-512 host (Xeon Platinum
8573C), comparing the AVX-512 block scanners with the AVX2 ones on the
same machine: 2.5-4.8x faster for 16-, 32- and 64-byte records.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas merged commit 05716a7 into main Sep 23, 2026
33 checks passed
@mrecachinas
mrecachinas deleted the docs/final-benchmarks branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant