Skip to content

perf(scan): replace rayon with a spin-then-park scan pool - #67

Merged
mrecachinas merged 3 commits into
perf/parallel-scansfrom
perf/scan-pool
Sep 23, 2026
Merged

mrecachinas merged 3 commits into
perf/parallel-scansfrom
perf/scan-pool

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Waking rayon's workers costs 15–40 µs (median) per parallel scan. That pushed the parallel thresholds up to MiB sizes, and made first() slower than serial when the match comes early.

This PR replaces rayon with a small spin-then-park pool behind the same par::map_chunks / find_first_chunk interface from #66.

Design

  • Dispatch: idle workers spin for 50 µs, then park, so back-to-back scans start in 0.5–3 µs, depending on how many chunks a job has. The caller works on its own job, and workers claim chunks dynamically.
  • Job lifetime: a job lives on the caller's stack. A worker sets a flag on its own cache line before reading it; the caller unpublishes the job and waits only for flagged workers, never for parked ones.
  • Participation: a job draws in only as many workers as it has chunks beyond the caller's; the rest see it and sit it out without touching shared state. The only contended atomic left is the chunk claim.
  • Concurrency: one job runs at a time. Concurrent or nested callers run their chunks inline.
  • Panics: a chunk panic is caught and re-raised in the caller.
  • Fork: a forked child starts its own pool.
  • Configuration: HEXHAMMING_NUM_THREADS sets the size. RAYON_NUM_THREADS is still honored for compatibility, and 1 disables the pool. Threads start on the first scan that splits.
  • Chunking: chunks are sized so a scan at the threshold gets 8 of them, with up to 4 per thread. The claim counter sits on its own cache line; tiny chunks with per-chunk counters cost about 12 µs of contention.

rayon is no longer a library dependency (criterion still uses it for benches).

Thresholds. With cheaper dispatch, splitting pays off much earlier, so the thresholds are re-measured:

  • Block scanners: 4 MiB → 512 KiB (0.8× at 256 KiB, 1.4–1.7× at 512 KiB, 2.5× at 1 MiB).
  • Generic widths: 1 MiB → 64 KiB (2.0–3.5× at 128 KiB).
  • The Catalog planner's parallel-speedup model is refitted to match.

Interleaved A/B (Apple M4 Max, CPython 3.14, #66 → this PR):

Case rayon pool Speedup
best 0.5 MB×16 6.66 µs 4.80 µs 1.39×
best 1 MB×16 13.2 µs 4.08 µs 3.23×
best 2 MB×16 26.2 µs 6.86 µs 3.81×
best 4 MB×16 49.3 µs 11.8 µs 4.19×
best 8 MB×64 76.2 µs 19.6 µs 3.88×
best 16 MB×16 116 µs 45.1 µs 2.56×
best 64 MB×16 (memory-bound) 314 µs 269 µs 1.17×
first 16 MB×16, match @25% 72.8 µs 14.4 µs 5.04×
best 0.5 MB×20 (generic) 79.2 µs 13.6 µs 5.80×
many_best 100q × 1024×16 21.8 µs 7.73 µs 2.82×

Calls and scans under 512 KiB are unchanged (±3%). This table predates the dispatch-cost commit below, which is faster again near the threshold.

Dispatch cost. Profiling the pool with empty chunks showed about 3 µs per job even with 2 chunks. Every job made all 15 workers do contended read-modify-writes on a shared users counter (on join and leave) and on completed. At 512 KiB that was most of the scan. The second commit keeps uninvolved workers out and moves join/leave to per-worker cache lines. Spinning workers, best of 5 × 20000 calls:

Chunks Empty chunks 1 µs chunks
2 2.96 → 0.42 µs 2.56 → 1.55 µs
8 2.94 → 1.46 µs 4.12 → 2.67 µs
16 4.72 → 2.95 µs 5.10 → 3.79 µs
64 9.43 → 7.78 µs 11.12 → 8.79 µs

End to end, interleaved: 0.5 MiB × 16 B scans are 1.50–1.72× faster, 0.5 MiB × 64 B 1.61–1.99×, and 1–2 MiB 1.04–1.36×. Larger scans and generic widths are within noise; that was checked against a second build of the previous pool used as a control. At the thresholds the parallel median beats serial: 512 KiB × 16 B takes 3.3–3.7 µs against 6.7–6.9 µs.

A sweep with lower thresholds (128 KiB / 16 KiB) did not justify moving them. Below 512 KiB, splitting wins on best-case times but loses on medians (0.15–0.96×) for block-scanner widths, and gains little (0.85–1.33×) for generic widths.

Tried and dropped: letting each participant claim the chunk of its own number through its own cache line. Empty 16-chunk jobs got faster (2.95 → 1.60 µs), but with real work they got slower (16 × 1 µs: 3.79 → 4.80 µs), because a late-joining worker's chunk waits for the caller.

Tests

  • The pool keeps chunk order, propagates panics and stays usable afterwards.
  • It serves 8 concurrent callers with nested jobs.
  • It survives 2000 back-to-back jobs that cross the spin/park boundary.
  • Alternating small and large jobs, so workers left out of one must rejoin the next, runs every chunk exactly once.
  • It ran clean in 40× and 20× stress loops, and under Miri (data-race and aliasing checks, 4 threads, 16 seeds).
  • A new CI job, pool-sanitizers, runs these tests under ThreadSanitizer and Miri on Linux. (TSan's runtime is broken on this Mac, so CI is its first run.)
  • Python tests cover scans in a forked child and HEXHAMMING_NUM_THREADS=1/3.

Review focus: the job hand-off in src/par.rs: the per-worker active flags, helpers, and the unpublish protocol in Pool::run.

@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 03:29
@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown

📊 Benchmark Comparison Results

Status Benchmark Base PR Delta
✅ check_bytes_arrays_all_within_dist[16384 elems,s=64,mid] 0.0388ms 0.0211ms 45.6% faster
✅ check_bytes_arrays_all_within_dist[16384 elems,s=64,end] 0.0388ms 0.0212ms 45.5% faster
✅ check_bytes_arrays_first_within_dist[16384 elems,s=64,end] 0.0337ms 0.0197ms 41.5% faster
✅ check_bytes_arrays_best_within_dist[16384 elems,s=64,end] 0.0367ms 0.0217ms 40.8% faster
✅ check_bytes_arrays_best_within_dist[16384 elems,s=64,mid] 0.0361ms 0.0216ms 40.0% faster
✅ check_bytes_arrays_best_within_dist[16384 elems,s=64,at 0] 0.0356ms 0.0216ms 39.3% faster
➖ 98 benchmarks within noise (click to expand)
Benchmark Base PR Delta
check_bytes_arrays_all_within_dist[ 512 elems,s=16,at 0] 0.0006ms 0.0005ms 20.6% faster
check_bytes_arrays_all_within_dist[ 512 elems,s=16,end] 0.0005ms 0.0005ms 3.1% faster
check_bytes_arrays_all_within_dist[ 512 elems,s=16,mid] 0.0005ms 0.0005ms 2.7% faster
check_bytes_arrays_all_within_dist[ 1024 elems,s=32,at 0] 0.0016ms 0.0016ms 1.2% slower
check_bytes_arrays_all_within_dist[ 1024 elems,s=32,end] 0.0017ms 0.0017ms 1.8% slower
check_bytes_arrays_all_within_dist[ 1024 elems,s=32,mid] 0.0017ms 0.0017ms 0.6% faster
check_bytes_arrays_all_within_dist[16384 elems,s=64,at 0] 0.0386ms 0.0211ms 45.4% faster
check_bytes_arrays_best_within_dist[ 512 elems,s=16,at 0] 0.0001ms 0.0001ms 7.2% slower
check_bytes_arrays_best_within_dist[ 512 elems,s=16,end] 0.0005ms 0.0005ms 2.0% slower
check_bytes_arrays_best_within_dist[ 512 elems,s=16,mid] 0.0003ms 0.0003ms 3.9% slower
check_bytes_arrays_best_within_dist[ 1024 elems,s=32,at 0] 0.0014ms 0.0014ms 0.5% faster
check_bytes_arrays_best_within_dist[ 1024 elems,s=32,end] 0.0013ms 0.0014ms 5.2% slower
check_bytes_arrays_best_within_dist[ 1024 elems,s=32,mid] 0.0014ms 0.0014ms 1.3% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,at 0] 0.0001ms 0.0001ms 6.0% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,end] 0.0004ms 0.0004ms 0.0% slower
check_bytes_arrays_first_within_dist[ 512 elems,s=16,mid] 0.0002ms 0.0002ms 5.2% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,at 0] 0.0001ms 0.0001ms 4.7% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,end] 0.0014ms 0.0014ms 0.5% slower
check_bytes_arrays_first_within_dist[ 1024 elems,s=32,mid] 0.0007ms 0.0007ms 1.6% slower
check_bytes_arrays_first_within_dist[16384 elems,s=64,at 0] 0.0001ms 0.0002ms 107.4% slower
check_bytes_arrays_first_within_dist[16384 elems,s=64,mid] 0.0169ms 0.0106ms 37.2% faster
check_bytes_within_dist[127 bytes,d=500] 0.0001ms 0.0001ms 5.1% faster
check_bytes_within_dist[16 bytes,d=0] 0.0001ms 0.0001ms 14.4% slower
check_bytes_within_dist[64 bytes,d=100] 0.0001ms 0.0001ms 12.7% slower
hex_within_dist 0.0001ms 0.0001ms 6.8% faster
fixed_width_array_scan_matrix[all-exact_early-16] 0.0008ms 0.0008ms 2.1% faster
fixed_width_array_scan_matrix[all-exact_early-32] 0.0017ms 0.0016ms 4.8% faster
fixed_width_array_scan_matrix[all-exact_late-16] 0.0008ms 0.0008ms 2.5% faster
fixed_width_array_scan_matrix[all-exact_late-32] 0.0017ms 0.0017ms 0.2% slower
fixed_width_array_scan_matrix[all-exact_mid-16] 0.0008ms 0.0008ms 1.8% faster
fixed_width_array_scan_matrix[all-exact_mid-32] 0.0018ms 0.0017ms 4.5% faster
fixed_width_array_scan_matrix[all-random_no_match-16] 0.0007ms 0.0007ms 1.8% faster
fixed_width_array_scan_matrix[all-random_no_match-32] 0.0016ms 0.0016ms 0.3% faster
fixed_width_array_scan_matrix[all-threshold_d-16] 0.0008ms 0.0008ms 1.8% faster
fixed_width_array_scan_matrix[all-threshold_d-32] 0.0017ms 0.0017ms 0.0% faster
fixed_width_array_scan_matrix[all-threshold_d_minus_1-16] 0.0007ms 0.0007ms 0.7% faster
fixed_width_array_scan_matrix[all-threshold_d_minus_1-32] 0.0016ms 0.0016ms 0.8% faster
fixed_width_array_scan_matrix[all-threshold_d_plus_1-16] 0.0008ms 0.0008ms 1.8% faster
fixed_width_array_scan_matrix[all-threshold_d_plus_1-32] 0.0017ms 0.0017ms no change
fixed_width_array_scan_matrix[best-exact_early-16] 0.0001ms 0.0001ms 5.7% slower
fixed_width_array_scan_matrix[best-exact_early-32] 0.0001ms 0.0001ms 1.0% slower
fixed_width_array_scan_matrix[best-exact_late-16] 0.0010ms 0.0010ms 0.0% faster
fixed_width_array_scan_matrix[best-exact_late-32] 0.0013ms 0.0014ms 2.4% slower
fixed_width_array_scan_matrix[best-exact_mid-16] 0.0005ms 0.0005ms 1.2% slower
fixed_width_array_scan_matrix[best-exact_mid-32] 0.0007ms 0.0007ms 2.4% slower
fixed_width_array_scan_matrix[best-random_no_match-16] 0.0009ms 0.0009ms 1.1% slower
fixed_width_array_scan_matrix[best-random_no_match-32] 0.0013ms 0.0014ms 2.6% slower
fixed_width_array_scan_matrix[best-threshold_d-16] 0.0009ms 0.0009ms 1.2% faster
fixed_width_array_scan_matrix[best-threshold_d-32] 0.0014ms 0.0014ms 0.9% slower
fixed_width_array_scan_matrix[best-threshold_d_minus_1-16] 0.0009ms 0.0009ms 1.0% slower
fixed_width_array_scan_matrix[best-threshold_d_minus_1-32] 0.0014ms 0.0014ms 2.7% faster
fixed_width_array_scan_matrix[best-threshold_d_plus_1-16] 0.0009ms 0.0009ms 1.1% faster
fixed_width_array_scan_matrix[best-threshold_d_plus_1-32] 0.0014ms 0.0014ms 0.3% slower
fixed_width_array_scan_matrix[first-exact_early-16] 0.0001ms 0.0001ms 3.7% slower
fixed_width_array_scan_matrix[first-exact_early-32] 0.0001ms 0.0001ms 1.0% slower
fixed_width_array_scan_matrix[first-exact_late-16] 0.0007ms 0.0007ms 1.3% slower
fixed_width_array_scan_matrix[first-exact_late-32] 0.0014ms 0.0014ms 0.5% faster
fixed_width_array_scan_matrix[first-exact_mid-16] 0.0004ms 0.0004ms 1.0% slower
fixed_width_array_scan_matrix[first-exact_mid-32] 0.0007ms 0.0007ms 0.6% slower
fixed_width_array_scan_matrix[first-random_no_match-16] 0.0007ms 0.0007ms 1.4% slower
fixed_width_array_scan_matrix[first-random_no_match-32] 0.0014ms 0.0014ms 0.4% faster
fixed_width_array_scan_matrix[first-threshold_d-16] 0.0004ms 0.0004ms 1.2% slower
fixed_width_array_scan_matrix[first-threshold_d-32] 0.0007ms 0.0007ms 1.7% faster
fixed_width_array_scan_matrix[first-threshold_d_minus_1-16] 0.0007ms 0.0007ms 1.2% slower
fixed_width_array_scan_matrix[first-threshold_d_minus_1-32] 0.0014ms 0.0014ms 0.5% faster
fixed_width_array_scan_matrix[first-threshold_d_plus_1-16] 0.0004ms 0.0004ms 1.6% slower
fixed_width_array_scan_matrix[first-threshold_d_plus_1-32] 0.0007ms 0.0007ms 0.5% slower
hd_bytes[1000-diff] 0.0001ms 0.0001ms 7.2% slower
hd_bytes[1000-same] 0.0001ms 0.0001ms 2.4% slower
hd_bytes[1024-diff] 0.0001ms 0.0001ms 4.4% faster
hd_bytes[1024-same] 0.0001ms 0.0001ms 2.0% slower
hd_bytes[3-diff] 0.0001ms 0.0001ms 1.5% slower
hd_bytes[3-same] 0.0001ms 0.0001ms 1.2% faster
hd_bytes[64-diff] 0.0001ms 0.0001ms 3.4% slower
hd_bytes_buffer[bytearray] 0.0001ms 0.0002ms 84.0% slower
hd_bytes_buffer[memoryview] 0.0001ms 0.0001ms 5.3% slower
hd_bytes_gil_boundary[16376] 0.0005ms 0.0005ms 1.3% slower
hd_bytes_gil_boundary[16384] 0.0005ms 0.0005ms 0.7% slower
hd_bytes_gil_boundary[16392] 0.0005ms 0.0005ms 1.4% slower
hd_bytes_random[1024] 0.0001ms 0.0001ms 11.0% slower
hd_bytes_random[16] 0.0001ms 0.0001ms 10.1% faster
hd_bytes_random[64] 0.0001ms 0.0001ms 4.6% faster
hd_string[1000-diff] 0.0001ms 0.0001ms 1.9% slower
hd_string[1000-same] 0.0001ms 0.0001ms 5.1% slower
hd_string[1024-diff] 0.0001ms 0.0001ms 0.2% faster
hd_string[1024-same] 0.0001ms 0.0001ms 0.6% slower
hd_string[3-diff] 0.0001ms 0.0001ms 8.5% slower
hd_string[3-same] 0.0001ms 0.0001ms 7.5% slower
hd_string[64-diff] 0.0001ms 0.0001ms 0.1% slower
hd_string_random[1024] 0.0001ms 0.0001ms 1.8% slower
hd_string_random[16] 0.0001ms 0.0001ms 4.8% faster
hd_string_random[64] 0.0001ms 0.0001ms 1.0% faster
stdlib_bytes_distance_random[1024] 0.0030ms 0.0029ms 0.3% faster
stdlib_bytes_distance_random[16] 0.0004ms 0.0004ms 1.5% faster
stdlib_bytes_distance_random[64] 0.0005ms 0.0005ms 0.8% faster
stdlib_hex_distance_random[1024] 0.0035ms 0.0035ms 0.3% faster
stdlib_hex_distance_random[16] 0.0004ms 0.0004ms 1.1% slower
stdlib_hex_distance_random[64] 0.0005ms 0.0005ms 0.9% slower

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

mrecachinas and others added 3 commits September 23, 2026 07:54
Waking rayon's workers costs 15-40 us (median) per parallel scan, which
set the parallel thresholds at MiB sizes and made early-match first()
scans slower than serial. The scan pool behind par::map_chunks and
find_first_chunk replaces it:

- Idle workers spin for 50 us, then park, so back-to-back scans start
  in ~3 us.
- The caller works on its own job, and workers claim chunks
  dynamically.
- A job lives on the caller's stack. Workers register before reading
  it, and the caller unpublishes it and waits only for registered
  workers, never for parked ones.
- One job runs at a time: concurrent or nested callers run their
  chunks inline.
- Chunk panics are caught and re-raised in the caller.
- A forked child starts its own pool.
- HEXHAMMING_NUM_THREADS sets the size (RAYON_NUM_THREADS is honored
  for compatibility, 1 disables it). Threads start on the first scan
  that splits.
- Chunks are sized so a scan at the threshold gets 8 of them, up to 4
  per thread. Completion is counted once per participant and the claim
  counter has its own cache line: tiny chunks with per-chunk counters
  cost ~12 us of contention.

rayon is no longer a dependency (criterion still uses it for benches).

With the cheaper dispatch, thresholds drop from 4 MiB to 512 KiB for
block scanners (0.8x at 256 KiB, 1.4-1.7x at 512 KiB, 2.5x at 1 MiB)
and from 1 MiB to 64 KiB for generic widths (2.0-3.5x at 128 KiB). The
Catalog planner's parallel-speedup model is refitted to match.

Interleaved A/B (Apple M4 Max, CPython 3.14, rayon layer -> pool):
  best/0.5MBx16 nomatch          6.66 -> 4.80 us   1.39x
  best/1MBx16 nomatch            13.2 -> 4.08 us   3.23x
  best/2MBx16 nomatch            26.2 -> 6.86 us   3.81x
  best/4MBx16 nomatch            49.3 -> 11.8 us   4.19x
  best/8MBx64 nomatch            76.2 -> 19.6 us   3.88x
  best/16MBx16 nomatch            116 -> 45.1 us   2.56x
  best/64MBx16 nomatch            314 -> 269 us    1.17x (memory bound)
  first/16MBx16 match@25%        72.8 -> 14.4 us   5.04x
  best/0.5MBx20 nomatch (generic) 79.2 -> 13.6 us  5.80x
  many_best/100q x 1024x16       21.8 -> 7.73 us   2.82x
  many_first/1000q x 16384x16     518 -> 398 us    1.30x
  calls and scans under 512 KiB: unchanged (+-3%)

Tests: the pool keeps order, propagates panics and stays usable after
one, serves 8 concurrent callers with nested jobs, and survives 2000
back-to-back jobs that cross the spin/park boundary. It ran 40x clean
in a stress loop and clean under Miri (data-race and aliasing checks,
4 threads). A new CI job runs these tests under ThreadSanitizer and
Miri on Linux. Python tests cover scans in a forked child and
HEXHAMMING_NUM_THREADS=1/3.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
SDE cannot follow the execve of the Python subprocess in
test_num_threads_environment_variable
(PreparePindForFollowExecve fails). The test still runs natively on
every platform, and the fork test runs under emulation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
Profiling the scan pool with empty chunks showed a job cost ~3 us even
with 2 chunks: every job made all 15 workers do contended
read-modify-writes on one shared `users` counter (join and leave) and on
`completed`, whether or not the job had work for them. At the parallel
threshold that was most of the scan: a 512 KiB scan spent ~4.7 us of its
~5 us in dispatch.

* A job only draws in as many workers as it has chunks beyond the
  caller's; the rest see the new generation and sit it out without
  touching shared state.
* Workers flag that they have joined on their own cache line instead of
  bumping a shared counter. The caller clears the job pointer and waits
  for each flag, which keeps the SeqCst join/leave protocol intact.
* The `completed` counter is gone: once the caller's own claim loop
  ends every chunk is claimed, and a worker's chunks are done when its
  flag clears.

Dispatch cost, spinning workers, best of 5 x 20000 calls:

  chunks     empty chunks           1 us chunks
       2     2.96 -> 0.42 us        2.56 -> 1.55 us
       8     2.94 -> 1.46 us        4.12 -> 2.67 us
      16     4.72 -> 2.95 us        5.10 -> 3.79 us
      64     9.43 -> 7.78 us       11.12 -> 8.79 us

Scans, interleaved, best of 3:

  0.5 MiB x 16 B   best 4.13 -> 2.43 us (1.70x), first 1.72x, all 1.50x
  0.5 MiB x 64 B   1.61-1.99x
  1-2 MiB          1.04-1.36x
  4-64 MiB, and generic widths: within noise, checked against a second
  build of the previous pool as a control

Tried and dropped: letting each participant claim the chunk of its own
number through its own cache line. Empty 16-chunk jobs got faster
(2.95 -> 1.60 us), but with real work they got slower (16 x 1 us:
3.79 -> 4.80 us), since a late-joining worker's chunk waits until the
caller takes it over.

Miri (8 seeds) and 20x stress runs are clean; a new test alternates
small and large jobs so workers left out of one must rejoin the next.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas merged commit a2e3667 into main Sep 23, 2026
32 checks passed
@mrecachinas
mrecachinas deleted the perf/scan-pool branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant