Skip to content

perf(scan): dynamic chunking, parallel first, parallel many-query - #66

Merged
mrecachinas merged 6 commits into
feat/catalog-mihfrom
perf/parallel-scans
Sep 23, 2026
Merged

mrecachinas merged 6 commits into
feat/catalog-mihfrom
perf/parallel-scans

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Large array scans used to run as four static rayon jobs, and first was always serial.

A new par module splits scans into chunks of whole records: 256 KiB each, up to 4 per thread, aligned to 16-record blocks. Workers claim chunks dynamically, so P/E-core imbalance evens out, and results come back in chunk order. The API code only sees map_chunks and find_first_chunk, so #67 can swap the executor.

Changes per API

  • first: a serial prefix answers early matches without waking workers. After it, the lowest-numbered matching chunk wins and later chunks are skipped.
  • best: uses the same prefix.
    • An exact match in the prefix returns immediately.
    • Otherwise the prefix's best becomes the limit for the rest.
    • In the rest, an exact match skips every later chunk.
    • The prefix is 1/64 of the scan, clamped to 16–256 KiB, so it costs about 1.5% of serial time on scans without early matches.
  • all: chunk results are concatenated in order.
  • Many-query APIs: whole queries are spread across threads, each scanned serially. With fewer queries than threads over a catalog big enough to split, each query's scan is split instead.
  • Early-exit batches (first_many, best_many): the threshold is checked against full scans, but these scans stop at a match (best at an exact one), so a batch can clear it on paper and still finish almost at once. Such batches start with a serial prefix:
    • Queries are answered one by one, each within what is left of a budget (1/32 of the threshold, in records actually read).
    • Past the budget, the prefix carries on while the batch, extrapolated from the queries so far, stays below the threshold, reading at most the threshold in all.
    • The query that stops it, and the rest, are then planned as before. A cheap batch never wakes the pool; one ordered cheap-first spends at most the threshold serially.
    • The prefix runs out of line: inlined, the loop spilled its state to the stack and ran 4× slower per query on perf(scan): replace rayon with a spin-then-park scan pool #67's build.
  • Catalog: large indexed many-query batches run queries in parallel. The planner's linear-scan model uses the new threshold and a measured speedup curve.

Thresholds come from a crossover sweep with splitting forced on:

  • 16/64-byte records: 0.6× at 2 MiB, 1.2× at 4 MiB, 1.8× at 8 MiB. The block-scanner threshold drops from 16 to 4 MiB.
  • Generic widths: 1.6× at 1 MiB. Their threshold drops from 5 to 1 MiB.

Interleaved A/B (Apple M4 Max, CPython 3.14, #65 → this PR):

Case Before After Speedup
first 16 MB×16, no match 204 µs 79.4 µs 2.57×
first 64 MB×64, no match 875 µs 270 µs 3.24×
first 16 MB×16, match @50% 103 µs 64.2 µs 1.60×
first 16 MB×16, match @25% 51.4 µs 55.0 µs 0.93×
best 8 MB×16 104 µs 62.2 µs 1.67×
best 16 MB×16 112 µs 77.6 µs 1.44×
best 4 MB×20 (generic) 625 µs 175 µs 3.57×
best 100k×128, early exact match 15.8 µs 0.22 µs 71.9×
many_best 100q × 16384×16 331 µs 92.3 µs 3.58×
many_first 1000q × 16384×32 6.56 ms 694 µs 9.45×

Scans under the thresholds, and 100q × 1024×16, are unchanged.

Early-exit batches. Without the serial prefix, first_many over 100 queries × 1024×16-byte records with a permissive threshold went from 0.74 µs to 5.5 µs once #67 lowered the thresholds. Interleaved, best of 5, µs (this PR without → with the prefix):

Case Without With
first_many permissive, 1000×1024×16 56.1 3.63
first_many permissive, 10000×256×16 109 33.9
first_many permissive, 100×1024×20 55.2 0.68
first_many permissive, 1000×1024×20 65.7 5.44
first_many no match, 1000×1024×16 94.7 94.9
best_many no match, 1000×1024×16 113 109
first_many no match, 100×1024×20 84.2 91.5

Heavy batches stay within rayon's run-to-run noise (about ±15%); all_many cannot stop early and has no prefix.

Tests

  • Splitting is forced on small inputs and compared with serial scans for widths 1–64, including counts past the serial prefix, planted exact and near matches on both sides of it, and every threshold.
  • The parallel-over-queries path is forced the same way, with serial-prefix budgets from none, through windows ending before, inside or after a match, to the whole batch, at a threshold that stops the prefix at its budget and one that lets it carry on.
  • par has its own tests: plans cover every record exactly once, and results keep chunk order.

@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 02:49
@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison Results

✅ All benchmarks within noise

104 benchmarks compared, no significant changes detected.

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

mrecachinas and others added 6 commits September 23, 2026 07:54
Large array scans used four static rayon jobs, and first was always
serial. A new par module splits scans into chunks of whole records
(256 KiB each, up to 4 per thread, aligned to 16-record blocks). Workers
claim chunks dynamically, so P/E-core imbalance evens out and results
come back in chunk order. The API code only sees map_chunks and
find_first_chunk.

- first: a serial prefix answers early matches without waking workers.
  After it, the lowest-numbered matching chunk wins and later chunks
  are skipped.
- best: the same prefix. An exact match there returns immediately;
  otherwise the prefix best becomes the limit for the rest, where an
  exact match skips every later chunk. The prefix is 1/64 of the scan,
  clamped to 16-256 KiB, so it costs ~1.5% of serial time on scans
  without early matches.
- all: chunk results are concatenated in order.
- Many-query APIs spread whole queries across threads, each scanned
  serially. With fewer queries than threads over a catalog big enough
  to split, each query's scan is split instead.
- Catalog: indexed many-query batches run queries in parallel when
  their estimated cost is large, and the planner's linear-scan model
  uses the new threshold and a measured speedup curve.

Thresholds come from a crossover sweep with the split forced on:
16/64-byte records are 0.6x at 2 MiB, 1.2x at 4 MiB and 1.8x at 8 MiB,
so the block-scanner threshold drops from 16 to 4 MiB. Generic widths
are 1.6x at 1 MiB, so theirs drops from 5 MiB to 1 MiB.

Interleaved A/B (Apple M4 Max, CPython 3.14, previous layer -> this):
  first/16MBx16 nomatch            204 -> 79.4 us    2.57x
  first/64MBx64 nomatch            875 -> 270 us     3.24x
  first/16MBx16 match@50%          103 -> 64.2 us    1.60x
  first/16MBx16 match@25%         51.4 -> 55.0 us    0.93x
  best/8MBx16 nomatch              104 -> 62.2 us    1.67x
  all/8MBx64 nomatch               107 -> 58.8 us    1.81x
  best/16MBx16 nomatch             112 -> 77.6 us    1.44x
  best/64MBx16 nomatch             339 -> 277 us     1.23x
  best/4MBx20 nomatch (generic)    625 -> 175 us     3.57x
  best/100000x128 exact early     15.8 -> 0.22 us   71.9x
  many_best/100q x 16384x16        331 -> 92.3 us    3.58x
  many_first/1000q x 16384x32     6559 -> 694 us     9.45x
  scans under the thresholds and 100q x 1024x16: unchanged

Tests force splitting on small inputs and compare with serial scans for
widths 1-64, including counts past the serial prefix, exact and near
matches planted on both sides of it, and every threshold. A second test
forces the parallel-over-queries path. par has its own tests: plans
cover every record exactly once, and results keep chunk order.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
Multi-query scans decided whether to spread queries across threads from
their worst case, query_count × catalog bytes. `first` stops at its first
match and `best` at an exact one, so a batch that clears the threshold on
paper can finish almost immediately, and waking the pool then costs far
more than the scans. With a permissive max_dist, first_many over
1000×1024×16-byte records took 56 µs against 2.9 µs serially.

Batches of early-exit scans now start with a serial prefix: queries are
answered one at a time, each scanning at most what is left of a budget
(1/8 of the parallel threshold, in bytes actually read), until the budget
is spent or a query needs more. That query and the rest are then planned
as before. A cheap batch never wakes the pool; an expensive one spends at
most the budget serially, including the window of the query scanned
again. all_many cannot stop early and is unchanged.

Interleaved, best of 5, µs, first_many / best_many (this layer, rayon):

  first permissive 1000×1024×16    56.4 → 3.9
  first permissive 10000×256×16   111.2 → 39.9
  first permissive 100×1024×20     46.4 → 0.69
  first permissive 1000×1024×20    59.8 → 6.0
  fixed-width heavy batches        within noise
  generic-width heavy batches      ~20% slower (90 → 111 for 100×1024×20)

The generic-width cost is the budget: 128 KiB of serial scanning at the
generic kernel's ~7 GB/s, still below rayon's ~40 µs dispatch.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
The prefix loop recomputed its window as `budget - budget % width` for
every query, putting an integer division on the loop-carried chain, and
counted reads in bytes. Counting in records needs one division per batch.
With codegen as it falls on the next layer, the old loop cost ~2 ns more
per query than a plain serial loop: first_many over 1000 queries that
each match their first record took 5.5 us there, against 2.9 us now (one
thread, Rust level).

When the rest of a batch is not planned across threads, catalogs below
the threshold are now scanned with the serial closure directly, skipping
the per-query split checks that could never split them.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
The serial prefix stopped at its budget (1/8 of the parallel threshold,
in records read) even when every query so far had stopped on its first
record, so a cheap batch just past the budget still woke the pool, split
into many tiny chunks. first_many over 1000 queries of 20 bytes, each
matching its first record, took 13 us against 4.7 us serially.

Past the budget, the prefix now carries on while the batch, extrapolated
from the queries answered so far, would stay below the threshold, and
never reads more than the threshold in all. A batch ordered with its
cheap queries first therefore spends at most the threshold serially
before the rest is spread across threads; batches that look expensive
stop at the budget as before. The split test now also runs with a
threshold that lets the prefix carry on.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
Inlined into map_queries_with, the prefix loop shared registers with
everything the parallel phase keeps live, and depending on how the rest
of the crate compiled it spilled most of its state to the stack: with the
next layer's scan pool, first_many over 1000 queries that each match
their first record ran at 7.7 ns per query against 1.8 ns on this layer,
from identical source. As its own function it gets the registers to
itself and compiles the same on both.

Rust level, one thread, permissive first_many, µs:

                    this layer   next layer
  1000x1024x16        1.9 (2.9)    1.8 (7.7)
  10000x256x16       19.3        19.1 (80)

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
With batches that project below the threshold carrying on past the
budget, the budget only has to sample enough queries to judge the batch;
it no longer has to cover cheap batches itself. At 1/8 of the threshold
it cost heavy batches on this layer up to ~50% (rayon's workers go idle
while the prefix runs); at 1/32 they are back within noise of no prefix,
and cheap batches are unchanged.

Interleaved, best of 5, µs (no prefix / 1/8 / 1/32, this layer):

  first no-match 1000x1024x16      94.7 / 111.9 /  94.9
  first no-match 100x1024x20       84.2 / 113.0 /  91.5
  first dup@rand 100x1024x20       72.0 / 104.4 /  84.1
  first permissive 1000x1024x16    56.1 /   3.6 /   3.6
  first permissive 100x1024x20     55.2 /  0.65 /  0.68

On the next layer's pool 1/32 is as good or better throughout.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas merged commit f81ca88 into main Sep 23, 2026
31 checks passed
@mrecachinas
mrecachinas deleted the perf/parallel-scans branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant