Repository navigation
perf(scan): dynamic chunking, parallel first, parallel many-query - #66
Merged
Merged
Conversation
📊 Benchmark Comparison Results✅ All benchmarks within noise104 benchmarks compared, no significant changes detected. Legend: ✅ Faster (>5%) · |
Large array scans used four static rayon jobs, and first was always serial. A new par module splits scans into chunks of whole records (256 KiB each, up to 4 per thread, aligned to 16-record blocks). Workers claim chunks dynamically, so P/E-core imbalance evens out and results come back in chunk order. The API code only sees map_chunks and find_first_chunk. - first: a serial prefix answers early matches without waking workers. After it, the lowest-numbered matching chunk wins and later chunks are skipped. - best: the same prefix. An exact match there returns immediately; otherwise the prefix best becomes the limit for the rest, where an exact match skips every later chunk. The prefix is 1/64 of the scan, clamped to 16-256 KiB, so it costs ~1.5% of serial time on scans without early matches. - all: chunk results are concatenated in order. - Many-query APIs spread whole queries across threads, each scanned serially. With fewer queries than threads over a catalog big enough to split, each query's scan is split instead. - Catalog: indexed many-query batches run queries in parallel when their estimated cost is large, and the planner's linear-scan model uses the new threshold and a measured speedup curve. Thresholds come from a crossover sweep with the split forced on: 16/64-byte records are 0.6x at 2 MiB, 1.2x at 4 MiB and 1.8x at 8 MiB, so the block-scanner threshold drops from 16 to 4 MiB. Generic widths are 1.6x at 1 MiB, so theirs drops from 5 MiB to 1 MiB. Interleaved A/B (Apple M4 Max, CPython 3.14, previous layer -> this): first/16MBx16 nomatch 204 -> 79.4 us 2.57x first/64MBx64 nomatch 875 -> 270 us 3.24x first/16MBx16 match@50% 103 -> 64.2 us 1.60x first/16MBx16 match@25% 51.4 -> 55.0 us 0.93x best/8MBx16 nomatch 104 -> 62.2 us 1.67x all/8MBx64 nomatch 107 -> 58.8 us 1.81x best/16MBx16 nomatch 112 -> 77.6 us 1.44x best/64MBx16 nomatch 339 -> 277 us 1.23x best/4MBx20 nomatch (generic) 625 -> 175 us 3.57x best/100000x128 exact early 15.8 -> 0.22 us 71.9x many_best/100q x 16384x16 331 -> 92.3 us 3.58x many_first/1000q x 16384x32 6559 -> 694 us 9.45x scans under the thresholds and 100q x 1024x16: unchanged Tests force splitting on small inputs and compare with serial scans for widths 1-64, including counts past the serial prefix, exact and near matches planted on both sides of it, and every threshold. A second test forces the parallel-over-queries path. par has its own tests: plans cover every record exactly once, and results keep chunk order. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
Multi-query scans decided whether to spread queries across threads from their worst case, query_count × catalog bytes. `first` stops at its first match and `best` at an exact one, so a batch that clears the threshold on paper can finish almost immediately, and waking the pool then costs far more than the scans. With a permissive max_dist, first_many over 1000×1024×16-byte records took 56 µs against 2.9 µs serially. Batches of early-exit scans now start with a serial prefix: queries are answered one at a time, each scanning at most what is left of a budget (1/8 of the parallel threshold, in bytes actually read), until the budget is spent or a query needs more. That query and the rest are then planned as before. A cheap batch never wakes the pool; an expensive one spends at most the budget serially, including the window of the query scanned again. all_many cannot stop early and is unchanged. Interleaved, best of 5, µs, first_many / best_many (this layer, rayon): first permissive 1000×1024×16 56.4 → 3.9 first permissive 10000×256×16 111.2 → 39.9 first permissive 100×1024×20 46.4 → 0.69 first permissive 1000×1024×20 59.8 → 6.0 fixed-width heavy batches within noise generic-width heavy batches ~20% slower (90 → 111 for 100×1024×20) The generic-width cost is the budget: 128 KiB of serial scanning at the generic kernel's ~7 GB/s, still below rayon's ~40 µs dispatch. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
The prefix loop recomputed its window as `budget - budget % width` for every query, putting an integer division on the loop-carried chain, and counted reads in bytes. Counting in records needs one division per batch. With codegen as it falls on the next layer, the old loop cost ~2 ns more per query than a plain serial loop: first_many over 1000 queries that each match their first record took 5.5 us there, against 2.9 us now (one thread, Rust level). When the rest of a batch is not planned across threads, catalogs below the threshold are now scanned with the serial closure directly, skipping the per-query split checks that could never split them. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
The serial prefix stopped at its budget (1/8 of the parallel threshold, in records read) even when every query so far had stopped on its first record, so a cheap batch just past the budget still woke the pool, split into many tiny chunks. first_many over 1000 queries of 20 bytes, each matching its first record, took 13 us against 4.7 us serially. Past the budget, the prefix now carries on while the batch, extrapolated from the queries answered so far, would stay below the threshold, and never reads more than the threshold in all. A batch ordered with its cheap queries first therefore spends at most the threshold serially before the rest is spread across threads; batches that look expensive stop at the budget as before. The split test now also runs with a threshold that lets the prefix carry on. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
Inlined into map_queries_with, the prefix loop shared registers with
everything the parallel phase keeps live, and depending on how the rest
of the crate compiled it spilled most of its state to the stack: with the
next layer's scan pool, first_many over 1000 queries that each match
their first record ran at 7.7 ns per query against 1.8 ns on this layer,
from identical source. As its own function it gets the registers to
itself and compiles the same on both.
Rust level, one thread, permissive first_many, µs:
this layer next layer
1000x1024x16 1.9 (2.9) 1.8 (7.7)
10000x256x16 19.3 19.1 (80)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
With batches that project below the threshold carrying on past the budget, the budget only has to sample enough queries to judge the batch; it no longer has to cover cheap batches itself. At 1/8 of the threshold it cost heavy batches on this layer up to ~50% (rayon's workers go idle while the prefix runs); at 1/32 they are back within noise of no prefix, and cheap batches are unchanged. Interleaved, best of 5, µs (no prefix / 1/8 / 1/32, this layer): first no-match 1000x1024x16 94.7 / 111.9 / 94.9 first no-match 100x1024x20 84.2 / 113.0 / 91.5 first dup@rand 100x1024x20 72.0 / 104.4 / 84.1 first permissive 1000x1024x16 56.1 / 3.6 / 3.6 first permissive 100x1024x20 55.2 / 0.65 / 0.68 On the next layer's pool 1/32 is as good or better throughout. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
mrecachinas
force-pushed
the
perf/parallel-scans
branch
from
September 23, 2026 11:59
8ea7780 to
23352ab
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large array scans used to run as four static rayon jobs, and
firstwas always serial.A new
parmodule splits scans into chunks of whole records: 256 KiB each, up to 4 per thread, aligned to 16-record blocks. Workers claim chunks dynamically, so P/E-core imbalance evens out, and results come back in chunk order. The API code only seesmap_chunksandfind_first_chunk, so #67 can swap the executor.Changes per API
first: a serial prefix answers early matches without waking workers. After it, the lowest-numbered matching chunk wins and later chunks are skipped.best: uses the same prefix.all: chunk results are concatenated in order.first_many,best_many): the threshold is checked against full scans, but these scans stop at a match (bestat an exact one), so a batch can clear it on paper and still finish almost at once. Such batches start with a serial prefix:Thresholds come from a crossover sweep with splitting forced on:
Interleaved A/B (Apple M4 Max, CPython 3.14, #65 → this PR):
first16 MB×16, no matchfirst64 MB×64, no matchfirst16 MB×16, match @50%first16 MB×16, match @25%best8 MB×16best16 MB×16best4 MB×20 (generic)best100k×128, early exact matchmany_best100q × 16384×16many_first1000q × 16384×32Scans under the thresholds, and 100q × 1024×16, are unchanged.
Early-exit batches. Without the serial prefix,
first_manyover 100 queries × 1024×16-byte records with a permissive threshold went from 0.74 µs to 5.5 µs once #67 lowered the thresholds. Interleaved, best of 5, µs (this PR without → with the prefix):first_manypermissive, 1000×1024×16first_manypermissive, 10000×256×16first_manypermissive, 100×1024×20first_manypermissive, 1000×1024×20first_manyno match, 1000×1024×16best_manyno match, 1000×1024×16first_manyno match, 100×1024×20Heavy batches stay within rayon's run-to-run noise (about ±15%);
all_manycannot stop early and has no prefix.Tests
parhas its own tests: plans cover every record exactly once, and results keep chunk order.