Skip to content

perf(neon): block scanners for fixed-width arrays and pairwise kernels - #59

Merged
mrecachinas merged 2 commits into
perf/pyo3-pass-modulefrom
perf/neon-block-scanners
Sep 23, 2026
Merged

mrecachinas merged 2 commits into
perf/pyo3-pass-modulefrom
perf/neon-block-scanners

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

Replaces the per-record NEON fixed-width scanners with block scanners. Each block of records gets its distances computed at once and reduced with an ADDP tree. One compare against a single limit then produces a one-bit-per-record hit mask.

Width Records/block Reduction
8 B 16 3-level ADDP
16 B 16 15 ADDP, nibble mask
32 B 8 ADDP + vpaddlq_u8 (u16 totals)
64 B 4 ADDP + vpaddlq_u8/u16 (u32 totals)

Widths 8 and 64 previously had no specialization: a generic loop made an indirect kernel call per record. Semantics are unchanged:

  • first returns the lowest index.
  • best returns the minimum distance, breaking ties by lowest index, and returns immediately on an exact match.
  • all returns hits in ascending index order.

Pairwise distances get const-generic kernels for widths 8/16/20/32/64.

Interleaved A/B in one process (Apple M4 Max, CPython 3.14):

Case Before After Speedup
best 1024×8 B, no match 1863 ns 138 ns 13.5×
all 1024×16 B 367 ns 231 ns 1.59×
best 1024×32 B 552 ns 460 ns 1.20×
best 16384×64 B 24.4 µs 14.0 µs 1.74×
pairwise_into 10000×8 B 17.8 µs 3.4 µs 5.2×
pairwise_into 10000×32 B 22.5 µs 5.6 µs 4.0×
many-query all, 1000 q × 16384×16 B 5.32 ms 3.29 ms 1.62×

Dense matches. A follow-up commit writes each matching block's results into reserved capacity instead of pushing them one by one, and copies a block whose records all match in one straight loop. Before it, dense all results were 6–9% slower than on main: Python _into, 1024×16 with every record matching, went 1291 → 1364 ns. It now takes 957 ns. At the Rust level, 1024 records: all matching is 1.16–1.66× faster, half matching 1.0–1.26×, and no match is unchanged.

Tests

  • Rust oracle tests for every width, covering counts across block boundaries, limits, duplicates and complements.
  • The Python randomized oracle test now covers widths 8/16/32/64 and limits at the width boundary.
  • Criterion benches now cover widths 8/64, plus a new pairwise_into group.
  • A test checks each scanner's full-block mask against what mask() reports when every lane matches.

@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 01:29
@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison Results

✅ All benchmarks within noise

104 benchmarks compared, no significant changes detected.

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

@mrecachinas
mrecachinas force-pushed the perf/neon-block-scanners branch from 4373817 to 695a87a Compare September 23, 2026 01:43
@mrecachinas
mrecachinas force-pushed the perf/neon-block-scanners branch from 695a87a to 2f5b6d7 Compare September 23, 2026 01:54
@mrecachinas
mrecachinas force-pushed the perf/neon-block-scanners branch from 2f5b6d7 to 9843637 Compare September 23, 2026 02:08
mrecachinas and others added 2 commits September 23, 2026 07:53
Replace the per-record NEON fixed-width scanners with block scanners that
compute distances for a whole block of records at once, reduce them with
an ADDP tree, compare against a single limit and iterate hits from a
one-bit-per-lane mask:

  width  records/block  reduction
  8 B    16             3-level ADDP, byte totals
  16 B   16             15 ADDP, nibble mask
  32 B   8              ADDP tree + vpaddlq_u8, u16 totals
  64 B   4              ADDP tree + vpaddlq_u8/u16, u32 totals

Widths 8 and 64 were previously not specialized at all (generic loop with
an indirect kernel call per record). `best` keeps its lowest-index
tie-break and exact-match early exit; `all` stays in ascending order.

Pairwise distances get const-generic kernels for widths 8/16/20/32/64,
used by bytes_pairwise_distances and bytes_pairwise_distances_into when
the algorithm is native or neon.

Interleaved A/B in one process (Apple M4 Max, CPython 3.14, ns/call,
previous layer -> this layer):

  first/1024x8/nomatch      1593.0 ->  122.9   12.96x
  best/1024x8/nomatch       1863.3 ->  138.3   13.48x
  all/1024x16/nomatch        366.6 ->  231.0    1.59x
  best/1024x16/nomatch       306.8 ->  237.9    1.29x
  best/1024x32/nomatch       552.0 ->  459.7    1.20x
  best/1024x64/nomatch      1583.5 ->  935.9    1.69x
  best/16384x64/nomatch     24.38us -> 13.98us  1.74x
  pairwise_into/10000x8     17.83us ->  3.40us  5.24x
  pairwise_into/10000x20    27.99us ->  5.41us  5.17x
  pairwise_into/10000x32    22.54us ->  5.60us  4.02x
  pairwise_list/10000x20    50.36us -> 33.10us  1.52x
  many_all/1000q x 16384x16  5.32ms ->  3.29ms  1.62x

Tests: Rust oracle tests for every width across block-boundary counts,
limits, duplicates and complements; the Python randomized oracle test now
covers widths 8/16/32/64 and limits at the width boundary. Criterion
benches cover widths 8/64 and add a pairwise_into group.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
Found while re-measuring the README: dense all_within_dist results
(packed/into) were 6-9% slower than before the block scanners. Every
match went through Vec::push, and a block whose records all matched was
walked lane by lane. Each block with matches now reserves its records
once and writes into spare capacity, and a fully matching block is
copied out in one straight loop.

Rust level, 1024 records, best of 7:

  width   all match           half match          no match
  8 B     1111 ->  720 ns     804 -> 679 ns       unchanged
  16 B    1200 ->  725 ns     898 -> 713 ns       unchanged
  32 B    1423 ->  900 ns    1008 -> 845 ns       unchanged
  64 B    1539 -> 1329 ns    1289 -> 1297 ns      unchanged

Python, 1024x16 all matching, vs main: packed 1944 -> 1629 ns,
into 1291 -> 957 ns (the previous commit: 2043 and 1364 ns).

A test checks that each scanner's full-block mask is exactly what mask()
reports when every lane matches.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas force-pushed the perf/neon-block-scanners branch from 9843637 to e4deeb8 Compare September 23, 2026 11:59
@mrecachinas
mrecachinas merged commit e41efae into main Sep 23, 2026
31 checks passed
@mrecachinas
mrecachinas deleted the perf/neon-block-scanners branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant