Skip to content

perf(python): raw METH_FASTCALL bindings for the hot single-call APIs - #64

Merged
mrecachinas merged 2 commits into
perf/x86-avx512from
perf/raw-fastcall
Sep 23, 2026
Merged

mrecachinas merged 2 commits into
perf/x86-avx512from
perf/raw-fastcall

Conversation

@mrecachinas

@mrecachinas mrecachinas commented Sep 23, 2026 •

Copy link
Copy Markdown
Owner

PyO3's generic argument extraction and trampoline are most of the cost of a small call. In a samply profile of hamming_distance_string (16 chars):

Frame Self time
<&str as FromPyObject>::extract 7.4%
PyUnicode_AsUTF8AndSize 5.5%
extract_arguments_fastcall 5.1%
trampoline frames 3%+

This PR re-registers the nine hot functions as hand-written METH_FASTCALL | METH_KEYWORDS functions in the PyO3-created module: the string, bytes and within-distance calls, the array scans, and hamming_distances_bytes. PyO3 still owns module creation and the rest of the API.

What the raw functions preserve

  • Keyword arguments, __self__, __module__ (pickling), docstrings and text signatures.
  • PyO3's exception types and messages, and its buffer acceptance rules.
  • Each API's GIL policy:
    • Single-call and array scans detach at their size thresholds for any buffer.
    • Pairwise stays attached for non-bytes buffers.
    • check_hexstrings_within_dist now also detaches at 4096+ chars, like hamming_distance_string.
  • Panic containment, with detached sections re-attaching during unwinding.

Compact-ASCII fast path. str arguments that are compact ASCII are read straight from the PyASCIIObject payload, saving a further 3.5–4 ns per call. This only applies to GIL-enabled CPython. At import, the peek is verified against PyUnicode_AsUTF8AndSize using ASCII and non-ASCII samples, and it disables itself on any mismatch.

Interleaved A/B (Apple M4 Max, CPython 3.14):

Call Before Raw, no peek Raw Speedup
hamming_distance_string 16 26.3 ns 20.6 ns 17.0 ns 1.55×
hamming_distance_string 64 27.4 ns 22.2 ns 18.4 ns 1.49×
hamming_distance_bytes 16 24.5 ns 17.0 ns 17.1 ns 1.43×
hamming_distance_bytes bytearray 64 43.1 ns 30.7 ns 30.8 ns 1.40×
check_hexstrings_within_dist 16 29.0 ns 24.2 ns 20.9 ns 1.38×
check_bytes_within_dist 16 25.8 ns 18.8 ns 19.0 ns 1.35×

Tests

  • Metadata, keyword and argument-error parity, and buffer/unicode parity.
  • A pickle round-trip for every module function.
  • A 1197-case differential harness against the PyO3 bindings passes. It covers positional/keyword calls, duplicate and unknown kwargs, wrong types, str/bytes subclasses, bytearray/memoryview/numpy/array buffers, non-contiguous buffers, invalid hex, lone surrogates, and huge and __index__ thresholds.
  • Type-checks against CPython 3.10, 3.12 and 3.14.

Other interpreters: the raw bindings rely on CPython's tuple macros and object layouts, which PyO3 doesn't expose on PyPy. They are now built only for CPython. PyPy and GraalPy keep the PyO3 bindings, which stay registered underneath. #68's PGO wheel build found this, since it compiles for every interpreter in the manylinux image, and #57's PyPy CI leg now covers it. Tests pass on PyPy 3.11.

Review focus: src/raw_python.rs is unsafe CPython C-API code. The ABI assumptions behind the ASCII peek are documented at str_ascii_fast.

@mrecachinas
mrecachinas added this pull request to stack #60 September 23, 2026 02:09
@github-actions

Copy link
Copy Markdown

📊 Benchmark Comparison Results

✅ All benchmarks within noise

104 benchmarks compared, no significant changes detected.

Legend: ✅ Faster (>5%) · ⚠️ Slower (5-30%) · ❌ Regression (>30%) · 🆕 New · 🗑️ Removed

mrecachinas and others added 2 commits September 23, 2026 07:53
PyO3's generic argument extraction and trampoline are most of the
cost of a small call. In a samply profile of
hamming_distance_string(16 chars), <&str as FromPyObject>::extract is
7.4% self time, PyUnicode_AsUTF8AndSize 5.5%, extract_arguments_fastcall
5.1% and the trampoline frames 3%+.

Nine functions (the string, bytes and within-distance calls, the array
scans and hamming_distances_bytes) are re-registered as hand-written
METH_FASTCALL | METH_KEYWORDS functions in the PyO3-created module.
PyO3 still owns module creation and the rest of the API. The raw
functions keep:
- keyword arguments, __self__, __module__, docstrings and text
  signatures;
- PyO3's exception types and messages, and its buffer acceptance
  rules;
- each API's GIL policy: the single-call and array scans detach at
  their size thresholds for any buffer, and pairwise stays attached for
  non-bytes buffers. check_hexstrings_within_dist now also detaches at
  4096+ chars, like hamming_distance_string;
- panic containment, with detached sections re-attaching during
  unwinding.

Compact-ASCII str arguments are read straight from the PyASCIIObject
payload, which is worth another 3.5-4 ns per call. This is limited to
GIL-enabled CPython (the ABI details are in raw_python.rs) and checked
at import against PyUnicode_AsUTF8AndSize for ASCII and non-ASCII
samples. It disables itself on any mismatch.

build.rs now emits PyO3's cfgs (Py_GIL_DISABLED, Py_LIMITED_API, PyPy,
...), which also removes the unexpected-cfg warnings.

Interleaved A/B (Apple M4 Max, CPython 3.14, previous layer -> raw
without ASCII peek -> raw):
  hamming_distance_string/16       26.3 -> 20.6 -> 17.0 ns  1.55x
  hamming_distance_string/64       27.4 -> 22.2 -> 18.4 ns  1.49x
  hamming_distance_bytes/16        24.5 -> 17.0 -> 17.1 ns  1.43x
  hamming_distance_bytes/bytearray 43.1 -> 30.7 -> 30.8 ns  1.40x
  check_hexstrings_within_dist/16  29.0 -> 24.2 -> 20.9 ns  1.38x
  check_bytes_within_dist/16       25.8 -> 18.8 -> 19.0 ns  1.35x
  check_hexstrings_within_dist/1000,d=200  30.6 -> 23.8 ns  1.29x

Tests: metadata, keyword and argument-error parity, buffer/unicode
parity, and a new pickle round-trip for every module function. A
1197-case differential harness against the PyO3 bindings passes
(positional/keyword calls, duplicate and unknown kwargs, wrong types,
str/bytes subclasses, bytearray/memoryview/numpy/array buffers,
non-contiguous buffers, invalid hex, lone surrogates, huge and
__index__ thresholds). Builds type-check against CPython 3.10, 3.12
and 3.14.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
The raw FASTCALL bindings use CPython's tuple macros and object layouts,
which PyO3 does not expose on PyPy, so the PyPy wheel no longer
compiled (found by the PGO wheel build). They are now built only for
CPython; elsewhere the PyO3 bindings they replace stay registered.

Tests pass on PyPy 3.11 (141 passed, 5 skipped) and CPython 3.14.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: c7edd725-5714-4044-acf4-c9afc6ec8cf8
@mrecachinas
mrecachinas merged commit ed5f5e9 into main Sep 23, 2026
31 checks passed
@mrecachinas
mrecachinas deleted the perf/raw-fastcall branch September 23, 2026 13:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant