Repository navigation
Pass batch operations to the binding as a structure of arrays - #129
Draft
danielAsaboro wants to merge 2 commits into
Draft
danielAsaboro wants to merge 2 commits into
danielAsaboro wants to merge 2 commits into
Conversation
Each operation used to be a JS object that the native side read with four named property lookups plus a typed array lookup per buffer. Operations now live in a flat Uint32Array (type and column family packed into word 0, then buffer index, offset and length per slice) next to a list of distinct backing ArrayBuffers, so the binding resolves each backing buffer once per batch and does only pointer arithmetic per operation. try* writes no longer push a null promise per op.
A batch op whose buffer was detached between put/get and flush made the binding throw INVALID_BATCH from inside _flush, which nobody awaited: the rejection went unhandled and flush() and the op promises never settled. Reject every op and the flush with the error instead. On main the same op was silently written as an empty value.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Each batch operation was a JS object the binding read with four named-property lookups (
typewas a prototype getter, so a call back into JS) plus a typed array lookup per buffer. Operations now live in one Uint32Array per batch: word 0 packs the type into the upper 8 bits and the column family index into the lower 24, then (buffer index, offset, length) per slice, next to a deduplicated list of backing ArrayBuffers. The binding resolves each backing buffer once per batch and bounds-checks every slice.The bounds check also fixes a silent corruption: on main, a value whose ArrayBuffer is detached between
put()andflush()is written as an empty value. It now rejects the op and the flush withINVALID_BATCH(second commit, with 2 tests that fail on main).Numbers, thread CPU, two builds of the same toolchain alternating in one process, 21 rounds:
append()of 16 blocks: 194,540 → 161,737 ns (0.83, 18/21)Method: two builds from the same toolchain (main and this branch) were loaded into one process and run alternately (AB, then BA) for 21 rounds after a warm-up round. The numbers are medians of
process.threadCpuUsage. The hyperdrive row is 11 rounds throughdrive.batch(). Run with Node 24.15 on an Apple M3 Pro (darwin-arm64), which is the only platform I built.The full suite passes on Node and Bare (91/91). hypercore-storage (79/79) passes on this branch. hypercore (457/457) passes on the first commit; the second commit only changes the JS error path.
This changes
binding.cc, so the JS and the binary have to ship together with fresh prebuilds for every platform.