Repository navigation
Conversation
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
Third of the nz map series, and the first to decode values with every warp of the block. Still opt-in behind `LIBCUDF_PARQUET_NZ_MAP=1`. `decode_next_passes_wide` (delta_binary.cuh) decodes one DELTA pass per warp and returns each lane's value in a register. A single call stops at the end of the current DELTA block, which is what makes the passes independent: within a block the header is already parsed into `cur_min_delta` and `cur_bitwidths`, so every pass is addressable with O(num_warps_t) pointer arithmetic, whereas finding the next block's first byte needs a serial varint parse of its header. `active` is how many warps have a pass to decode: `num_warps_t`, or fewer when the block has fewer left. The spec requires a block to hold a multiple of 128 values, so four passes, and `active` is then always `num_warps_t` -- which holds only while `num_warps_t` divides those four, hence the static_assert. `init_binary_block` is weaker than the spec and accepts any block whose mini-blocks hold a multiple of 32 values, so a narrower block runs the same loop with fewer warps, and `active` is then also what keeps `mini_block_idx` inside `cur_bitwidths`. What this buys over the single-warp path is the serial dependency between a block's passes: there, each pass ends by writing `last_value` from its last lane, so pass i+1 cannot start its scan until pass i finishes, and one warp of four does the unpack. Here all four unpack concurrently and combine through a per-warp total, which also stops routing values through the decoder's rolling buffer and cuts the value loop from two block barriers per 32 values to two per 128. `warp_pass_total` is a parameter sized by `num_warps_t` rather than a member, so each caller sizes it from its own `decode_block_size` and a kernel running two decoders can give each its own array. The two prefix sums over it are taken before the trailing barrier and unrolled over `num_warps_t` with predicated adds: the caller's loop has no barrier between iterations, so a warp re-entering the function overwrites its slot and a read placed after the barrier would race it; and a loop bounded by the runtime `warp_idx` and `active` compiles to a variable-length ladder over what is a four-element array. Lanes of an active warp past the stream's `value_count` still carry an ordinary index and a value built from the block's min delta alone, and the decoder's cursor advances a mini-block at a time, so neither notices when a stream holds fewer values than a caller wants. Both nz map consumers therefore compare the map's `nz_count` against `value_count` before writing anything and report `DATA_STREAM_OVERRUN` if the stream is short. Without that, a page whose definition levels ask for more values than its delta header declares read without error: the DLBA consumer wrote invented lengths whose offsets ran past the character data, and the DELTA_BINARY_PACKED consumer from NVIDIA#24501 copied rolling-buffer slots the page never wrote. The count includes any valid values before `first_row`, which is the prefix of the stream a read needs, so a bounded read that uses fewer values than the page declares is unaffected. The DLBA consumer keeps a post-loop check as a backstop, and returns before the offsets scan rather than running it over unwritten lengths. `init_binary_block` now rejects a block size that does not fit in 32 bits. `block_size` is a uleb128 and so is 64 bits wide, while `values_per_mb` and the pass counts derived from it are 32: a crafted header can divide evenly in 64 bits and narrow to zero, leaving `passes_per_mini_block == 0` and a loop thread 0 cannot leave. The single-warp path never derived a pass count from `block_size`, so this is reachable only through the new loop. A mini-block's byte length is now computed in 64 bits. The bitwidth comes from the page unvalidated, so up to 255, and `values_per_mb` is bounded only by what fits in 32, so `bitwidth * values_per_mb / 8` could wrap and place the mini-block's start below the block, where the `block_end` checks still pass and the unpack reads the wrong bytes. The wide loop walks further ahead than the single-warp path, which is what made this worth closing now; the same expression on the single-warp path is widened too rather than left as its twin. `classify_nz_map_kind` now accepts a second kernel mask, so flat DELTA_LENGTH_BYTE_ARRAY pages get a map. Routing matches the DELTA_BINARY pair: the consumer runs on the producer's stream, so it is ordered behind it without an event, and the level-decoding kernel's skip is gated on the same flag that decides whether the consumer is launched. The host now records which encodings still have a page without a map, so each level-decoding kernel is launched only when its encoding has work for it rather than on one bool scoped to DELTA_BINARY. `decode_delta_length_byte_array_kernel` keeps its name and its decode path; it gains that flag and the early return. Tests: `NzMapMatchesLevelDecoder` gains a nullable DELTA_LENGTH_BYTE_ARRAY column, without which its consumer is never selected. `NzMapNarrowDeltaBlock` covers blocks of 32 and 64 values, where `active` clamps below `num_warps_t`, and a 256-value block where it does not. `NzMapRejectsShortDeltaStream` builds pages that ask for more values than their stream declares, for both consumers, plus controls that read a stream holding exactly enough and a bounded read holding more than it needs. The two 512 MiB delta benchmarks gain a `validity` axis. The existing nullable sweep is 8 MiB, which is L2-resident on an H100, so the nz map's extra traffic of 4 bytes per value could not show up there -- and for a string column that cost is carried against a much larger payload than the fixed-width case. The nested path builds its own data profile and ignores `validity`, so those cells are skipped rather than run three times over on identical data. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
vyasr
force-pushed
the
nz-map/03-delta-length-byte-array
branch
from
October 16, 2026 19:20
e0dc837 to
662ab36
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR adds the DELTA_LENGTH_BYTE_ARRAY consumer for the nz map, still behind
LIBCUDF_PARQUET_NZ_MAPand still off by default. It follows #24501, which added the producer and the DELTA_BINARY_PACKED consumer;classify_nz_map_kindwidens by one encoding and the host routes DLBA pages the same way it already routes delta binary ones.The new kernel is
decode_delta_length_byte_array_kernel_with_nz_map. Like the delta binary consumer it reads the map instead of decoding levels, so it has no level-decoding warp and puts the whole block on values. The existingdecode_delta_length_byte_array_kernelis unchanged apart from skipping the pages the consumer takes, and we skip launching it entirely when every DLBA page in the subpass has a map.The other half of this PR is a block-wide delta decode.
decode_next_passes_widedecodes one DELTA pass per warp into registers instead of running a single warp through the rolling buffer one pass at a time, with a cross-warp running total to carry the prefix sum between passes. A DELTA block narrower than the thread block, which the spec forbids butinit_binary_blockaccepts, goes through the same loop with fewer warps active. This is what makes the consumer worth having for strings: without it the decode is still serialized through one warp no matter how many threads write values.Both nz map consumers now reject a page whose map asks for more values than its delta header declares, before writing any output. Without that check neither noticed a short stream: the loop produces a value for every lane of a pass, and the decoder's cursor moves a mini-block at a time, so both run past the declared count. For DLBA that wrote invented lengths whose offsets ran past the character data. The DELTA_BINARY_PACKED consumer from #24501 had the same gap, copying buffer slots the page never wrote, so it is fixed here too.
A
validityaxis is added to the 512 MiB delta benchmarks so the nullable cases are measurable there; it is skipped for the nested table builder, which does not take a null probability.Benchmark numbers to follow in a comment; the machine this series is measured on is mid-rebuild.
Contributes to #23965.
Checklist