Repository navigation
Conversation
Signed-off-by: Vyas Ramasubramani <vyasr@nvidia.com>
We've seen occasional errors in CI (https://github.com/NVIDIA/cudf/actions/runs/38751937396/job/116494190204?pr=24626#step:13:5112) that likely come from dask/distributed#9374. This change works around that issue by bumping the periodic check, making the race less likely.
…rkaround Workaround flaky test in cudf-polars
Fix oldest numba-cuda CI pin for current UCXX
…lta-binary Add the flat nz map producer and its DELTA_BINARY consumer
Third of the nz map series, and the first to decode values with every warp of the block. Still opt-in behind `LIBCUDF_PARQUET_NZ_MAP=1`. `decode_next_passes_wide` (delta_binary.cuh) decodes one DELTA pass per warp and returns each lane's value in a register. A single call stops at the end of the current DELTA block, which is what makes the passes independent: within a block the header is already parsed into `cur_min_delta` and `cur_bitwidths`, so every pass is addressable with O(num_warps_t) pointer arithmetic, whereas finding the next block's first byte needs a serial varint parse of its header. `active` is how many warps have a pass to decode: `num_warps_t`, or fewer when the block has fewer left. The spec requires a block to hold a multiple of 128 values, so four passes, and `active` is then always `num_warps_t` -- which holds only while `num_warps_t` divides those four, hence the static_assert. `init_binary_block` is weaker than the spec and accepts any block whose mini-blocks hold a multiple of 32 values, so a narrower block runs fewer warps, and `active` is then also what keeps `mb` inside `cur_bitwidths`. What this buys over the single-warp path it replaces is the serial dependency between a block's passes: there, each pass ends by writing `last_value` from its last lane, so pass i+1 cannot start its scan until pass i finishes, and one warp of four does the unpack. Here all four unpack concurrently and combine through a per-warp total, which also stops routing values through the decoder's rolling buffer and cuts the value loop from two block barriers per 32 values to two per 128. `warp_pass_total` is a parameter sized by `num_warps_t` rather than a member, so each caller sizes it from its own `decode_block_size` and a kernel running two decoders can give each its own array. The two prefix sums over it are taken before the trailing barrier and unrolled over `num_warps_t` with predicated adds: the caller's loop has no barrier between iterations, so a warp re-entering the function overwrites its slot and a read placed after the barrier would race it; and a loop bounded by the runtime `w` and `active` compiles to a variable-length ladder over what is a four-element array. `init_binary_block` now rejects a block size that does not fit in 32 bits. `block_size` is a uleb128 and so is 64 bits wide, while `values_per_mb` and the pass counts derived from it are 32: a crafted header can divide evenly in 64 bits and narrow to zero, leaving `ppm == 0` and a loop thread 0 cannot leave. The single-warp path never derived a pass count from `block_size`, so this is reachable only through the new loop. A mini-block's byte length is now computed in 64 bits. The bitwidth comes from the page unvalidated, so up to 255, and `values_per_mb` is bounded only by what fits in 32, so `bitwidth * values_per_mb / 8` could wrap and place the mini-block's start below the block, where the `block_end` checks still pass and the unpack reads the wrong bytes. The wide loop walks further ahead than the single-warp path, which is what made this worth closing now; the same expression on the single-warp path is widened too rather than left as its twin. `classify_nz_map_kind` now accepts a second kernel mask, so flat DELTA_LENGTH_BYTE_ARRAY pages get a map. Routing matches the DELTA_BINARY pair: both kernels take the caller's page mask unmodified and route on the page's own `has_nz_map(DELTA_FLAT)`, with the level-decoding kernel's skip gated on the same flag that decides whether the consumer is launched, so the two decisions are one bit. `decode_delta_length_byte_array_kernel` keeps its name and its decode path; it gains that flag and the early return. The consumer reports `DATA_STREAM_OVERRUN` when the loop stops short of `nz_count`. A page whose delta stream holds fewer values than its definition levels call for leaves the tail lengths unwritten, and they are not zero-filled either, because the validity map marks those rows valid -- so the offsets scan that follows would run over uninitialized memory and produce offsets that index outside the character data. Tests: `NzMapMatchesLevelDecoder` gains a nullable DELTA_LENGTH_BYTE_ARRAY column, without which its consumer is never selected. `NzMapNarrowDeltaBlock` covers blocks of 32 and 64 values, where `active` clamps below `num_warps_t`, and a 256-value block where it does not. The two 512 MiB delta benchmarks gain a `validity` axis. The existing nullable sweep is 8 MiB, which is L2-resident on an H100, so the nz map's extra traffic of 4 bytes per value could not show up there -- and for a string column that cost is carried against a much larger payload than the fixed-width case. The nested path builds its own data profile and ignores `validity`, so those cells are skipped rather than run three times over on identical data. Validation: PARQUET_TEST 564 passed / 1 pre-existing skip and HYBRID_SCAN_TEST 106/106, both with the nz map off and on. compute-sanitizer racecheck and memcheck over the delta and nz map tests with the feature on: 0 hazards, 0 errors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
vyasr
force-pushed
the
nz-map/03-delta-length-byte-array
branch
from
October 15, 2026 21:56
6d53bf0 to
51f043b
Compare
Owner
Author
|
Superseded by NVIDIA#24639, now that PR 2 (NVIDIA#24501) has merged and this branch is rebased onto main. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Self-review PR, not for merge. Targets
nz-map/02-flat-producer-delta-binaryso the diff is only PR 3's content rather than PR 2's as well. To be closed and reopened againstrapidsai/cudf:mainonce NVIDIA#24501 merges and this branch is rebased.Third of the nz map series. Adds the DELTA_LENGTH_BYTE_ARRAY consumer and the block-wide value loop it uses, still behind
LIBCUDF_PARQUET_NZ_MAP=1.What is here
decode_next_passes_widedecodes one DELTA pass per warp and returns each lane's value in a register, where the path it replaces ran a single warp through a rolling buffer. A call stops at the end of the current DELTA block, which is what makes the passes independent: within a block the header is already parsed, so every pass is addressable with pointer arithmetic, while finding the next block's first byte needs a serial varint parse.classify_nz_map_kindaccepts a second kernel mask so flat DELTA_LENGTH_BYTE_ARRAY pages get a map. Routing matches the DELTA_BINARY pair exactly: both kernels take the caller's page mask unmodified and route on the page's ownhas_nz_map(DELTA_FLAT), with the level-decoding kernel's skip gated on the same flag that decides whether the consumer launches, so the two decisions are one bit.Worth a closer look
The consumer's
DATA_STREAM_OVERRUNcheck. A page whose delta stream holds fewer values than its definition levels call for leaves the tail lengths unwritten, and they are not zero-filled either because the validity map marks those rows valid. The offsets scan that follows would then run over uninitialized memory and produce offsets indexing outside the character data. Three reviewers found this independently and it is the one real bug this round turned up.mini_block_bytes. The byte length of a mini-block is now computed in 64 bits. The bitwidth is an unvalidated page byte, so up to 255, andvalues_per_mbis bounded only by what fits in 32, so the product could wrap and place a mini-block's start below the block, where theblock_endchecks still pass and the unpack reads the wrong bytes. The wide loop walks further ahead than the single-warp path, which is what made this worth closing; the two single-warp sites are widened too rather than left as twins.The two prefix sums over
warp_pass_total. Both are taken before the trailing barrier, not after. The caller's loop has no barrier between iterations, so a warp re-entering the function overwrites its slot and a read placed after the barrier races it — racecheck catches the inverted form. They are unrolled overnum_warps_twith predicated adds because a loop bounded by the runtimewarp_idxandactivecompiles to a variable-length ladder over what is a four-element array.init_binary_blocknow rejects a block size that does not fit in 32 bits.block_sizeis a uleb128 and so 64 bits wide, whilevalues_per_mband the pass counts derived from it are 32: a crafted header can divide evenly in 64 bits, narrow to zero, and leave a loop thread 0 cannot exit. Reachable only through the new loop.Validation
PARQUET_TEST 564 passed / 1 pre-existing skip and HYBRID_SCAN_TEST 106/106, both with the nz map off and on. compute-sanitizer racecheck and memcheck over the delta and nz map tests with the feature on: 0 hazards, 0 errors. An nsys trace confirms the five kernels partition pages as intended — producer, both consumers, and both level-decoding kernels, with the nested column's pages staying on the level-decoding path.
Deliberately not here
Fusing the string offsets scan into the wide loop's existing carry (~16% of DRAM traffic plus a block scan per 128 values) waits for PR 4 so one change covers both string consumers. Splitting
has_flat_nz_mapper kernel mask, which would restore the producer overlap the byte-array stream loses here, waits for PR 5 and blocks the default flip. Wideningmemcpy_blockbelongs outside this stack entirely — it is a shared I/O utility the encoder also uses. All three are written up in the series plan with their triggers.