Summary
Batch ingestion of a 96-video corpus fails in Ray Data's global repartition immediately before VideoFrameTextDedup. The failure occurs after video splitting, ASR, and almost all frame OCR, but before frame-text deduplication, audio/visual fusion, embedding, or VDB writing.
A one-video mixed canary passes, so this is a scale-specific pipeline failure.
Observed failure
At the final progress update:
ReadBinary->SplitBlocks: 96/96VideoSplitActor: 73,669/73,669ASRGPUActor: 73,669/73,669VideoFrameOCRGPUActor: 51,047/51,369- global
Repartition: 0/51,369, with 478 queued blocks and about 30.4 GiB in the object store
VideoFrameTextDedup: 0/1
Ray then raises:
pyarrow.lib.ArrowInvalid: offset overflow while concatenating arrays, consider casting input from string to large_string first
The stack is in Ray's shuffle reduce path:
shuffle_task_spec.py: reduce -> accessor.size_bytes()
pandas_block.py: sample(...).values
pandas ArrowExtensionArray.take()
pyarrow.ChunkedArray.take()
No video rows reached the embedding or VDB stages.
Execution details
- nv-ingest commit:
9341e13326ba656972401b980def3438128021ac
- Ray: 2.56.1
- PyArrow: 25.0.0
- pandas: 2.3.3
- video inputs: 96 MP4 files
- video frame rate: 1 FPS
- frame hash dedup: enabled
- frame OCR text dedup: enabled, max dropped frames 2
- audio/visual fusion: enabled
- one-video canary output: 35 rows, successful
The relevant video executor/operator files are unchanged from the earlier benchmark commit 3ad2c5591034a672c8dffd8bc8189f0228834745.
Likely scaling seam
VideoFrameTextDedup and AudioVisualFuser are per-source self-joins and declare:
REQUIRES_GLOBAL_BATCH = True
GLOBAL_BATCH_GROUP_KEYS = ("source_path",)The executor defaults callable-class concurrency to 1. With concurrency 1, the group-key branch is not used and the executor calls repartition(num_blocks=1), coalescing every video's frame rows into one large block. Ray fails while sizing/sampling that concatenated pandas/Arrow block.
The original large Megaretriever run did not expose this because it stalled in the text VDB stage before reaching full-video ingestion; only the small one-video canary ran.
Expected behavior / acceptance
- Preserve the invariant that all rows for one
source_path reach the same dedup/fusion invocation.
- Do not coalesce unrelated videos into one global block.
- Support cumulative intermediate payloads beyond Arrow
string's 32-bit offset limit, either by bounded per-source partitioning or a safe large_string schema at the relevant boundary.
- Add a regression test that exercises the actual Ray repartition/pandas-Arrow seam, plus a multi-video scale qualification.
Summary
Batch ingestion of a 96-video corpus fails in Ray Data's global repartition immediately before
VideoFrameTextDedup. The failure occurs after video splitting, ASR, and almost all frame OCR, but before frame-text deduplication, audio/visual fusion, embedding, or VDB writing.A one-video mixed canary passes, so this is a scale-specific pipeline failure.
Observed failure
At the final progress update:
ReadBinary->SplitBlocks: 96/96VideoSplitActor: 73,669/73,669ASRGPUActor: 73,669/73,669VideoFrameOCRGPUActor: 51,047/51,369Repartition: 0/51,369, with 478 queued blocks and about 30.4 GiB in the object storeVideoFrameTextDedup: 0/1Ray then raises:
The stack is in Ray's shuffle reduce path:
No video rows reached the embedding or VDB stages.
Execution details
9341e13326ba656972401b980def3438128021acThe relevant video executor/operator files are unchanged from the earlier benchmark commit
3ad2c5591034a672c8dffd8bc8189f0228834745.Likely scaling seam
VideoFrameTextDedupandAudioVisualFuserare per-source self-joins and declare:The executor defaults callable-class concurrency to 1. With concurrency 1, the group-key branch is not used and the executor calls
repartition(num_blocks=1), coalescing every video's frame rows into one large block. Ray fails while sizing/sampling that concatenated pandas/Arrow block.The original large Megaretriever run did not expose this because it stalled in the text VDB stage before reaching full-video ingestion; only the small one-video canary ran.
Expected behavior / acceptance
source_pathreach the same dedup/fusion invocation.string's 32-bit offset limit, either by bounded per-source partitioning or a safelarge_stringschema at the relevant boundary.