Skip to content

perf(reorder): cut the plain gate candidate from buffered reads, not re-serialized text - #310

Merged
jayhesselberth merged 1 commit into
mainfrom
perf/reorder-max-mem
Oct 17, 2026
Merged

jayhesselberth merged 1 commit into
mainfrom
perf/reorder-max-mem

Conversation

@jayhesselberth

Copy link
Copy Markdown
Member

The reorder (--max / --order any) path called serialize_block(&all), re-serializing the whole input to FASTQ text only so the plain gate candidate could parse it back, which held a second input-sized buffer. This cuts the candidate directly from the buffered RawBlock + offsets. This is a memory win only. Refs #303.

Changes

  • block.rs: new BlockSource enum (Parsed | Buffered) so write_plain_layout builds blocks from either source; the two compress.rs callers pass Parsed.
  • parse.rs: new block_ranges_from_lens; block_ranges delegates to it.
  • Platform is now resolved with resolve_platform_block on buffered reads (as the clustered path already did) instead of resolve_platform_buf on re-serialized text. A new test asserts they agree on a mixed-header fixture. An exotic header (e.g. trailing tab) could in theory detect differently; none found.
  • No format change.

Evidence (SRR453566_1 GAIIx, 1.84 GB)

  • Archives byte-identical to main (e8dd3d3): --max at 1/8/32 threads and --order any at 8/32 (180,584,067 B each).
  • Peak RSS --max: 9.67 -> 8.18 GB (1t), 10.86 -> 9.28 (8t), 11.27 -> 9.73 (32t). --order any: 9.95 -> 8.18 (8t), 10.24 -> 8.55 (32t). About 15% saving.
  • Wall time within noise (hyperfine, 5 runs, exclusive node): --max 8t 69.6+-2.0 vs 69.5+-2.8 s; 32t 62.3+-3.0 vs 61.7+-2.5 s.
  • Roundtrip: --max decode equals input as a sorted set of name/seq/qual records (reorders reads and drops repeated +, as documented).
  • Tests: plain_candidate_from_slices_matches_serialized; workspace nextest 623 passed, 1 skipped; doctests, clippy, fmt clean.

Part (b) of #303 (overlapping encode_reference_frame with pass 2) was measured and gave no wall or RSS gain, so it was dropped. Overlapping it with names/permutation or the plain candidate was not tried.

🤖 Generated with Claude Code

@jayhesselberth
jayhesselberth merged commit 182acfb into main Oct 17, 2026
8 checks passed
@jayhesselberth
jayhesselberth deleted the perf/reorder-max-mem branch October 17, 2026 22:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant