Repository navigation
perf(reorder): cut the plain gate candidate from buffered reads, not re-serialized text - #310
Merged
Merged
Conversation
…re-serialized text
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The reorder (
--max/--order any) path calledserialize_block(&all), re-serializing the whole input to FASTQ text only so the plain gate candidate could parse it back, which held a second input-sized buffer. This cuts the candidate directly from the bufferedRawBlock+ offsets. This is a memory win only. Refs #303.Changes
block.rs: newBlockSourceenum (Parsed|Buffered) sowrite_plain_layoutbuilds blocks from either source; the twocompress.rscallers passParsed.parse.rs: newblock_ranges_from_lens;block_rangesdelegates to it.resolve_platform_blockon buffered reads (as the clustered path already did) instead ofresolve_platform_bufon re-serialized text. A new test asserts they agree on a mixed-header fixture. An exotic header (e.g. trailing tab) could in theory detect differently; none found.Evidence (SRR453566_1 GAIIx, 1.84 GB)
--maxat 1/8/32 threads and--order anyat 8/32 (180,584,067 B each).--max: 9.67 -> 8.18 GB (1t), 10.86 -> 9.28 (8t), 11.27 -> 9.73 (32t).--order any: 9.95 -> 8.18 (8t), 10.24 -> 8.55 (32t). About 15% saving.--max8t 69.6+-2.0 vs 69.5+-2.8 s; 32t 62.3+-3.0 vs 61.7+-2.5 s.--maxdecode equals input as a sorted set of name/seq/qual records (reorders reads and drops repeated+, as documented).plain_candidate_from_slices_matches_serialized; workspace nextest 623 passed, 1 skipped; doctests, clippy, fmt clean.Part (b) of #303 (overlapping
encode_reference_framewith pass 2) was measured and gave no wall or RSS gain, so it was dropped. Overlapping it with names/permutation or the plain candidate was not tried.🤖 Generated with Claude Code