Repository navigation
Use zflate instead of libdeflate for BGZF, sort runs and indexes - #35
Open
DylanCosto wants to merge 1 commit into
Open
DylanCosto wants to merge 1 commit into
DylanCosto wants to merge 1 commit into
Conversation
- bonefish-core depends on zflate 1.0.0 (pure-Rust port of libdeflate) in place of libdeflater: BGZF input (dorado BAM, bgzipped FASTQ), the sorter's level-1 runs, sorted BAM output and BAI/CSI indexes. Tests' BGZF and gzip helpers use it too. A build without jemalloc or mimalloc no longer needs a C compiler; zflate needs Rust 1.91 or newer. - --sort-level is checked to be 0-12 (clap range, and SortedBamWriter::new for library callers) instead of panicking in a compression thread. Output identical: genome-wide sample SAM 9bc7ff3e...; sorted BAM records from FASTQ and from dorado BAM input; BAI and CSI (689 Mbp contig) region and unmapped queries agree with samtools' own indexes; levels 0, 1 and 12 give the same records. CPU time unchanged; the genome-wide BAM is 465 bytes smaller at level 6. cargo test --release --features jemalloc (AVX2/AVX-512 required) and the validation scripts pass; arm64 Linux and macOS type-check.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces the
libdeflater(C libdeflate) dependency with zflate 1.0.0, a pure-Rust port of libdeflate, everywhere bonefish compresses or decompresses DEFLATE:BamFastq, header/@RGreading, BAM detection) and bgzipped FASTQ/FASTA (BgzfParallel)parser_stress.rsPlain (non-BGZF) gzip input still goes through flate2/zlib-rs.
--sort-levelis now checked to be 0-12 (clap range on the CLI, andSortedBamWriter::newfor library callers), so a bad level is an error instead of a panic in a compression thread.SortConfig.levelstaysi32.Build: a build without
jemalloc/mimallocno longer needs a C compiler (cargo tree -i ccis empty on default features). The bioconda recipe still needs one for jemalloc. zflate needs Rust 1.91+; CI uses stable and the Dockerfile 1.98.Verification
Against a build of
main(b44551d), same flags (--release --features jemalloc), 24 threads, GRCh38 no-alt:@PGremoved9bc7ff3e9f498cc1530b2b1d78c47e10, same as v0.4.0samtools quickcheckok; 465 bytes smaller*)samtools index's-y*queries =samtools index -c--sort-level0, 1, 12--sort-level13, -1cargo test --release --locked --features jemallocwithBONEFISH_REQUIRE_SIMD=avx2,avx512bw,avx512vl,avx512vbmi: 212 passed, 0 failed, 5 ignored (as on main)scripts/validationunit tests,verify_evidence.py,verify_clean_dp_evidence.py: passcargo clippy --all-targets: no warningscargo check --all-targetsforaarch64-unknown-linux-gnuandaarch64-apple-darwin: clean (not run on arm64 hardware; CI's arm runners cover that)