Environment
- perbase 1.4.0 (bioconda, build
h15397dd_0) — the latest available on bioconda, so upgrading is not an option
- Linux x86_64
- Input is BAM (not CRAM), coordinate-sorted with a
.bai index
Summary
base-depth fails with a CSV error when --ref-fasta and --keep-zeros are used together, and aborts partway through writing the output file. Either flag alone works fine.
Steps to reproduce
perbase base-depth --threads 16 --keep-zeros \
--bed-file probeCov.hg38.bed \
--ref-fasta hg38.fa \
WU-cfDNA.dna.sscs.sorted.bam --output out2.tsvprobeCov.hg38.bed is a targeted panel BED. The BAM is indexed. Any BED interval that contains both covered and uncovered positions is enough to trigger it — for example a 100 bp interval where the first 48 bp have no reads:
printf 'chr1\t1053200\t1053300\n' > span.bed
# fails
perbase base-depth --bed-file span.bed --ref-fasta hg38.fa --keep-zeros in.bam > out.tsv # exit 1
Both flags are individually fine:
perbase base-depth --bed-file span.bed --ref-fasta hg38.fa in.bam > out.tsv # exit 0, 21 fields/row
perbase base-depth --bed-file span.bed --keep-zeros in.bam > out.tsv # exit 0, 20 fields/row
perbase base-depth --bed-file span.bed --ref-fasta hg38.fa --keep-zeros in.bam > out.tsv # exit 1
Observed
[2026-09-21T08:41:34Z ERROR perbase] CSV error: found record with 21 fields, but the previous record has 20 fields
followed by a secondary panic:
thread '<unnamed>' panicked at /opt/conda/conda-bld/perbase_.../src/lib/par_granges.rs:265:49:
Sent a serializable to writer: "SendError(..)"
Process exits non-zero.
Analysis
With --ref-fasta, the output table gains a REF_BASE column (field 3):
REF POS REF_BASE DEPTH A C G T N R Y S W K M INS DEL REF_SKIP FAIL COUNT_OF_MATE_RESOLUTIONS NEAR_MAX_DEPTH
But that column is only populated for positions that produced a pileup record. Positions emitted purely because of --keep-zeros have no reference base, and serialize without that field at all — dropping from 21 fields to 20:
chr1 1053201 0 0 0 0 0 ... false <- zero depth, 20 fields
chr1 1053248 C 9 0 9 0 0 0 0 ... false <- covered, 21 fields
The CSV writer fixes the field count from the first record and rejects any record that differs (UnequalLengths), so as soon as a file contains both kinds of rows it aborts. The panic above is a knock-on effect: the writer has already exited, and the
worker thread panics when its channel send fails.
This also explains why the failure looks sample-dependent: a BAM whose BED regions are uniformly covered (all 21-field rows) or uniformly uncovered (all 20-field rows) completes normally. Only samples with mixed coverage fail — which, for a
targeted panel, is most of them.
Impact
On a 165-BAM panel run, the majority of samples aborted. More concerning, the abort leaves a truncated output file behind that looks complete. One sample's output contained 4008 correct 21-field rows followed by a single 20-field row with no
trailing newline — the file was written right up to the point of failure and then abandoned, with no marker in the file itself. Any pipeline that globs these .tsv files will silently consume partial results.
Expected
Any of the following would resolve it:
- Always emit
REF_BASE, writing an empty value (or N) for zero-depth positions, so the field count is constant; or
- Never emit
REF_BASE when --keep-zeros is in effect, so the shape is constant; or
- Configure the CSV writer to tolerate variable field counts / not infer the schema from the first record.
Also happy to know whether --keep-zeros + --ref-fasta is intended to be a supported combination at all.
Workaround
Run base-depth with --keep-zeros but without --ref-fasta (constant 20 fields, no abort), and populate REF_BASE afterwards from the reference FASTA. In my testing over 4008 positions, the column perbase emits is exactly the reference base
at that position, so the column can be reconstructed losslessly — the output is byte-identical apart from that one column.
Environment
h15397dd_0) — the latest available on bioconda, so upgrading is not an option.baiindexSummary
base-depthfails with a CSV error when--ref-fastaand--keep-zerosare used together, and aborts partway through writing the output file. Either flag alone works fine.Steps to reproduce
perbase base-depth --threads 16 --keep-zeros \ --bed-file probeCov.hg38.bed \ --ref-fasta hg38.fa \ WU-cfDNA.dna.sscs.sorted.bam --output out2.tsvprobeCov.hg38.bedis a targeted panel BED. The BAM is indexed. Any BED interval that contains both covered and uncovered positions is enough to trigger it — for example a 100 bp interval where the first 48 bp have no reads:Both flags are individually fine:
Observed
followed by a secondary panic:
Process exits non-zero.
Analysis
With
--ref-fasta, the output table gains aREF_BASEcolumn (field 3):But that column is only populated for positions that produced a pileup record. Positions emitted purely because of
--keep-zeroshave no reference base, and serialize without that field at all — dropping from 21 fields to 20:The CSV writer fixes the field count from the first record and rejects any record that differs (
UnequalLengths), so as soon as a file contains both kinds of rows it aborts. The panic above is a knock-on effect: the writer has already exited, and theworker thread panics when its channel send fails.
This also explains why the failure looks sample-dependent: a BAM whose BED regions are uniformly covered (all 21-field rows) or uniformly uncovered (all 20-field rows) completes normally. Only samples with mixed coverage fail — which, for a
targeted panel, is most of them.
Impact
On a 165-BAM panel run, the majority of samples aborted. More concerning, the abort leaves a truncated output file behind that looks complete. One sample's output contained 4008 correct 21-field rows followed by a single 20-field row with no
trailing newline — the file was written right up to the point of failure and then abandoned, with no marker in the file itself. Any pipeline that globs these
.tsvfiles will silently consume partial results.Expected
Any of the following would resolve it:
REF_BASE, writing an empty value (orN) for zero-depth positions, so the field count is constant; orREF_BASEwhen--keep-zerosis in effect, so the shape is constant; orAlso happy to know whether
--keep-zeros+--ref-fastais intended to be a supported combination at all.Workaround
Run
base-depthwith--keep-zerosbut without--ref-fasta(constant 20 fields, no abort), and populateREF_BASEafterwards from the reference FASTA. In my testing over 4008 positions, the column perbase emits is exactly the reference baseat that position, so the column can be reconstructed losslessly — the output is byte-identical apart from that one column.