Update bcftools by mertydn · Pull Request #8435 · galaxyproject/tools-iuc

mertydn · 2026-09-18T08:53:41Z

This PR updates bcftools tool wrappers from 1.21 to 1.24 to expose newly added functionalities from recent bcftools releases, align with upstream changes, and fix bugs and documentation errors.

General & Macro Updates

Added shared --verbosity controls across wrappers (macro_verbosity).
Added shared --no-version controls across wrappers (macro_no_version).
Added @THREADS@ handling where supported.
Updated input handling: removed legacy vcf.gz references; all compressed VCF inputs now properly use vcf_bgzip.
Updated test fixtures: migrated from uncompressed mpileup.vcf to compressed mpileup.vcf.gz.
Added automatic bgzip and tabix indexing for tabular allele frequency files (--AF-file).
Enabled u (uncompressed BCF) and z (compressed VCF) output types, and added a shared compression_level (0-9) parameter.
Renamed the common collapse option any to all to match bcftools 1.22+.

Tool-Specific Changes

`bcftools_annotate`

Added parameters: --force, --single-overlaps, --merge-logic, --pair-logic, --keep-sites.
Fixed typo in help documentation examples (inut.vcf -> input.vcf) and added @TARGETS_HELP@.
Added two tests covering the new parameters and merge logic.

`bcftools_call`

Added parameters: --group-samples-tag and --keep-unseen-allele.
Updated tests to use mpileup.vcf.gz fixture.
Added a test verifying the new 1.24 parameters.

`bcftools_concat`

Added parameters: --naive-force, --drop-genotypes.
Added exact value for --rm-dups.
Fixed naive mode when output_type=v: forces intermediate compressed output (z) followed by bcftools view conversion to prevent compression type mismatch errors.
Separated --naive-force and --drop-genotypes into distinct test cases (since bcftools does not allow combining them).

`bcftools_consensus`

Fixed a Cheetah template syntax bug ($mode.haplotype.rule was missing $).
Added the -H I value.
Added parameters: --missing and --prefix.
Added a new test for the 1.24 parameters.

`bcftools_convert_from_vcf`

Enabled the GL tag option for .gen format output (previously commented out).
Added @THREADS@, @VERBOSITY@, and @NO_VERSION@.

`bcftools_convert_to_vcf`

Added @THREADS@, @VERBOSITY@, and @NO_VERSION@.
Expanded help documentation describing supported input formats (GEN, HAP, TSV).
Updated test to verify shared options.

`bcftools_csq`

Added parameters: --greedy and GFF options (--force, --unify-chr-names).
Added @THREADS@ support.
Updated test 3 to assert the new options.

`bcftools_filter`

Updated --SnpGap to accept variant-type suffixes with regex validation.
Added tests explicitly exercising 2:indel and 2:mnp syntax.
Added usage examples to the help section.

`bcftools_gtcheck`

Modernized parameter interface to match 1.24:
- Removed obsolete parameters (all-sites, plot, target-sample, query-sample).
- Added --distinctive-sites, --dry-run, --error-probability (-E), --keep-refs, --n-matches, --no-HWE-prob, --pairs (-p), --pairs-file (-P), and --use (-u).
Allowed genotypes (-g) input to accept vcf_bgzip.
Added sample, include, and exclude restriction macros.
Expanded documentation explaining sample swap and contamination detection.

`bcftools_merge`

Added parameters: --gvcf (with or without reference FASTA), --force-single, --missing-to-ref, --local-alleles, --unobserved (* / **), --missing-rules, and --filter-logic.
Added snp-ins-del option to --merge.
Added a test verifying the new parameters.

`bcftools_mpileup`

Updated default values to align with bcftools 1.24 built-in defaults:
- tandem-qual: 100 → 500
- min-ireads: 1 → 2
- gap-frac: 0.002 → 0.05
Renamed --non-reference to --no-reference (per bcftools 1.22+).
Added parameters: --config (preset profile) and -M / --max-read-len.
Added new output annotation tags: NMBZ, QM, SCR, INFO/FS, INFO/MQSBZ.
Fixed documentation: replaced misplaced csq help text with proper mpileup documentation and examples.
Added a test covering --config, --no-version, and filtering options.

`bcftools_norm`

Replaced --do-not-normalize with a structured realign_mode select:
- all (default, preserves previous wrapper behavior by normalizing all indels)
- skip (-N, disables realignment)
- length (-N NUM, skips realignment for events longer than specified bp)
Added a regression test asserting that -N is not present when running with default settings.
Added parameters: --multi-overlaps, --force, --gff-annot, --keep-sum, and exact value for -d.
Expanded help text with atomization examples.

`bcftools_query`

Added parameters: --disable-automatic-newline (-N), --print-filtered (-F), and --force-samples.
Documented additional format tags (%INFO, %FORMAT, %FUNCTION, %N_PASS, %PBINOM, %VKX) and added practical query examples.

`bcftools_reheader`

Added --fai option to update contig headers from a FASTA index.
Added @THREADS@ support.
Re-enabled and fixed sample renaming test cases.

`bcftools_roh`

Updated viterbi_training parameter from boolean to a float convergence threshold, as required by bcftools roh.
Added --AF-tag parameter.
Fixed sliding buffer size/overlap conditions for numeric checks.
Added tests covering --viterbi-training "1e-10" and general options.
Expanded help text explaining ROH detection and RG/ST output records.

`bcftools_stats`

Replaced legacy boolean --verbose flag with the shared macro_verbosity macro.
Updated test cases to use the bgzipped mpileup.vcf.gz fixture.
Re-enabled the compressed VCF test case.
Added test covering --1st-allele-only, --depth, --af-bins, --split-by-ID, and --verbosity.

`bcftools_view`

Added --trim-unseen-allele (-A / -AA).
Added allele count/frequency type selectors (nref, alt1, minor, major, nonmajor) for -c, -C, -q, -Q.
Added --force-samples and --no-update subset options.
Updated test 1 to assert new parameters.
Added practical examples to the help section.

FOR CONTRIBUTOR:

I have read the CONTRIBUTING.md document and this tool is appropriate for the tools-iuc repo.
Use of AI
- The contribution is mostly AI generated
- The contribution has been assisted by AI
License permits unrestricted use (educational + commercial)
This PR adds a new tool or tool collection
This PR updates an existing tool or tool collection
This PR does something else (explain below)

…' to 'all'

…ro_verbosity and macro_no_version, add test cases

…tests - Replace --do-not-normalize with realign_mode select (all, skip, length) - Keep 'all' as default to preserve backwards-compatible normalization behavior - Expose --multi-overlaps, --force, --gff-annot, and --keep-sum - Wire shared macro_verbosity and macro_no_version - Add test for new options and regression test for default normalization - Expand documentation with atomization examples

…d tests - Support SnpGap string syntax (e.g. 2:indel, 2:mnp) with regex validation - Wire shared macro_verbosity and macro_no_version - Add tests for inverted targets and SnpGap variant-type syntax - Add usage examples to help section

…ig and tests - Update defaults to match bcftools 1.24 (tandem-qual 500, min-ireads 2, gap-frac 0.05) - Rename --non-reference to --no-reference - Add --config, --max-read-len and etc. params - Wire shared macro_verbosity and macro_no_version - Replace misplaced csq help text with proper mpileup documentation - Add test for new parameters

… options - Add --gvcf support for gVCF block merging - Expose --force-single, --missing-to-ref (-0), and --local-alleles - Add --unobserved to drop unobserved alleles (* / **) - Add --missing-rules and --filter-logic - Wire shared macro_no_version - Add test covering new parameters

…re shared macros - Expose --trim-unseen-allele (-A / -AA) - Add allele count/frequency type selectors (alt1, minor, major, etc.) - Add --force-samples and --no-update options - Wire shared macro_verbosity and macro_no_version - Update test 1 to cover new parameters - Add usage examples to help section

…and tests - Change viterbi_training from boolean to float convergence threshold - Expose --AF-tag parameter - Fix buffer_size and buffer_overlap conditions - Wire shared macro_verbosity - Add tests covering general options and --viterbi-training - Expand documentation for ROH and Viterbi training

…e shared macros - Expose --group-samples-tag for single-sample calling - Expose --keep-unseen-allele to retain unobserved alleles - Wire shared macro_verbosity and macro_no_version - Update test inputs to use bgzipped mpileup fixture - Add test for 1.24 parameters

… new options - Expose --force, --single-overlaps, and --pair-logic - Add --merge-logic for tabular annotations - Add --keep-sites to preserve filtered sites - Wire shared macro_verbosity and macro_no_version - Fix typo in help examples and add targets help - Add tests covering new parameters

…ing, -H I, --prefix - Fix missing $ prefix in mode.haplotype.rule Cheetah evaluation - Add -H I (IUPAC code for all genotypes) - Add --missing option to mark missing genotypes with a custom character - Add --prefix to prepend string to output sequence names - Wire shared macro_verbosity - Add test for new 1.24 parameters

- Expose --distinctive-sites, --dry-run, and --error-probability (-E) - Add --keep-refs, --n-matches, and --no-HWE-prob - Add --pairs (-p), --pairs-file (-P), and --use (-u) - Add samples, include, and exclude restriction macros - Wire shared macro_verbosity - Update test and expand help documentation

…d expand docs - Expose --disable-automatic-newline (-N) and --print-filtered (-F) - Add --force-samples option - Wire shared macro_verbosity - Document new format fields (%INFO, %FORMAT, %PBINOM, %VKX) - Add practical usage examples to help section

@threads

… wire verbosity - Expose --fai option to update contig headers from FASTA index - Add @threads@ support - Wire shared macro_verbosity - Fix and re-enable sample renaming tests - Expand help documentation

@threads

… and tests - Expose --greedy option - Add GFF options: --force and --unify-chr-names - Add @threads@ support - Wire shared macro_verbosity and macro_no_version - Update test 3 to cover new options

… tests - Replace --verbose boolean with shared macro_verbosity - Update test cases to use bgzipped mpileup fixture - Re-enable compressed VCF test case - Add test for depth, af-bins, split-by-ID, and verbosity options - Fix user_tstv regex validator

@threads

…os, update docs - Enable GL option for .gen tag selection - Add @threads@ support - Wire shared macro_verbosity and macro_no_version - Capitalize labels for Galaxy consistency - Update tests and expand format documentation

@threads

… and tests - Add @threads@ support - Wire shared macro_verbosity and macro_no_version - Update test 7 to cover shared options - Expand input format documentation in help section

- Wire shared macro_verbosity in general options - Add verbosity check to test 5

- Wire shared macro_verbosity and macro_no_version - Update test 2 to check for verbosity and no-version options

…ength cutoff for -N

…plit conditional

1. Better explanation of options 2. Remove unused "none" macro - the default 3. Use previously unused "id" macro where appropriate.

--collapse is deprecated but still works. Remove support for it from the wrapper later, then reintroduce --pair-logic with more descriptive help.

These need to be done properly. Leave to next update.

wm75 · 2026-11-13T12:03:53Z

@mertydn I just pushed some minimal fixes. The idea would be to get this merged fast, then do another round of improvements and bring back the gt_check and annotate changes I dismissed this time on v1.25, which is out since 2 weeks by now).

mvdbeek · 2026-11-13T16:27:35Z

Deployment status: success

https://github.com/galaxyproject/tools-iuc/actions/runs/43450795773

mertydn force-pushed the update-bcftools branch from 9c450c1 to bf7be7a Compare September 18, 2026 08:57

mertydn force-pushed the update-bcftools branch 2 times, most recently from 9d4e01f to ee075fa Compare October 3, 2026 00:43

mertydn added 27 commits October 14, 2026 20:38

bcftools macros: bump VERSION_SUFFIX to 1

aeb3dc1

bcftools macros: update input handling, drop vcf.gz support

590f3bb

bcftools macros: bgzip and tabix-index AF file before use

710c07f

bcftools macros: update collapse options, rename collapse option 'any…

d50f6dd

…' to 'all'

bcftools macros: add compression_level param, enable all output formats

058b0e8

bcftools macros: clarify --include help

bace859

bcftools macros: add shared --verbosity and --no-version macros

269b3bd

bcftools_concat: add --naive-force, --drop-genotypes, wire shared mac…

b905638

…ro_verbosity and macro_no_version, add test cases

bcftools_reheader: add --fai option, re-enable sample renaming tests,…

27839fb

… wire verbosity - Expose --fai option to update contig headers from FASTA index - Add @threads@ support - Wire shared macro_verbosity - Fix and re-enable sample renaming tests - Expand help documentation

bcftools_csq: add --greedy, GFF options (--force, --unify-chr-names),…

366f737

… and tests - Expose --greedy option - Add GFF options: --force and --unify-chr-names - Add @threads@ support - Wire shared macro_verbosity and macro_no_version - Update test 3 to cover new options

bcftools_convert_to_vcf: add threads, wire shared macros, update docs…

bd7a448

… and tests - Add @threads@ support - Wire shared macro_verbosity and macro_no_version - Update test 7 to cover shared options - Expand input format documentation in help section

bcftools_cnv: wire shared verbosity macro and update tests

a221e83

- Wire shared macro_verbosity in general options - Add verbosity check to test 5

bcftools_isec: wire shared verbosity and no-version macros, update tests

57f4f72

- Wire shared macro_verbosity and macro_no_version - Update test 2 to check for verbosity and no-version options

bcftools test-data: replace mpileup.vcf with bgzipped version

0f0568d

wm75 reviewed Oct 23, 2026

View reviewed changes

Comment thread tools/bcftools/bcftools_concat.xml Resolved

mertydn added 2 commits October 23, 2026 09:16

bcftools stats: restore verbose parameter

effc87c

bcftools: remove empty general options from cnv and consensus

36fa961

wm75 requested changes Oct 23, 2026

View reviewed changes

wm75 reviewed Oct 23, 2026

View reviewed changes

Comment thread tools/bcftools/bcftools_cnv.xml Resolved

wm75 reviewed Oct 23, 2026

View reviewed changes

Comment thread tools/bcftools/test-data/mpileup.vcf Resolved

mertydn added 7 commits October 24, 2026 22:07

bcftools: revert mpileup test data to uncompressed vcf

9736620

bcftools_concat: nest ligate and drop-genotypes in conditionals

a7c98b9

bcftools: extract shared rm_dups macro for concat and norm

5a9897b

bcftools_norm: restore default indel normalization and add optional l…

e058a22

…ength cutoff for -N

bcftools_norm: restrict gff-annot format to gff3

d4c8471

bcftools_norm: convert sort parameter to boolean switch for lex order

02b1bbb

bcftools_norm: move force and keep-sum options inside multiallelics s…

e2e1583

…plit conditional

mertydn requested a review from wm75 November 12, 2026 12:21

wm75 added 10 commits November 13, 2026 09:57

Simplify version commmand

7625495

Improve macros for --collapse option

cd0e9fe

1. Better explanation of options 2. Remove unused "none" macro - the default 3. Use previously unused "id" macro where appropriate.

Quote query params safely

6b96355

Remove new --pair-logic param support as long as --collapse is used.

fb95f28

--collapse is deprecated but still works. Remove support for it from the wrapper later, then reintroduce --pair-logic with more descriptive help.

Replace double- with single-quoting

e2ccf91

Revert changes to gtcheck.xml

6651de7

These need to be done properly. Leave to next update.

More quoting fixes

b801d67

Do not expose disable_automatic_newline option - it's not needed

c9d624b

Fix mpileup quoting

6be83f0

Adjust mpileup test to quoting change

c0e6ca1

wm75 added 2 commits November 13, 2026 15:13

Fix test and quoting broken by previous commits

9651a2a

Fix merge test again

027d76c

wm75 approved these changes Nov 13, 2026

View reviewed changes

wm75 merged commit c083da6 into galaxyproject:main Nov 13, 2026
13 checks passed

Navigation Menu

Repository navigation

Update bcftools - #8435

Update bcftools#8435
wm75 merged 52 commits into
galaxyproject:mainfrom
mertydn:update-bcftools

mertydn commented Sep 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wm75 commented Nov 13, 2026

Uh oh!

Uh oh!

mvdbeek commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

mertydn commented Sep 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

General & Macro Updates

Tool-Specific Changes

bcftools_annotate

bcftools_call

bcftools_concat

bcftools_consensus

bcftools_convert_from_vcf

bcftools_convert_to_vcf

bcftools_csq

bcftools_filter

bcftools_gtcheck

bcftools_merge

bcftools_mpileup

bcftools_norm

bcftools_query

bcftools_reheader

bcftools_roh

bcftools_stats

bcftools_view

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wm75 commented Nov 13, 2026

Uh oh!

Uh oh!

mvdbeek commented Nov 13, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mertydn commented Sep 18, 2026 •

edited

Loading

`bcftools_annotate`

`bcftools_call`

`bcftools_concat`

`bcftools_consensus`

`bcftools_convert_from_vcf`

`bcftools_convert_to_vcf`

`bcftools_csq`

`bcftools_filter`

`bcftools_gtcheck`

`bcftools_merge`

`bcftools_mpileup`

`bcftools_norm`

`bcftools_query`

`bcftools_reheader`

`bcftools_roh`

`bcftools_stats`

`bcftools_view`