Repository navigation
Update multi-sample metagenomic binning workflow - #1452
Conversation
Test Results (powered by Planemo)Test Summary
Errored Tests
|
paulzierep
changed the title
paulzierep
changed the title
Test Results (powered by Planemo)Test Summary
Errored Tests
|
COMEBin runs the longest of all binners and the workflow tests stay close to the CI time limit. Turn it off in the published test case while keeping the other binners, both refinement tools and every existing assertion unchanged. COMEBin stays enabled by default in the workflow.
|
Tests: turn COMEBin off in the published test case to keep the suite inside the CI time limit. COMEBin is the longest binning step: in the last CI runs its five assembly jobs took ~4h30m here, which brought the suite close to the six hour limit (the previous version already needed 5h41m). With COMEBin disabled the test still covers SemiBin, MetaBAT2, MaxBin2 and CONCOCT plus both refinement tools, cross-sample coverage and all existing output assertions. COMEBin stays enabled by default in the workflow and its behaviour is verified separately. |
Reduce the CI fixture from five to two mock samples so the N-squared cross-alignment matrix (25 -> 4 alignments) keeps the test within the runtime limit now that COMEBin is disabled. Update the per-sample element assertions and the coverage-matrix column count (13 -> 7, i.e. 3 fixed plus 2 per-sample columns).
Test Results (powered by Planemo)Test Summary
Failed Tests
|
bebatut
left a comment
bebatut
left a comment
There was a problem hiding this comment.
Thanks for the workflow! It looks good!
I made some suggestions, mostly to remove in annotation the text that is already in the label of the step. For me, annotation should not repeat the label but bring extra explanation of the step.
Would it be possible to change the order of the inputs to have:
- The assemblies
- The reads
- The boolean inputs for binners
- Binnette inputs
?
Also, instead of having a big box with explanation on the top (see screenshot below), would it be possible to have sub-boxing corresponding to "coverage", "binning", "refinement" steps with the corresponding explanation in each box?
That would make it easier for future contributors to identify the different blocks.
UI/UX comment: I found the selection boolean not 100% intuitive. It would be nice to have "Yes/No" and the annotation below.
bebatut
left a comment
bebatut
left a comment
There was a problem hiding this comment.
Thanks for the workflow! It looks good!
I made some suggestions, mostly to remove in annotation the text that is already in the label of the step. For me, annotation should not repeat the label but bring extra explanation of the step.
Would it be possible to change the order of the inputs to have:
- The assemblies
- The reads
- The boolean inputs for binners
- Binnette inputs
?
Also, instead of having a big box with explanation on the top (see screenshot below), would it be possible to have sub-boxing corresponding to "coverage", "binning", "refinement" steps with the corresponding explanation in each box?
That would make it easier for future contributors to identify the different blocks.
UI/UX comment: I found the selection boolean not 100% intuitive. It would be nice to have "Yes/No" and the annotation below.
Replace the five boolean binner toggles with Yes/No selectors gated through map_param_value steps, reorder the inputs so reads and assemblies come first, group the workflow into annotated frames, and rewrite step annotations and the README to explain what each step does.
|
Thanks @bebatut. Updated the step annotations, ordered inputs as assemblies → reads → binner selectors → Binette settings, and split the editor layout into coverage, binning and refinement frames with explanations. Binner selectors now show Yes/No and feed boolean gates internally. The README follows the analysis stages and explains the per-sample coverage-depth columns. The |
bebatut
left a comment
bebatut
left a comment
There was a problem hiding this comment.
Thanks @paulzierep. It seems good on my side (do not have the time to review in details 😅 ), just one comment about changes in the CHANGELOG file
Test Results (powered by Planemo)Test Summary
Errored Tests
|
|
Every tool step now carries an annotation that says what the step does instead of repeating its label, covering the Flatten/Cross Product/Apply Rules steps, the per-binner |
|
Please hold this PR until galaxyproject/galaxy#24665 is resolved. This workflow contains or embeds mapped A minimal two-step example on EU reproduced the problem: 1,001 input datasets generated 3,000 jobs before cancellation, repeatedly scheduling the same inputs. Caching is a suspected factor, not yet a confirmed cause. |
|
Thanks @bebatut — followed up on the remaining wording in The reordered inputs, Yes/No binner choices, separate coverage/binning/refinement frames and release 1.1 CHANGELOG section were already in place. The exported Also simplified the README introduction, linked to the IWC workflow pages, and documented that combining DAS Tool and Binette results requires downstream dereplication but can still retain contigs in multiple bins. The embedded README is synchronized. |
|
Added to README and paired-end reads input: multi-sample binning is most effective with up to ~20 samples, performance peaks around 20 (Mineeva et al., Nat. Commun. 2025), and using more than 20 can lead to extensive resource usage (N² mappings, large storage). For >20 or unrelated environments, single-sample binning is often cheaper. |
bebatut
left a comment
bebatut
left a comment
There was a problem hiding this comment.
Thanks a lot @paulzierep. It looks good to me!
Waiting now for galaxyproject/galaxy#24665
Summary
Publish Multi-sample binning 1.0 as a reusable workflow accepting paired-end
reads and assemblies from any assembler. Every assembly is mapped against all
original per-sample read sets, retaining separate sample coverage profiles for
binning rather than pooling those reads into one coverage profile.
This complements #1453: cross-sample abundance patterns can improve binning for
related environments, but the all-samples × all-assemblies mapping costs more
compute and storage. The workflow also supports co-assemblies while retaining
coverage from the original samples.
Changes and interface
list:pairedand assemblies aslist.Reads should already be quality-filtered and host-removed.
COMEBin. All default to on; keep at least one enabled.
standalone completeness/contamination-weight defaults remain 40 / 2.
samples using total bases / total reads, rounded to the nearest integer.
outputs. Coverage tables retain the original samples' coverage columns.
with Markdown documentation. Yes/No binner selectors feed internal boolean gates.
the single-mode selector from Update single-sample metagenomic binning workflow #1453 is intentionally not exposed here.
This standalone workflow stops at binning/refinement; it does not add grouping,
assembly, dRep, abundance estimation or final reporting.
Validation
collection assertions/minima are retained and fixtures are referenced by URL.
pairscollection directly to Bowtie2; remove the obsolete zip step. Upgrade COMEBin
to 1.1.0+galaxy1 with its exact installable Tool Shed revision.
gxwf validate: 38 tool states pass, zero failures/skips; IWC lint passes.382 connections, zero invalid.
tool upgrades and does not validate this revision. A fresh unchanged Multi
exact case is submitted on EU; results and GitHub CI remain pending.
de-interlacer outputs; uncached retries are running. This remains unresolved,
not a validated pass of the embedded workflow.
Context and related PRs
This is part of the FAIRyMAGs workflow modularization: publish reusable binning
workflows alongside an end-to-end MAG-generation parent, rather than maintaining
separate copies of the same binning logic. Each deliverable has its own branch
and PR so it can be reviewed independently.
Suggested review/merge order: Single → Multi → Parent. The parent embeds its
children, so importing it does not require importing the standalone workflows first.
Developed within FAIRyMAGs, with links
to the related workflows and learning resources in the README.
IWC checklist
FOR CONTRIBUTOR:
FOR REVIEWERS:
This workflow does/runs/performs … xyz … to generate/analyze/etc …namefield should be human readable (spaces are fine, no underscore, dash only where spelling dictates it), no abbreviation unless generally understood-) over underscore (_), prefer all lowercase. Folder becomes repository in iwc-workflows organization and is included in TRS id