12 — Cohort Analysis with Sample Groups#
A population-scale study: multiple [[sample_groups]] expand per-sample instances, the per-sample GVCFs are gathered and joint-genotyped as one cohort, and a single multiqc aggregation step consumes the whole cohort's QC directory.
What It Demonstrates#
{sample}rules expand once per sample across all groups (case/control here)- Cohort-level joint genotyping:
combine_gvcfsgathers every per-sample GVCF viaexpand_inputs(config.samples_list), andgenotype_gvcfscalls the whole cohort in one pass - The aggregation rule has no wildcard inputs → exactly one instance;
depends_onkeeps it behind every per-sample instance of the referenced rules (a template name expands to all its instances; the multiqc rule consumes the QC directory itself, which edge inference links to every producer writing under it) - Per-rule conda environments keep tool versions isolated (fastp / bwa-mem2+samtools / gatk / multiqc)
Pipeline Overview#
graph TD
F[fastp] --> B[bwa_mem2]
B --> M[markdup]
M --> H[haplotype_caller]
M --> Q[multiqc]
F --> Q
H --> C[combine_gvcfs]
C --> G[genotype_gvcfs]
Every rule with a {sample} placeholder fans out per sample (5 instances here: 2 CASE + 3 CTRL), then the chain narrows back down: per-sample GVCFs are gathered by combine_gvcfs into one cohort GVCF, joint genotyping produces the cohort callset, and multiqc aggregates the QC directory in a single instance.
Workflow Definition#
# examples/gallery/12_cohort_analysis.oxoflow
# 12 — Cohort Analysis with Sample Groups
# Population-scale study with per-sample analysis, cohort-level joint
# genotyping, and cohort-level QC aggregation
# (fastp → BWA-MEM2 → MarkDuplicates → HaplotypeCaller → CombineGVCFs →
# GenotypeGVCFs → MultiQC).
[workflow]
name = "cohort-analysis"
version = "1.0.0"
description = "Population-scale cohort study with per-sample analysis and cohort-level QC aggregation"
author = "oxo-flow examples"
[config]
known_sites = "/data/references/GRCh38/dbsnp_146.hg38.vcf.gz"
reference_fasta = "/data/references/GRCh38/genome.fa"
genome_build = "GRCh38"
[defaults]
threads = 4
memory = "8G"
[[rules]]
name = "fastp"
input = [
"raw/{sample}_R1.fq.gz",
"raw/{sample}_R2.fq.gz",
]
output = [
"trimmed/{sample}_R1.fq.gz",
"trimmed/{sample}_R2.fq.gz",
"qc/{sample}_fastp.json",
]
shell = "fastp -i {input[0]} -I {input[1]} -o {output[0]} -O {output[1]} --json {output[2]} --thread {threads}"
[rules.resources]
threads = 4
[rules.environment]
conda = "envs/fastp.yaml"
[[rules]]
name = "bwa_mem2"
input = [
"trimmed/{sample}_R1.fq.gz",
"trimmed/{sample}_R2.fq.gz",
]
output = ["aligned/{sample}.sorted.bam"]
shell = '''
rg=$(printf '@RG\tID:%s\tSM:%s\tPL:ILLUMINA' '{sample}' '{sample}')
bwa-mem2 mem -t {threads} -R "$rg" {config.reference_fasta} {input[0]} {input[1]} | samtools sort -@ 4 -o {output[0]}
samtools index {output[0]}
'''
[rules.resources]
threads = 8
memory = "16G"
[rules.environment]
conda = "envs/bwa_mem2.yaml"
[[rules]]
name = "markdup"
input = ["aligned/{sample}.sorted.bam"]
output = [
"dedup/{sample}.dedup.bam",
"qc/{sample}.markdup_metrics.txt",
]
shell = "gatk MarkDuplicates --VALIDATION_STRINGENCY SILENT -I {input[0]} -O {output[0]} -M {output[1]} --CREATE_INDEX true"
[rules.resources]
threads = 4
memory = "16G"
[rules.environment]
conda = "envs/gatk.yaml"
[[rules]]
name = "haplotype_caller"
input = ["dedup/{sample}.dedup.bam"]
output = ["gvcf/{sample}.g.vcf.gz"]
shell = "gatk HaplotypeCaller -I {input[0]} -R {config.reference_fasta} -O {output[0]} -ERC GVCF"
description = "Per-sample GVCF generation for joint genotyping"
[rules.resources]
threads = 4
memory = "16G"
[rules.environment]
conda = "envs/gatk.yaml"
[[rules]]
name = "combine_gvcfs"
# Gathers every per-sample GVCF produced by the {sample} rule above.
# expand_inputs is required: the paths only exist after expansion, so the
# pattern (not a literal input) creates the DAG edges.
input = []
expand_inputs = [
{ pattern = "gvcf/{sample}.g.vcf.gz", variables = { sample = "config.samples_list" } }
]
output = ["gvcf/cohort.g.vcf.gz"]
description = "Combine per-sample GVCFs into one multi-sample GVCF"
shell = """
gatk CombineGVCFs \
-R {config.reference_fasta} \
$(for f in {input}; do echo "-V $f "; done) \
-O {output[0]}
"""
[rules.resources]
threads = 4
memory = "16G"
[rules.environment]
conda = "envs/gatk.yaml"
[[rules]]
name = "genotype_gvcfs"
input = ["gvcf/cohort.g.vcf.gz"]
output = ["variants/cohort.genotyped.vcf.gz"]
description = "Joint genotyping across the whole cohort"
shell = "gatk GenotypeGVCFs -R {config.reference_fasta} -V {input[0]} -O {output[0]}"
[rules.resources]
threads = 4
memory = "16G"
[rules.environment]
conda = "envs/gatk.yaml"
[[rules]]
name = "multiqc"
input = ["qc/"]
output = ["reports/multiqc_report.html"]
shell = "multiqc qc/ -o reports/"
description = "Aggregate QC metrics for the entire cohort"
# The qc/ directory input already forms edges to every rule writing under
# it; depends_on keeps that ordering explicit regardless of file matching.
depends_on = [
"fastp",
"markdup",
]
[rules.resources]
threads = 2
memory = "4G"
[rules.environment]
conda = "envs/multiqc.yaml"
[[sample_groups]]
name = "control"
samples = [
"CTRL_001",
"CTRL_002",
"CTRL_003",
]
[sample_groups.metadata]
tissue = "blood"
[[sample_groups]]
name = "case"
samples = [
"CASE_001",
"CASE_002",
]
[sample_groups.metadata]
tissue = "tumor"
Running the Workflow#
Validate#
$ oxo-flow validate examples/gallery/12_cohort_analysis.oxoflow
✓ examples/gallery/12_cohort_analysis.oxoflow — 7 rules, 8 dependencies
⚠ Warning: The following input files do not exist:
- qc/
- raw/CTRL_001_R1.fq.gz
...
The ✓ line is what matters — validate exits 0 and the workflow is well-formed. The warnings are expected here: the raw/ FASTQs don't exist until you stage real data (see the note at the bottom of this page).
Dry-Run#
$ oxo-flow dry-run examples/gallery/12_cohort_analysis.oxoflow
Plan: would run: 23 | skip: 0 | completed: 0 (DAG size: 23)
Sample readiness: 0/5 complete, 5 waiting
⏳ CTRL_001 (missing: raw/CTRL_001_R1.fq.gz +1 more)
⏳ CTRL_002 (missing: raw/CTRL_002_R1.fq.gz +1 more)
⏳ CTRL_003 (missing: raw/CTRL_003_R1.fq.gz +1 more)
⏳ CASE_001 (missing: raw/CASE_001_R1.fq.gz +1 more)
⏳ CASE_002 (missing: raw/CASE_002_R1.fq.gz +1 more)
⚠ workflow-level inputs missing (block every sample): qc/
...
22. combine_gvcfs [run: never completed]
outputs: ["gvcf/cohort.g.vcf.gz"]
command: gatk CombineGVCFs -R /data/references/GRCh38/genome.fa $(for f in gvcf/CASE_001.g.vcf.gz gvcf/CASE_002.g.vcf.gz gvcf/CTRL_001.g.vcf.gz gvcf/CTRL_002.g.vcf.gz gvcf/CTRL_003.g.vcf.gz; do echo "-V $f "; done) -O gvcf/cohort.g.vcf.gz
23. genotype_gvcfs [run: never completed]
outputs: ["variants/cohort.genotyped.vcf.gz"]
Summary: 23 rules, total 110 threads declared, max 8 threads/rule
23 rules with memory requirements
2 sample groups, 0 pairs
To execute: oxo-flow run examples/gallery/12_cohort_analysis.oxoflow -j 2
The 7 template rules expand to 23 concrete tasks: 5 samples × 4 per-sample rules (fastp + bwa_mem2 + markdup + haplotype_caller) = 20, plus 3 single-instance rules. Instance names carry both the group role and the sample — fastp_case_CASE_001, fastp_control_CTRL_003 — and combine_gvcfs' command shows all five GVCFs injected by expand_inputs in sorted order. The Sample readiness block is the cohort-level view: each sample waits on its own raw inputs, while the workflow-level qc/ input blocks every sample.
DAG Structure#
$ oxo-flow graph examples/gallery/12_cohort_analysis.oxoflow
Workflow DAG: 7 rules, 8 dependencies
Depth: 6, Width: 2, Critical path: 6 steps
Critical path: fastp → bwa_mem2 → markdup → haplotype_caller → combine_gvcfs → genotype_gvcfs
graph shows the template DAG (before expansion — see the gallery index note): a depth-6 linear backbone, with multiqc branching off after markdup.
Expected Outputs#
| Path | Producer |
|---|---|
trimmed/{sample}_R1.fq.gz, trimmed/{sample}_R2.fq.gz, qc/{sample}_fastp.json |
fastp |
aligned/{sample}.sorted.bam |
bwa_mem2 |
dedup/{sample}.dedup.bam, qc/{sample}.markdup_metrics.txt |
markdup |
gvcf/{sample}.g.vcf.gz |
haplotype_caller |
gvcf/cohort.g.vcf.gz |
combine_gvcfs (fan-in over all samples) |
variants/cohort.genotyped.vcf.gz |
genotype_gvcfs |
reports/multiqc_report.html |
multiqc |
What's Next?#
- How-to guide: Cohort Sample Groups — declaring
[[sample_groups]]and controlling expansion. - Field reference:
sample_groups— Multi-Sample Cohorts (WC-02).
Input data and environments
Input paths under /data/references/... and raw/ are placeholders —
replace them with your own data. The validate warnings about missing
raw/ files and the qc/ directory are expected until real data is
staged. The referenced envs/*.yaml specs ship in
examples/envs/.