Skip to content

12 — Cohort Analysis with Sample Groups#

A population-scale study: multiple [[sample_groups]] expand per-sample instances, the per-sample GVCFs are gathered and joint-genotyped as one cohort, and a single multiqc aggregation step consumes the whole cohort's QC directory.

What It Demonstrates#

  • {sample} rules expand once per sample across all groups (case/control here)
  • Cohort-level joint genotyping: combine_gvcfs gathers every per-sample GVCF via expand_inputs (config.samples_list), and genotype_gvcfs calls the whole cohort in one pass
  • The aggregation rule has no wildcard inputs → exactly one instance; depends_on keeps it behind every per-sample instance of the referenced rules (a template name expands to all its instances; the multiqc rule consumes the QC directory itself, which edge inference links to every producer writing under it)
  • Per-rule conda environments keep tool versions isolated (fastp / bwa-mem2+samtools / gatk / multiqc)

Pipeline Overview#

graph TD
    F[fastp] --> B[bwa_mem2]
    B --> M[markdup]
    M --> H[haplotype_caller]
    M --> Q[multiqc]
    F --> Q
    H --> C[combine_gvcfs]
    C --> G[genotype_gvcfs]

Every rule with a {sample} placeholder fans out per sample (5 instances here: 2 CASE + 3 CTRL), then the chain narrows back down: per-sample GVCFs are gathered by combine_gvcfs into one cohort GVCF, joint genotyping produces the cohort callset, and multiqc aggregates the QC directory in a single instance.

Workflow Definition#

# examples/gallery/12_cohort_analysis.oxoflow
# 12 — Cohort Analysis with Sample Groups
# Population-scale study with per-sample analysis, cohort-level joint
# genotyping, and cohort-level QC aggregation
# (fastp → BWA-MEM2 → MarkDuplicates → HaplotypeCaller → CombineGVCFs →
#  GenotypeGVCFs → MultiQC).

[workflow]
name = "cohort-analysis"
version = "1.0.0"
description = "Population-scale cohort study with per-sample analysis and cohort-level QC aggregation"
author = "oxo-flow examples"

[config]
known_sites = "/data/references/GRCh38/dbsnp_146.hg38.vcf.gz"
reference_fasta = "/data/references/GRCh38/genome.fa"
genome_build = "GRCh38"

[defaults]
threads = 4
memory = "8G"

[[rules]]
name = "fastp"
input = [
    "raw/{sample}_R1.fq.gz",
    "raw/{sample}_R2.fq.gz",
]
output = [
    "trimmed/{sample}_R1.fq.gz",
    "trimmed/{sample}_R2.fq.gz",
    "qc/{sample}_fastp.json",
]
shell = "fastp -i {input[0]} -I {input[1]} -o {output[0]} -O {output[1]} --json {output[2]} --thread {threads}"

[rules.resources]
threads = 4

[rules.environment]
conda = "envs/fastp.yaml"

[[rules]]
name = "bwa_mem2"
input = [
    "trimmed/{sample}_R1.fq.gz",
    "trimmed/{sample}_R2.fq.gz",
]
output = ["aligned/{sample}.sorted.bam"]
shell = '''
rg=$(printf '@RG\tID:%s\tSM:%s\tPL:ILLUMINA' '{sample}' '{sample}')
bwa-mem2 mem -t {threads} -R "$rg" {config.reference_fasta} {input[0]} {input[1]} | samtools sort -@ 4 -o {output[0]}
samtools index {output[0]}
'''

[rules.resources]
threads = 8
memory = "16G"

[rules.environment]
conda = "envs/bwa_mem2.yaml"

[[rules]]
name = "markdup"
input = ["aligned/{sample}.sorted.bam"]
output = [
    "dedup/{sample}.dedup.bam",
    "qc/{sample}.markdup_metrics.txt",
]
shell = "gatk MarkDuplicates --VALIDATION_STRINGENCY SILENT -I {input[0]} -O {output[0]} -M {output[1]} --CREATE_INDEX true"

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "haplotype_caller"
input = ["dedup/{sample}.dedup.bam"]
output = ["gvcf/{sample}.g.vcf.gz"]
shell = "gatk HaplotypeCaller -I {input[0]} -R {config.reference_fasta} -O {output[0]} -ERC GVCF"
description = "Per-sample GVCF generation for joint genotyping"

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "combine_gvcfs"
# Gathers every per-sample GVCF produced by the {sample} rule above.
# expand_inputs is required: the paths only exist after expansion, so the
# pattern (not a literal input) creates the DAG edges.
input = []
expand_inputs = [
    { pattern = "gvcf/{sample}.g.vcf.gz", variables = { sample = "config.samples_list" } }
]
output = ["gvcf/cohort.g.vcf.gz"]
description = "Combine per-sample GVCFs into one multi-sample GVCF"
shell = """
gatk CombineGVCFs \
    -R {config.reference_fasta} \
    $(for f in {input}; do echo "-V $f "; done) \
    -O {output[0]}
"""

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "genotype_gvcfs"
input = ["gvcf/cohort.g.vcf.gz"]
output = ["variants/cohort.genotyped.vcf.gz"]
description = "Joint genotyping across the whole cohort"
shell = "gatk GenotypeGVCFs -R {config.reference_fasta} -V {input[0]} -O {output[0]}"

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "multiqc"
input = ["qc/"]
output = ["reports/multiqc_report.html"]
shell = "multiqc qc/ -o reports/"
description = "Aggregate QC metrics for the entire cohort"
# The qc/ directory input already forms edges to every rule writing under
# it; depends_on keeps that ordering explicit regardless of file matching.
depends_on = [
    "fastp",
    "markdup",
]

[rules.resources]
threads = 2
memory = "4G"

[rules.environment]
conda = "envs/multiqc.yaml"

[[sample_groups]]
name = "control"
samples = [
    "CTRL_001",
    "CTRL_002",
    "CTRL_003",
]

[sample_groups.metadata]
tissue = "blood"

[[sample_groups]]
name = "case"
samples = [
    "CASE_001",
    "CASE_002",
]

[sample_groups.metadata]
tissue = "tumor"

Running the Workflow#

Validate#

$ oxo-flow validate examples/gallery/12_cohort_analysis.oxoflow
✓ examples/gallery/12_cohort_analysis.oxoflow — 7 rules, 8 dependencies

  ⚠ Warning: The following input files do not exist:
    - qc/
    - raw/CTRL_001_R1.fq.gz
    ...

The ✓ line is what matters — validate exits 0 and the workflow is well-formed. The warnings are expected here: the raw/ FASTQs don't exist until you stage real data (see the note at the bottom of this page).

Dry-Run#

$ oxo-flow dry-run examples/gallery/12_cohort_analysis.oxoflow
Plan: would run: 23 | skip: 0 | completed: 0 (DAG size: 23)
Sample readiness: 0/5 complete, 5 waiting
    ⏳ CTRL_001 (missing: raw/CTRL_001_R1.fq.gz +1 more)
    ⏳ CTRL_002 (missing: raw/CTRL_002_R1.fq.gz +1 more)
    ⏳ CTRL_003 (missing: raw/CTRL_003_R1.fq.gz +1 more)
    ⏳ CASE_001 (missing: raw/CASE_001_R1.fq.gz +1 more)
    ⏳ CASE_002 (missing: raw/CASE_002_R1.fq.gz +1 more)
    ⚠ workflow-level inputs missing (block every sample): qc/
  ...

22. combine_gvcfs  [run: never completed]
     outputs: ["gvcf/cohort.g.vcf.gz"]
     command: gatk CombineGVCFs -R /data/references/GRCh38/genome.fa $(for f in gvcf/CASE_001.g.vcf.gz gvcf/CASE_002.g.vcf.gz gvcf/CTRL_001.g.vcf.gz gvcf/CTRL_002.g.vcf.gz gvcf/CTRL_003.g.vcf.gz; do echo "-V $f "; done) -O gvcf/cohort.g.vcf.gz

23. genotype_gvcfs  [run: never completed]
     outputs: ["variants/cohort.genotyped.vcf.gz"]

Summary: 23 rules, total 110 threads declared, max 8 threads/rule
         23 rules with memory requirements
         2 sample groups, 0 pairs

To execute:  oxo-flow run examples/gallery/12_cohort_analysis.oxoflow -j 2

The 7 template rules expand to 23 concrete tasks: 5 samples × 4 per-sample rules (fastp + bwa_mem2 + markdup + haplotype_caller) = 20, plus 3 single-instance rules. Instance names carry both the group role and the sample — fastp_case_CASE_001, fastp_control_CTRL_003 — and combine_gvcfs' command shows all five GVCFs injected by expand_inputs in sorted order. The Sample readiness block is the cohort-level view: each sample waits on its own raw inputs, while the workflow-level qc/ input blocks every sample.

DAG Structure#

$ oxo-flow graph examples/gallery/12_cohort_analysis.oxoflow
Workflow DAG: 7 rules, 8 dependencies
Depth: 6, Width: 2, Critical path: 6 steps

Critical path: fastp → bwa_mem2 → markdup → haplotype_caller → combine_gvcfs → genotype_gvcfs

graph shows the template DAG (before expansion — see the gallery index note): a depth-6 linear backbone, with multiqc branching off after markdup.

Expected Outputs#

Path Producer
trimmed/{sample}_R1.fq.gz, trimmed/{sample}_R2.fq.gz, qc/{sample}_fastp.json fastp
aligned/{sample}.sorted.bam bwa_mem2
dedup/{sample}.dedup.bam, qc/{sample}.markdup_metrics.txt markdup
gvcf/{sample}.g.vcf.gz haplotype_caller
gvcf/cohort.g.vcf.gz combine_gvcfs (fan-in over all samples)
variants/cohort.genotyped.vcf.gz genotype_gvcfs
reports/multiqc_report.html multiqc

What's Next?#

Input data and environments

Input paths under /data/references/... and raw/ are placeholders — replace them with your own data. The validate warnings about missing raw/ files and the qc/ directory are expected until real data is staged. The referenced envs/*.yaml specs ship in examples/envs/.