Skip to content

15 — Paired Experiment-Control (Multiple Pairs)#

The scalable version of Paired Experiment-Control (Single Pair): the same somatic pipeline driven by [[pairs]] with {experiment} / {control} / {pair_id} wildcards, fanning out one branch per pair.

What It Demonstrates#

  • [[pairs]] entries expand wildcard rules per pair; aggregation rules without pair wildcards stay single-instance
  • --germline-resource / --panel-of-normals are noted in-rule for production hardening
  • Read-group SM tags per branch make Mutect2 -normal {control} resolve correctly

Pipeline Overview#

graph TD
    FP[fastp_experiment / fastp_control] --> BW[bwa_mem2_experiment / bwa_mem2_control]
    BW --> MD[markdup_experiment / markdup_control]
    MD --> BQ[bqsr_experiment / bqsr_control]
    BQ --> MU[mutect2]
    MU --> FM[filter_mutect_calls]
    FM --> AN[annotate_somatic]
    AN --> CR[clinical_report]

Each rule row in the diagram above is a template that fans out per pair: with the two [[pairs]] entries (CASE_001, CASE_002), every paired rule produces two instances (fastp_control_CASE_001, fastp_control_CASE_002, …) and mutect2 consumes that pair's recalibrated experiment and control BAMs. The pair-scoped chain ends in a per-pair clinical report.

Workflow Definition#

# examples/gallery/15_paired_experiment_control_pairs.oxoflow
# 15 — Paired Experiment-Control (Multiple Pairs)
# The scalable version of 14: the same somatic pipeline driven by
# [[pairs]] with {experiment}/{control}/{pair_id} wildcards.

[workflow]
name = "multi-case-experiment-control"
version = "1.0.0"
description = "Parallel somatic variant calling for multiple experiment-control pairs"
author = "oxo-flow examples"

[config]
reference_fasta = "/data/references/GRCh38/genome.fa"
genome_build = "GRCh38"
known_sites = "/data/references/GRCh38/dbsnp_146.hg38.vcf.gz"

[defaults]
threads = 8
memory = "16G"

[[rules]]
name = "fastp_experiment"
input = [
    "raw/{experiment}_R1.fq.gz",
    "raw/{experiment}_R2.fq.gz",
]
output = [
    "trimmed/{experiment}_R1.fq.gz",
    "trimmed/{experiment}_R2.fq.gz",
    "qc/{experiment}_fastp.json",
]
shell = "fastp -i {input[0]} -I {input[1]} -o {output[0]} -O {output[1]} --json {output[2]} --thread {threads}"

[rules.resources]
threads = 8

[rules.environment]
conda = "envs/fastp.yaml"

[[rules]]
name = "fastp_control"
input = [
    "raw/{control}_R1.fq.gz",
    "raw/{control}_R2.fq.gz",
]
output = [
    "trimmed/{control}_R1.fq.gz",
    "trimmed/{control}_R2.fq.gz",
    "qc/{control}_fastp.json",
]
shell = "fastp -i {input[0]} -I {input[1]} -o {output[0]} -O {output[1]} --json {output[2]} --thread {threads}"

[rules.resources]
threads = 8

[rules.environment]
conda = "envs/fastp.yaml"

[[rules]]
name = "bwa_mem2_experiment"
input = [
    "trimmed/{experiment}_R1.fq.gz",
    "trimmed/{experiment}_R2.fq.gz",
]
output = ["aligned/{experiment}.sorted.bam"]
shell = """
rg=$(printf '@RG\tID:%s\tSM:%s\tPL:ILLUMINA' '{experiment}' '{experiment}')
bwa-mem2 mem -t {threads} -R "$rg" {config.reference_fasta} {input[0]} {input[1]} | samtools sort -@ 4 -o {output[0]}
"""

[rules.resources]
threads = 16
memory = "32G"

[rules.environment]
conda = "envs/bwa_mem2.yaml"

[[rules]]
name = "bwa_mem2_control"
input = [
    "trimmed/{control}_R1.fq.gz",
    "trimmed/{control}_R2.fq.gz",
]
output = ["aligned/{control}.sorted.bam"]
shell = """
rg=$(printf '@RG\tID:%s\tSM:%s\tPL:ILLUMINA' '{control}' '{control}')
bwa-mem2 mem -t {threads} -R "$rg" {config.reference_fasta} {input[0]} {input[1]} | samtools sort -@ 4 -o {output[0]}
"""

[rules.resources]
threads = 16
memory = "32G"

[rules.environment]
conda = "envs/bwa_mem2.yaml"

[[rules]]
name = "markdup_experiment"
input = ["aligned/{experiment}.sorted.bam"]
output = [
    "dedup/{experiment}.dedup.bam",
    "dedup/{experiment}.metrics.txt",
]
shell = "gatk MarkDuplicates --VALIDATION_STRINGENCY SILENT -I {input[0]} -O {output[0]} -M {output[1]}"

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "markdup_control"
input = ["aligned/{control}.sorted.bam"]
output = [
    "dedup/{control}.dedup.bam",
    "dedup/{control}.metrics.txt",
]
shell = "gatk MarkDuplicates --VALIDATION_STRINGENCY SILENT -I {input[0]} -O {output[0]} -M {output[1]}"

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "bqsr_experiment"
input = ["dedup/{experiment}.dedup.bam"]
output = ["recal/{experiment}.recal.bam"]
shell = """
  gatk BaseRecalibrator --read-validation-stringency SILENT -I {input[0]} -R {config.reference_fasta} --known-sites {config.known_sites} -O recal/{experiment}.recal.table && gatk ApplyBQSR -I {input[0]} -R {config.reference_fasta} --bqsr-recal-file recal/{experiment}.recal.table -O {output[0]}
"""

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "bqsr_control"
input = ["dedup/{control}.dedup.bam"]
output = ["recal/{control}.recal.bam"]
shell = """
  gatk BaseRecalibrator --read-validation-stringency SILENT -I {input[0]} -R {config.reference_fasta} --known-sites {config.known_sites} -O recal/{control}.recal.table && gatk ApplyBQSR -I {input[0]} -R {config.reference_fasta} --bqsr-recal-file recal/{control}.recal.table -O {output[0]}
"""

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "mutect2"
input = [
    "recal/{experiment}.recal.bam",
    "recal/{control}.recal.bam",
]
output = ["variants/{pair_id}.mutect2.vcf.gz"]
shell = """
  # Production runs should also pass --germline-resource and --panel-of-normals
  gatk Mutect2 --read-validation-stringency SILENT -I {input[0]} -I {input[1]} -normal {control} -R {config.reference_fasta} -O {output[0]}
"""

[rules.resources]
threads = 4
memory = "16G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "filter_mutect_calls"
input = ["variants/{pair_id}.mutect2.vcf.gz"]
output = ["variants/{pair_id}.mutect2.filtered.vcf.gz"]
shell = "gatk FilterMutectCalls -V {input[0]} -R {config.reference_fasta} -O {output[0]}"

[rules.resources]
threads = 2
memory = "8G"

[rules.environment]
conda = "envs/gatk.yaml"

[[rules]]
name = "annotate_somatic"
input = ["variants/{pair_id}.mutect2.filtered.vcf.gz"]
output = ["annotated/{pair_id}.annotated.vcf.gz"]
shell = "vep --input_file {input[0]} --output_file {output[0]} --format vcf --vcf --offline --cache"

[rules.resources]
threads = 4
memory = "8G"

[rules.environment]
conda = "envs/vep.yaml"

[[rules]]
name = "clinical_report"
input = ["annotated/{pair_id}.annotated.vcf.gz"]
output = ["reports/{pair_id}_clinical_report.html"]
shell = "python scripts/generate_report.py --input {input[0]} --output {output[0]} --pair {pair_id}"
description = "Generate clinical-grade variant report for a experiment-control pair"

[rules.resources]
threads = 2
memory = "4G"

[rules.environment]
conda = "envs/report.yaml"

[[pairs]]
pair_id = "CASE_001"
experiment = "EXP_01"
control = "CTRL_01"
experiment_type = "lung_adenocarcinoma"

[pairs.metadata]

[[pairs]]
pair_id = "CASE_002"
experiment = "EXP_02"
control = "CTRL_02"
experiment_type = "colorectal"

[pairs.metadata]

Running the Workflow#

Validate#

$ oxo-flow validate examples/gallery/15_paired_experiment_control_pairs.oxoflow
✓ examples/gallery/15_paired_experiment_control_pairs.oxoflow — 12 rules, 17 dependencies

  ⚠ Warning: The following input files do not exist:
    - raw/EXP_01_R1.fq.gz
    - raw/EXP_01_R2.fq.gz
    - raw/CTRL_01_R1.fq.gz
    - raw/CTRL_01_R2.fq.gz
    - raw/EXP_02_R1.fq.gz
    ...

The ✓ line is what matters — validate exits 0 and the workflow is well-formed. The warnings are expected here: the raw/ FASTQs don't exist until you stage real data (see the note at the bottom of this page).

Dry-Run#

$ oxo-flow dry-run examples/gallery/15_paired_experiment_control_pairs.oxoflow
Plan: would run: 24 | skip: 0 | completed: 0 (DAG size: 24)
Sample readiness: 0/4 complete, 4 waiting
    ⏳ EXP_01 (missing: raw/EXP_01_R1.fq.gz +1 more)
    ⏳ CTRL_01 (missing: raw/CTRL_01_R1.fq.gz +1 more)
    ⏳ EXP_02 (missing: raw/EXP_02_R1.fq.gz +1 more)
    ⏳ CTRL_02 (missing: raw/CTRL_02_R1.fq.gz +1 more)
  ...

17. mutect2_CASE_001  [run: never completed]
     outputs: ["variants/CASE_001.mutect2.vcf.gz"]
     command: # Production runs should also pass --germline-resource and --panel-of-normals
gatk Mutect2 ... -I recal/EXP_01.recal.bam -I recal/CTRL_01.recal.bam -normal CTRL_01 -O variants/CASE_001.mutect2.vcf.gz

Summary: 24 rules, total 152 threads declared, max 16 threads/rule
         24 rules with memory requirements

To execute:  oxo-flow run examples/gallery/15_paired_experiment_control_pairs.oxoflow -j 2

The 12 template rules expand to 24 tasks — each pair-scoped rule runs once per [[pairs]] entry, with instance names keyed by pair_id (fastp_control_CASE_001, mutect2_CASE_001, clinical_report_CASE_001, …). Compare the mutect2 command with workflow 14: -I recal/EXP_01.recal.bam and -normal CTRL_01 are literal there, while here {experiment} and {control} resolve them per pair. The in-rule comment about --germline-resource and --panel-of-normals is carried verbatim into the command — production runs should uncomment those flags with real resource paths.

DAG Structure#

$ oxo-flow graph examples/gallery/15_paired_experiment_control_pairs.oxoflow
Workflow DAG: 12 rules, 17 dependencies
Depth: 8, Width: 2, Critical path: 8 steps

Critical path: fastp_control → bwa_mem2_control → markdup_control → bqsr_control → mutect2 → filter_mutect_calls → annotate_somatic → clinical_report

graph shows the template DAG (before expansion — see the gallery index note). Unlike the single-pair workflow 14, the paired rules show cross-branch dependencies at every level — bwa_mem2_control [depends: fastp_control, fastp_experiment] and markdup_control [depends: bwa_mem2_control, bwa_mem2_experiment] — because each branch's files must be uniquely named per pair, so every stage waits on both the experiment and control upstream instances before writing its own pair-scoped outputs.

Expected Outputs#

Path Producer
trimmed/{experiment}_R1.fq.gz, trimmed/{control}_R1.fq.gz, … fastp_experiment / fastp_control
aligned/{experiment}.sorted.bam, aligned/{control}.sorted.bam bwa_mem2_experiment / bwa_mem2_control
dedup/{experiment}.dedup.bam, dedup/{control}.dedup.bam markdup_experiment / markdup_control
recal/{experiment}.recal.bam, recal/{control}.recal.bam bqsr_experiment / bqsr_control
variants/{pair_id}.mutect2.vcf.gz mutect2
variants/{pair_id}.mutect2.filtered.vcf.gz filter_mutect_calls
annotated/{pair_id}.annotated.vcf.gz annotate_somatic
reports/{pair_id}_clinical_report.html clinical_report

With the shipped [[pairs]] table, {pair_id} resolves to CASE_001 (lung adenocarcinoma) and CASE_002 (colorectal).

What's Next?#

Input data and environments

Input paths under /data/references/... and raw/ are placeholders — replace them with your own data. The validate warnings about missing raw/ files are expected until real data is staged. The referenced envs/*.yaml specs ship in examples/envs/.