Skip to content

Cohort Studies with Sample Groups#

This guide explains how to use [[sample_groups]] (WC-02) to run per-sample and per-group analyses across an entire cohort.

Problem#

Population-scale studies require the same pipeline steps to run independently for every sample in every group. Manually duplicating rules for 50+ samples is error-prone and unmaintainable.

Solution: [[sample_groups]]#

Define one [[sample_groups]] block per group. Each block contains a list of sample IDs. Rules that use {sample} or {group} placeholders are expanded once per (group, sample) combination.

[[sample_groups]]
name    = "control"
samples = ["CTRL_001", "CTRL_002", "CTRL_003"]

[[sample_groups]]
name    = "case"
samples = ["CASE_001", "CASE_002"]

Wildcard Placeholders#

Placeholder Replaced with
{sample} Individual sample name
{group} Group name for this sample

Expanded Rule Names#

For a rule align across the groups above the engine produces:

  • align_control_CTRL_001
  • align_control_CTRL_002
  • align_control_CTRL_003
  • align_case_CASE_001
  • align_case_CASE_002

Rules that do not reference {sample} or {group} (e.g., a multiqc step that takes the whole qc/ directory) run once and are kept as-is.

Group Metadata#

Attach arbitrary metadata to each group for use in downstream reporting:

[[sample_groups]]
name    = "treatment_arm_A"
samples = ["PT_A001", "PT_A002"]
[sample_groups.metadata]
drug  = "compound_X"
dose  = "100mg"

Minimal Example#

[workflow]
name = "cohort-minimal"

[config]
reference = "/data/ref/hg38.fa"

[[sample_groups]]
name    = "healthy"
samples = ["H001", "H002"]

[[sample_groups]]
name    = "disease"
samples = ["D001", "D002", "D003"]

[[rules]]
name   = "align"
input  = ["raw/{sample}_R1.fq.gz", "raw/{sample}_R2.fq.gz"]
output = ["aligned/{sample}.bam"]
shell  = "bwa mem -t {threads} -R '@RG\tID:{sample}\tSM:{sample}\tPL:ILLUMINA' {config.reference} {input[0]} {input[1]} | samtools sort -o {output[0]}"

[rules.resources]
threads = 8

[[rules]]
name   = "haplotype_caller"
input  = ["aligned/{sample}.bam"]
output = ["gvcf/{sample}.g.vcf.gz"]
shell  = "gatk HaplotypeCaller -I {input[0]} -R {config.reference} -O {output[0]} -ERC GVCF"

[rules.resources]
threads = 4

# Aggregation step — runs ONCE for all samples.
# The qc/ directory input infers edges to producers writing under it;
# depends_on keeps that ordering explicit (declared names expand to all
# sample instances).
[[rules]]
name   = "multiqc"
input  = ["qc/"]
output = ["reports/multiqc_report.html"]
shell  = "multiqc qc/ -o reports/"
depends_on = ["align"]

Combining Groups and Pairs#

You can use both [[sample_groups]] and [[pairs]] in the same workflow. They expand independently: group-wildcard rules are expanded over samples, and pair-wildcard rules are expanded over pairs.

Duplicate sample ownership

A sample in both a user-declared group and a pair is legal but the engine warns at parse time: unless every rule path distinguishes the instances (e.g. a {group} component), they share an output path and one is skipped as up-to-date while consuming the other's data. The warning prints once per parse naming all duplicates. See Duplicate sample ownership for the fix.

Full Example#

See examples/gallery/12_cohort_analysis.oxoflow for a complete population genomics pipeline including QC, alignment, deduplication, per-sample GVCF calling, cohort joint genotyping, and multi-QC aggregation.

Loading Groups from External File#

For large cohorts with many groups and samples, use sample_groups_file in [workflow] instead of inline [[sample_groups]]:

[workflow]
name = "cohort-analysis"
sample_groups_file = "metadata/groups.tsv"  # or .csv, .json

Supported formats:

TSV Format#

One row per group; the samples column is comma-separated. The same layout works for CSV (with , as delimiter). Every row must supply a cell for every column — a short row fails the sheet parse:

name    samples
healthy H001,H002
disease D001,D002

Extra columns beyond name and samples become group metadata — see Sheet columns as group metadata.

Both name and samples columns are required — a sheet missing either fails at parse with the schema contract in the error message.

CSV: quote comma-separated samples

In CSV the samples cell itself must be quoted, or the sample list spills across columns:

name,samples
healthy,"H001,H002"
disease,"D001,D002"

(TSV needs no quoting — the comma is not the delimiter there.)

Relative sheet paths

A relative sample_groups_file path resolves against the workflow directory (the directory containing the .oxoflow file), not the current working directory — so oxo-flow run … works from anywhere inside the project.

For groups with metadata, use JSON format.

JSON Format#

[
  {
    "name": "treatment_arm_A",
    "samples": ["PT_A001", "PT_A002"],
    "metadata": {
      "drug": "compound_X",
      "dose": "100mg"
    }
  },
  {
    "name": "treatment_arm_B",
    "samples": ["PT_B001", "PT_B002"],
    "metadata": {
      "drug": "compound_Y",
      "dose": "50mg"
    }
  }
]

You can combine inline [[sample_groups]] with sample_groups_file — entries from both sources are merged.

External file benefits

  • Easily manage large cohort definitions
  • Share group definitions across multiple workflows
  • Update sample lists without modifying workflow files
  • Supports metadata for downstream reporting

Pilot Subsets#

For a large cohort, test the full pipeline on a subset before scaling up — no workflow edits needed:

# Pilot: first 2 samples of the cohort, full pipeline
oxo-flow run cohort.oxoflow --samples first:2

# Scale up: completed pilot samples are skipped automatically
oxo-flow run cohort.oxoflow

--samples filters every source declared here ([[sample_groups]], sample_groups_file, sample_pattern) and prunes [[pairs]] whose samples were filtered out. See Pilot runs for details.

When the cohort's data arrives in batches (a sequencing center delivering fastq files over days), check readiness with dry-run and run what is complete with --samples ready.

Selecting Samples on the Command Line#

--samples is the single selection entry point. One parameter, three orthogonal semantics — replace, append, and filter (by name, pilot size, or readiness):

Spec Semantics
--samples @sheet.tsv Replace — the sheet's groups become the run set, overwriting the workflow's inline / auto-discovered / file-loaded groups
--samples +@sheet.tsv Append — same-name groups merge (union, deduplicated); new group names are added. [[pairs]] are untouched: appending can only add samples
--samples S1,S2 Filter on workflows with declared samples (unknown names fail — no phantom samples); declare on workflows that ship with no samples at all (the template-workflow invocation pattern)
--samples first:3 / --samples ready Filter — narrow the (possibly replaced, appended, or declared) set to a subset / to samples whose entry inputs are complete (see Incremental data arrival)

Sheet specs apply in order and can combine with filters: --samples @real.tsv,first:2 replaces with real.tsv then runs the first two. A later @path resets earlier +@path appends (the last replace wins).

Replacing fixture samples with real identifiers#

The core invocation-side use case: a workflow ships with fixture names (e.g. S1/S2) and the caller runs real identifiers without editing the file:

# samplesheet.tsv: one group per row (TSV / CSV / JSON)
#   name     samples
#   cohort   SRR6357072,SRR6357076

oxo-flow run cohort.oxoflow --samples @samplesheet.tsv

The sheet's groups replace the inline [[sample_groups]] (and any sample_pattern / sample_groups_file sources). Samples the workflow never declared are added as new samples, so {group}/{sample} expansion binds to the override. [[pairs]] whose experiment/control are no longer selected are dropped, and stale injected samples_<group> config keys for dropped groups are pruned. A sheet with no data rows (or rows with empty samples) fails the run loudly — it never silently falls back to the workflow's own samples.

Appending to a cohort#

# Adds S3 to the declared cohort (S2 is deduplicated):
#   name     samples
#   cohort   S2,S3
oxo-flow run cohort.oxoflow --samples +@more.tsv

Same-name groups merge with the declared samples (union, deduplicated, original order preserved); a new group name adds a whole new group. [[pairs]] stay as declared — appending never removes a pair's side.