Skip to content

03 — Parallel Samples#

Process multiple samples in parallel using wildcard expansion. This pattern is fundamental to bioinformatics — one workflow definition handles any number of samples.

Concepts Covered

  • {sample} wildcard expansion
  • Fan-out / fan-in patterns
  • Per-rule resource declarations (threads, memory)
  • Default resource settings via [defaults]

Workflow Definition#

# examples/gallery/03_parallel_samples.oxoflow
# 03 — Parallel Sample Processing
# Process multiple samples in parallel using wildcard expansion.
# Demonstrates: {sample} wildcards, fan-out/fan-in, resource declarations.

[workflow]
name = "parallel-samples"
version = "1.0.0"
description = "Process multiple samples in parallel using wildcards"
author = "oxo-flow examples"

[defaults]
threads = 2
memory = "4G"

# Define the sample list for {sample} wildcard expansion.
# Each entry under [[sample_groups]] creates one expanded instance
# per rule that uses the {sample} wildcard.
[[sample_groups]]
name = "all"
samples = ["sampleA", "sampleB", "sampleC"]

[[rules]]
name = "preprocess"
input = ["raw/{sample}.txt"]
output = ["processed/{sample}.clean.txt"]
shell = """
mkdir -p processed
sed '/^$/d' {input[0]} | sort > {output[0]}
"""

[[rules]]
name = "analyze"
input = ["processed/{sample}.clean.txt"]
output = ["analysis/{sample}.stats.txt"]
shell = """
mkdir -p analysis
lines=$(wc -l < {input[0]})
words=$(wc -w < {input[0]})
chars=$(wc -c < {input[0]})
echo "sample: {sample}" > {output[0]}
echo "lines: $lines" >> {output[0]}
echo "words: $words" >> {output[0]}
echo "chars: $chars" >> {output[0]}
"""

[rules.resources]
threads = 4
memory = "8G"

[[rules]]
name = "aggregate"
input = ["analysis/sampleA.stats.txt", "analysis/sampleB.stats.txt", "analysis/sampleC.stats.txt"]
output = ["results/combined_report.txt"]
shell = """
mkdir -p results
echo "=== Combined Analysis Report ===" > {output[0]}
echo "Generated by oxo-flow" >> {output[0]}
echo "" >> {output[0]}
cat {input} >> {output[0]}
"""

Key Concepts#

Wildcard Expansion#

The {sample} pattern in file paths is a wildcard. When oxo-flow encounters wildcards, it expands the rule into concrete instances based on:

  1. Input file discovery — scanning the filesystem for files matching the pattern
  2. Explicit configuration — listing sample names in a [[sample_groups]] table

Here, the [[sample_groups]] table defines three samples (sampleA, sampleB, sampleC), so the preprocess rule expands into three independent jobs that can run in parallel.

See the wildcards reference for every expansion source (file discovery, pairs, sample groups) and for regex constraints.

Resource Declarations#

Each rule can declare its resource requirements:

[rules.resources]
threads = 4     # CPU cores needed
memory = "8G"   # RAM needed (supports G, M, K, T suffixes)

The [defaults] section provides fallback values for rules that don't specify resources.

Fan-Out / Fan-In#

  • Fan-out: The preprocess and analyze rules create one job per sample → parallel execution
  • Fan-in: The aggregate rule collects all per-sample results into a single output

This example writes the fan-in inputs out explicitly to keep it minimal. In real workflows, expand_inputs derives them from the sample list automatically. Full mechanics: Fan-out vs Fan-in.

Running the Workflow#

Validate#

$ oxo-flow validate examples/gallery/03_parallel_samples.oxoflow
✓ examples/gallery/03_parallel_samples.oxoflow — 3 rules, 1 dependency

DAG Structure#

graph TD
    A1[preprocess<br/>sample=sampleA] --> B1[analyze<br/>sample=sampleA]
    A2[preprocess<br/>sample=sampleB] --> B2[analyze<br/>sample=sampleB]
    A3[preprocess<br/>sample=sampleC] --> B3[analyze<br/>sample=sampleC]
    B1 --> C[aggregate]
    B2 --> C
    B3 --> C

What's Next?#

Move on to Scatter-Gather to learn how to scatter work by chromosome, process the partitions in parallel, and merge the results.