Resource Tuning Guide#
This guide covers best practices for declaring CPU, memory, GPU, and disk resources in oxo-flow workflows.
Thread Declaration#
Match threads to the tool's actual parallelism capability. Oversubscribing wastes memory, undersubscribing wastes time.
| Tool | Recommended Threads | Notes |
|---|---|---|
| BWA-MEM2 | 12-16 | Saturates ~12-16 cores; more doesn't help |
| STAR | 16-32 | Scales well up to available cores |
| samtools sort | 4-8 + 2G/thread | Memory-bound: threads × 2GB per thread |
| samtools index | 2-4 | Limited parallelism |
| GATK HaplotypeCaller | 4-8 | Java parallelism limited |
| GATK MarkDuplicates | 1-2 | Mostly single-threaded |
| fastp | 8-16 | Good parallelization |
| FastQC | 2-4 | Limited parallelism |
# Example: BWA alignment
[[rules]]
name = "bwa_align"
[rules.resources]
threads = 16
memory = "32G" # 2× expected input size
Engine conventions
- Always declare resources in the
[rules.resources]sub-table. The deprecated rule-levelthreads/memoryshorthand still works, and takes precedence over the sub-table when both are present. - In the sub-table,
threads <= 1means "unset" — the engine falls back to[defaults].threads. To give a rule fewer threads than the default, declarethreads = 2or higher. Forcing exactly one thread is only possible via the deprecated rule-level shorthandthreads = 1(lint flags it as W025);resources.threads = 1is always read as "unset".
Memory Declaration#
Rule of Thumb#
| Operation Type | Memory Formula |
|---|---|
| Alignment | 2-4 × largest input file size |
| Variant calling (WGS) | 32-64G |
| Variant calling (panel) | 8-16G |
| Sorting/indexing | threads × 2G |
| Assembly | 100-200G for large genomes |
Common Bioinformatics Tools#
| Tool | Memory Recommendation |
|---|---|
| BWA-MEM2 | 32G for human WGS |
| STAR | 64G for human genome |
| GATK HaplotypeCaller | 32G for WGS, 8G for panels |
| GATK BaseRecalibrator | 16G |
| samtools sort | threads × 2G per thread |
| freebayes | 16G |
# Example: WGS variant calling
[[rules]]
name = "haplotype_caller"
[rules.resources]
threads = 8
memory = "64G"
GPU Resources#
SLURM GPU Request#
[[rules]]
name = "gpu_training"
[rules.resources]
threads = 8
memory = "64G"
[rules.resources.gpu_spec]
count = 2
model = "A100"
memory_gb = 40
Generated SLURM directive: --gres=gpu:A100:2:40g --mem-per-gpu=40G (the model string is passed through verbatim)
Common GPU Tools#
| Tool | GPU Memory | Notes |
|---|---|---|
| ParaBricks | 40+ GB per GPU | NVIDIA A100 recommended |
| Clara Parabricks | 32+ GB | GPU-accelerated variant calling |
| DeepVariant GPU | 16+ GB | Faster than CPU version |
PBS/SGE GPU#
GPU syntax varies by site. Use extra_args in the [cluster] section:
Resource Hints for Unknown Requirements#
When you don't know exact requirements:
[[rules]]
name = "novel_tool"
shell = "process_large_data.sh"
[rules.resource_hint]
input_size = "large" # ~100GB input
memory_scale = 2.5 # Need 2.5× input size = 250GB
runtime = "slow" # reserved: parsed, but not consumed by the scheduler — `lint` flags it with W037
Estimated memory: 100GB × 2.5 = 250GB
The runtime hint is a reserved scheduler knob — like io_bound, it is
parsed but not consumed today, and lint flags every use with W037.
Only input_size and memory_scale affect planning; for enforced limits
declare resources.time_limit / resources.memory / resources.threads
instead.
Resource Budgets#
[resource_budget]
max_threads = 64 # Cap any rule at 64 threads; also caps the local pool
max_memory = "256G" # Cap any rule at 256GB; also caps the local pool
max_jobs = 10 # Cap concurrently running jobs (clamp; see below)
max_threads/max_memory are hard per-rule caps on both paths, and the
budget is versioned with the workflow — a shard launched locally with
oxo-flow run inside an allocation honors the same .oxoflow file as a
cluster submission (issue #1080).
- Local (
oxo-flow run): a rule whose declaredthreads/memoryexceeds the cap aborts the run before any rule executes, naming the offender. The cap also bounds the local pool ceiling: at mostmax_threadsrun concurrently and at mostmax_memoryis committed. A budget is a ceiling, never a capacity claim — amax_threads = 64written for a cluster node does not raise a 10-CPU laptop's pool; the effective value ismin(detected, cap). A larger--max-threads/--max-memoryCLI value is clamped by the budget (the budget is a cap); a tighter CLI value wins; a silent CLI inherits the budget, and the run prints a one-line notice naming the active cap. - Cluster (
oxo-flow run --profile <name>): a rule whose declaredmemory/threadsexceeds the cap aborts the run before anything is submitted, naming every offender:
Error: [resource_budget] exceeded by 1 rule — nothing was submitted:
rule 'big_mem' requests 64G but [resource_budget] max_memory = 32G
oxo-flow cluster submitgenerates scripts without the budget check — only therun --profilecluster path enforces it there.- An unreadable or zero cap is an error, never "no cap":
max_memory = "lots"andmax_threads = 0abort with a message.dry-runresolves the same caps, prints them in the summary, and uses the thread cap for its-jsuggestion.
Local fast-fail example:
Error: resource budget too small for 1 rule; no rules were run:
- rule 'big_mem' requires 1024MB memory but the run's memory cap is 16MB (--max-memory / [resource_budget] max_memory)
max_jobs is a run-wide concurrency cap and is enforced on both paths:
- Local (
oxo-flow run): it clamps the requested-j/--jobsconcurrency. Asking for-j 16undermax_jobs = 4runs at most 4 jobs at once and prints a warning saying so — it never errors, since a job count is a total, not a per-rule request. - Cluster (
oxo-flow run --profile <name>): it clamps the resolved in-flight submission cap (--max-submitted> profile value > driver default) the same way, with the same warning style.
The per-rule sentence above applies only to max_threads/max_memory: a
cap on jobs is a total, so there is no "single rule exceeds it" condition —
the budget throttles how much runs at once instead.
CPU Oversubscription Warning#
Thread auto-detection is intentionally generous: the scheduling pool can
use every detected CPU. That is a performance good and an operational
problem — on a shared 64-core node, -j 8 with 8-thread rules occupies all
64 cores for hours, heats the package, and starves co-tenants. The engine
therefore warns before execution when the plan's worst-case thread
demand reaches 80 % of the detected CPUs (issue #1080).
The estimate is per scheduled parallel wave:
min(-j, wave width) × heaviest rule threads in that wave, with both
factors clamped by the effective pool ceiling. A strictly sequential
workflow (one rule per wave) is never reported as saturating even with a
large -j; a single 64-thread rule on a 64-core box is.
WARN planned thread demand 64 = up to 8 concurrent jobs × 8 threads/rule,
vs 64 detected CPUs (100%) — the run will occupy all cores; pass
--max-threads N or lower -j to leave headroom for the OS and co-tenants
The detected CPU count is cgroup-aware: num_cpus honours affinity/cpuset
and cgroup v1/v2 CPU quotas (container --cpus, systemd CPUQuota=,
cpu.max), so a 4-core host under a CPUQuota=200% scope is reported as
2 CPUs. The warning is computed against the effective pool ceiling a
local run will build (min(detected, [resource_budget] max_threads), or an
explicit --max-threads), so a cap that leaves headroom silences it; a cap
at or above the host still reports saturation. The estimate is per
scheduled parallel wave, restricted to the rules the run will actually
execute (-t/--module filtering included).
dry-run uses the same detected count (and the workflow budget) for its
suggested -j, so a preview shows a safe starting point before the run
does.
HPC vs Local Best Practices#
| Environment | Recommendation |
|---|---|
| Local workstation | Declare what you have (undersubscribe for stability) |
| Local server | Declare 80-90% of capacity |
| HPC cluster | Declare what scheduler allocates |
| Cloud | Minimize for cost efficiency |
Example: Same Workflow, Different Targets#
Because rule names must be unique within one workflow, keep the per-target variants in separate files (or give them distinct names):
# workflows/local.oxoflow — local development (undersubscribe)
[[rules]]
name = "align"
[rules.resources]
threads = 4
memory = "8G"
# workflows/hpc.oxoflow — HPC production (full allocation)
[[rules]]
name = "align"
[rules.resources]
threads = 32
memory = "128G"
partition = "highmem"
Disk Space#
Declare disk requirements for large intermediate files:
[[rules]]
name = "assembly"
shell = "assemble.sh"
[rules.resources]
disk = "500G" # Warn if <500GB available
oxo-flow emits warnings when disk requirements exceed available space but cannot enforce usage.
Troubleshooting#
Job Killed by OOM#
- Increase memory declaration
- Check actual memory usage with system monitoring
- Consider splitting input into smaller chunks
Timeout Killing Child Processes#
Timeout kills the rule's whole process tree on Unix systems: SIGTERM first, then SIGKILL after a 10-second grace window (deepest descendants first).
Solution: Use wrapper script that manages its own cleanup:
#!/bin/bash
cleanup() { kill $(jobs -p) 2>/dev/null; }
trap cleanup EXIT
your_long_running_command &
wait
Oversubscription Warnings#
If warnings appear but workflow succeeds, you can:
- Reduce declarations to match system
- Keep declarations and accept warnings
- Increase system resources
Optimizing with DAG Metrics#
Use oxo-flow graph to get structural insights before tuning resources:
The header shows key metrics:
┌──────────────────────────────────────────────────────────┐
│ Workflow DAG: 12 rules, 15 dependencies │
│ Depth: 5, Width: 4, Critical path: 5 steps │
└──────────────────────────────────────────────────────────┘
Interpreting metrics for resource planning#
| Metric | What it tells you | Resource implication |
|---|---|---|
| Depth | Number of sequential stages | Determines minimum wall-clock time; each level is a synchronization barrier |
| Width | Max rules at any single level | Your peak parallelism — set -j to at least this value |
| Critical path | Longest chain of dependencies | The bottleneck — focus optimization efforts here |
| Rules | Total workflow nodes | Overall scope; large counts may benefit from cluster backends |
Actionable guidance#
- Width = 1: Your DAG is fully sequential. Before adding more threads, consider whether you can split large rules into independent sub-tasks to create parallelism.
- Width >
-j: Some parallel rules will queue. Increase-jto match or exceed width for maximum throughput. - Critical path ≈ Depth: All levels are equally deep — no obvious bottleneck branch. If runtime is too high, optimize the slowest rule at each level.
- Critical path < Depth: Some branches are shallower. The critical path rules are your optimization priority — give them more threads/memory.
Example: tuning a diamond workflow#
Insights:
- Maximum parallelism is 2 (width) —
-j 2is sufficient - The critical path is
source → left → merge—rightis not on it - If
lefttakes 2 hours, that's your bottleneck regardless ofright's speed - Optimize
left(more threads, faster tool) before worrying aboutright