Skip to content

Run on a Cluster#

This guide explains how to execute oxo-flow workflows on HPC and cloud backends: the schedulers SLURM, PBS, SGE, and LSF, plus Kubernetes, AWS Batch, and GCP Batch for cloud runs.


Overview#

oxo-flow's cluster module translates each rule into a cluster job submission. Resource requirements declared in the .oxoflow file (threads, memory, gpu, time_limit) are mapped to the appropriate scheduler directives. The disk field is not mapped to any scheduler directive — it only produces a local warning during oxo-flow run.

There are two ways to reach a scheduler, and they suit different jobs:

What it does Use when
run --profile <NAME> Submits, tracks jobs to completion, and updates the checkpoint Normal execution — you want the workflow to run
cluster submit Writes job scripts and stops You want to inspect or hand-edit scripts before submitting

run --profile is the everyday path; it inherits everything run does — wildcard expansion, checkpoint/resume, --samples, --rerun, config-change invalidation. See Cluster submission for the [cluster] profile block. cluster submit remains the escape hatch and is documented below.

Environment wrapping is applied automatically — conda, mamba, docker, singularity/apptainer, pixi, venv, and modules environments are wrapped in the generated scripts. Each rule applies one backend (the first declared in the resolver order mamba → conda → pixi → docker → singularity → venv → modules), so declaring e.g. both singularity and modules fails loudly instead of silently dropping the module loads.


Supported Schedulers#

Scheduler Status Directive prefix
SLURM Supported #SBATCH
PBS/Torque Supported #PBS
SGE Supported #$
LSF Supported #BSUB
Kubernetes Supported (spec generation) k8s Job
AWS Batch Supported (spec generation) aws-batch
GCP Batch Supported (spec generation) gcp-batch

Declaring Resources#

Set resource requirements per rule:

[[rules]]
name = "align"
input = ["{sample}_R1.fastq.gz"]
output = ["aligned/{sample}.bam"]
environment = { singularity = "docker://biocontainers/bwa:0.7.17" }
shell = "bwa mem -t {threads} ref.fa {input} | samtools sort -o {output}"

[rules.resources]
threads = 16
memory = "32G"
time_limit = "24h"

Note on gpu: gpu = 0 means "no GPUs" — the engine omits the GPU directive entirely, exactly as if the field were absent. Only declare it when you need at least one GPU (gpu = 1 or higher).

Resource fields#

Field Type Example Description
threads Integer 16 Number of CPU cores
memory String "32G" RAM allocation
gpu Integer 1 Number of GPUs (simple count)
gpu_spec Table See below Detailed GPU specification
disk String "100G" Local disk space — local warning only, never emitted as a scheduler directive
time_limit String "24h" Wall-time limit

Memory and walltime conventions#

One engine-wide parser reads both memory and disk, so what you see in dry-run/run resource output is what the scheduler gets:

  • Bare numbers are megabytes (memory = "4096" = 4 GB) — on every backend. LSF's own default unit is KB, so oxo-flow converts the parsed figure ×1024 and submits an explicit KB number to -M / rusage[mem=…]; the same config cannot schedule 4 GB locally and 4 MB on the cluster.
  • Accepted forms: a bare number, or a single-letter K/M/G/T suffix ("4096", "8G", "16384M", "1T"). Fractional ("16.5G") and multi-letter-suffix ("4096MB", "2GB") figures parse to nothing engine-wide and are rejected at validate time — the same predicate check's E004 diagnostic applies, so a config that loads is a config the pool, the quota, and every scheduler render agree on. Because PBS mem= and SGE h_vmem= default to their own units (words/bytes), those two backends receive the figure with an explicit unit (mem=8192mb, h_vmem=16384M); SLURM's --mem and LSF's converted KB figure need no help (their defaults match or are converted).
  • time_limit accepts the suffix forms 90s, 30m, 24h, 2d and the scheduler-style colon forms 01:00:00, 24:00:00, 1-06:00:00 — one shared parser for the local executor and every backend, so the same value means the same limit everywhere. Bare numbers ("1500") are rejected at validate time: ambiguous between seconds and LSF minutes — write "1500s" or "25m".
  • A present but unparseable walltime is never silently replaced by a default: every backend renders it verbatim (with a warning in the log), so the scheduler rejects the script loudly instead of killing a long job at a hidden 1-hour limit. The local executor logs a warning and falls back to the global timeout.

GPU Specification#

For basic GPU requests, use the gpu field:

[rules.resources]
gpu = 2  # Request 2 GPUs

For advanced GPU configuration, use gpu_spec:

[rules.resources.gpu_spec]
count = 2
model = "a100"       # GPU model (optional — SLURM only)
memory_gb = 40       # Per-GPU memory in GB (optional — SLURM only)

count works on SLURM, PBS, and SGE; model and memory_gb only affect SLURM directives (LSF ignores gpu_spec entirely).

Different schedulers handle GPU requests differently:

Scheduler GPU Directive Notes
SLURM --gres=gpu:2 or --gres=gpu:a100:2:40g Full support for model and memory spec
PBS gpu=2 Basic count only; model selection varies by site
SGE -l gpu=2 Basic count only; requires queue configuration
LSF -gpu "num=2" Basic count only (rendered as a quoted num= run-options string)

SLURM Example#

oxo-flow generates SLURM job scripts automatically. For the align rule above with a sample group batch = ["S1", "S2", "S3"], the script generated for the first instance is:

#!/bin/bash
#SBATCH --chdir=/abs/path/to/workdir
#SBATCH --job-name=align_batch_S1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=1-00:00:00
#SBATCH --output=logs/align_batch_S1.out
#SBATCH --error=logs/align_batch_S1.err

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

Note: one script is generated per rule instance. Wildcards expand before the scripts are written, exactly as they do for run, so a 3-sample scatter produces align_batch_S1.sh, align_batch_S2.sh, and align_batch_S3.sh with concrete paths in each. The instance names match the ones dry-run plans, so a script maps back to a planned rule by name. The sh -c '…' sh '<command>' tail is the bash shim described under Environment Wrapping, and singularity is substituted when apptainer is absent.

Note the workdir pinning at the top of each script (--chdir / cd / -wd / -cwd): without it SLURM starts a job in the submission directory, PBS in $HOME, and SGE in a queue-configured directory, so a rule's relative output paths would silently land outside the run directory. The examples below show the pin for each backend.


PBS Example#

#!/bin/bash
#PBS -N align_batch_S1
#PBS -l nodes=1:ppn=16,mem=32768mb,walltime=1-00:00:00
#PBS -o logs/align_batch_S1.out
#PBS -e logs/align_batch_S1.err
cd /abs/path/to/workdir || exit 1

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

SGE Example#

#!/bin/bash
#$ -wd /abs/path/to/workdir
#$ -N align_batch_S1
#$ -pe smp 16
#$ -l h_vmem=32768M
#$ -l h_rt=1-00:00:00
#$ -o logs/align_batch_S1.out
#$ -e logs/align_batch_S1.err

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

LSF Example#

#!/bin/bash
#BSUB -cwd /abs/path/to/workdir
#BSUB -J align_batch_S1
#BSUB -n 16
#BSUB -R 'rusage[mem=33554432] span[hosts=1]'
#BSUB -M 33554432
#BSUB -W 24:00
#BSUB -o logs/align_batch_S1.out
#BSUB -e logs/align_batch_S1.err

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

The memory figure is explicit KB, and span[hosts=1] rides in the -R string — why both are emitted that way is explained once in the cluster command's LSF script section. -W 24:00 is the parsed form of the rule's time_limit = "24h" — LSF takes [HH:]MM. The workdir pin is explicit here too: the generated script carries #BSUB -cwd, the same pinned-directory directive the other backends get.


Environment Wrapping#

When generating cluster scripts, oxo-flow automatically wraps commands through the environment resolver (rules without an environment run the command directly). This table is the single source for the per-backend wrapping forms — other pages link here instead of restating them:

Backend Wrapping
Conda conda run --no-capture-output -n <env> bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; <command>'
Mamba <mamba\|micromamba\|conda> run -n <env> bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; <command>' (--no-capture-output added when the detected binary is conda)
Docker docker run --rm --user $(id -u):$(id -g) -v <workdir>:<workdir> -w <workdir> <image> sh -c '<shim>' sh '<command>'
Singularity / Apptainer <apptainer\|singularity> exec --bind <workdir>:<workdir> <image> sh -c '<shim>' sh '<command>'
Pixi pixi run --manifest-path <pixi.toml> bash -c '<command>'
Venv source <venv>/bin/activate && <command>
Modules module-init block, then module load <mod1> <mod2> && <command>

Three details worth knowing:

  • Container paths are absolute. docker -v and singularity --bind reject relative sources, and a scheduler may start a job from any directory, so oxo-flow renders the resolved workdir. The rule's command is handed to a shim inside the container — if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi — because the image's entrypoint shell runs first, and multi-line or pipefail-using scripts need bash, which not every image ships.
  • Paths outside the workdir are bound read-only (Docker backend). A [config] reference genome or index living elsewhere on the shared filesystem is appended to the Docker mount list automatically (-v /data/ref:/data/ref:ro), so the rule can read it from inside the container. The Singularity/Apptainer backend does not add these mounts — if your site needs them, keep inputs inside the workdir or wrap the command yourself. (There is no [cluster.container] sub-table: the [cluster] profile block accepts exactly eight keys — account, backend, extra_args, max_array_size, max_submitted, partition, poll_interval, walltime.)
  • apptainer wins when both it and singularity are installed, so clusters that renamed the binary keep working without configuration.

Environment Examples#

Conda with GPU for deep learning:

[[rules]]
name = "train_model"
input = ["data/train.h5"]
output = ["models/trained.pt"]
environment = { conda = "envs/pytorch.yaml" }
shell = "python train.py --input {input} --output {output}"

[rules.resources]
threads = 8
memory = "64G"
gpu = 2
time_limit = "24h"

The gpu field controls the scheduler allocation only (e.g. #SBATCH --gres=gpu:2) — there is no {resources.gpu} shell placeholder.

Singularity (common on HPC):

[[rules]]
name = "variant_call"
input = ["aligned/{sample}.bam"]
output = ["variants/{sample}.vcf"]
environment = { singularity = "docker://broadinstitute/gatk:4.4.0.0" }
shell = "gatk HaplotypeCaller -I {input} -O {output}"

[rules.resources]
threads = 16
memory = "32G"

Only one backend applies per rule, so combining e.g. singularity with modules in the same rule is rejected outright (the module loads would be silently dropped) — use the module-based example below for module-only setups.

Pixi for reproducible environments:

[[rules]]
name = "qc_check"
input = ["{sample}.fastq.gz"]
output = ["qc/{sample}_fastqc.html"]
environment = { pixi = "envs/pixi.toml" }  # the manifest FILE, not an env name
shell = "fastqc -t {threads} -o qc/ {input}"

[rules.resources]
threads = 4

Pure Module-based (traditional HPC):

[[rules]]
name = "align"
input = ["reads/{sample}.fq"]
output = ["aligned/{sample}.bam"]
environment = { modules = ["bwa/0.7.17", "samtools/1.17", "gcc/11"] }
shell = "bwa mem -t {threads} ref.fa {input} | samtools sort -o {output}"

[rules.resources]
threads = 32
memory = "64G"

Pre-build environments on cluster nodes

Ensure your conda environments, docker images, or singularity containers are available on all cluster nodes before submitting jobs. Use --skip-env-setup when environments are pre-built.


Resource Enforcement#

Local Execution#

When running locally (oxo-flow run), resource constraints are enforced:

  • Check: Before execution, verify resources are available
  • Reserve: Reserve resources before starting the job
  • Release: Release resources after completion (or on failure/timeout)
# Limit to 16 threads and 32GB memory for local execution
oxo-flow run pipeline.oxoflow --max-threads 16 --max-memory 32768

Cluster Execution#

On clusters, the scheduler enforces resources based on the generated directives. oxo-flow does not manage resources during cluster execution — the scheduler handles that.


Best Practices#

Use Singularity on clusters

Most HPC clusters do not allow Docker. Use Singularity instead — oxo-flow handles the conversion automatically when you specify singularity = "docker://...".

Set realistic time limits

Generous wall-time limits prevent premature job termination but may lower scheduling priority. Profile your jobs first.

Failure handling on the cluster path

--keep-going / -k is honored on the cluster path (issue #1093). By default a required = true rule's failure cancels every in-flight job (scancel/qdel/bkill) and the run exits non-zero. With -k the failure is still recorded and still fails the run, but jobs already in flight run to completion and the driver keeps dispatching ready independent instances until the queue drains; dependents of the failed rule stay blocked (never submitted). Interrupts (Ctrl-C / SIGTERM / SIGHUP) and driver-internal errors always cancel every in-flight job regardless of -k. A required = false failure never cancels either way — mark salvageable steps required = false (best-effort QC, optional per-sample stages) when their failure should leave the exit code at 0: the failure is recorded, the run exits 0 with the failures listed, and resume / --resume-failed re-queues what failed.

Aborted in-flight siblings are incomplete, not failed

When a run aborts, rules still in flight are killed and recorded as incomplete (neither completed nor failed) — resume re-runs them from their checkpoint, so no output is silently trusted from a killed job. A run stopped by SIGINT/SIGTERM/SIGHUP exits 128 + signum (130/143/129) and --json reports status "failed", even when every interrupted rule is required = false — an interrupt is never reported as a completed run.

Check resource availability

Use sinfo (SLURM), pbsnodes (PBS), or qhost (SGE) to verify available resources before submitting.

Cache environment setup

Use --cache-dir to persist environment setup state across runs for faster startup.


Browser-Based Cluster Monitoring#

The web UI can also drive cluster execution without a terminal: register an SSH cluster connection on the Clusters page (or the platform config file's [[clusters]] section), then pick Execute on cluster in the Run dialog. The workdir is staged to the remote host, oxo-flow run is launched there detached, and logs/files/report views work exactly as for local runs once the results are pulled back. The remote host needs the oxo-flow CLI on PATH, tar, and non-interactive SSH access (key auth, BatchMode).

Cluster connection management is admin-only outside personal mode. Details: Cluster Connections & Remote Execution in the REST API reference, and Deploy modes for the HPC serve mode.

Monitoring Jobs#

After submission, use your cluster's native tools:

# SLURM
squeue -u $USER

# PBS
qstat -u $USER

# SGE
qstat

# LSF
bjobs

Or use oxo-flow's status command with a checkpoint file:

oxo-flow status .oxo-flow/checkpoint.json

oxo-flow cluster status answers two questions. For a deployment that runs through run --profile <cluster>, it reads the newest run directory (.oxo-flow/runs/latest, or --run-dir <DIR>) and prints one row per rule instance — mapping scheduler ids back through jobs/<instance>/job.id — plus per-state counts:

$ oxo-flow cluster status
Cluster: run .oxo-flow/runs/latest
  completed 12   running 3    pending 47      failed 1
  instance            job      state    elapsed    exit
  align_all_9c4e12  1400_1   running  2:24:11    -

--json emits the same document for dashboards. For logs, oxo-flow cluster logs <id-or-instance> reads the engine-captured logs/<instance>.out/.err before the accounting store, so it works on sites without slurmdbd; --tail N trims the output.

With --all or no run directory it falls back to the raw scheduler view. With no ids it lists every job the scheduler currently reports for your user — the natural check right after a plain cluster submit:

$ oxo-flow cluster status --all
Cluster: Executing 'squeue -u alice --noheader -o %i|%T'...
Cluster: 2 queued jobs
  12345: running
  12346: pending

With ids (the ones submit/run printed) it answers exactly those, settling finished jobs from the scheduler's accounting store:

oxo-flow cluster status 12345 12346
oxo-flow cluster cancel 12345

Every cluster action resolves its backend the same way: the --backend / -b flag if given, then $OXO_FLOW_CLUSTER_BACKEND, then slurm. On a SLURM site you never type -b; on a PBS site set the environment variable once (export OXO_FLOW_CLUSTER_BACKEND=pbs). An explicit-but-typoed value is still rejected outright (unknown cluster backend '…' — expected slurm, pbs, sge, lsf, kubernetes (k8s), aws-batch, or gcp-batch) — the default never swallows a bad name.


Driver Defaults and the [cluster] Profile Block#

run --profile submits and tracks jobs through the backend driver, whose built-in defaults (max_submitted 50, max_array_size 1001, poll_interval 5s) are specified once in the DriverConfig table; the profile's [cluster] block overrides those three knobs. The full annotated TOML example (every key, the driver default each omission falls back to, and where a rule's own time_limit wins) lives once in Cluster submission — that copy is authoritative.

extra_args entries are emitted as scheduler directives verbatim and unvalidated — a typo reaches the scheduler as written (the same applies to cluster submit --extra-arg). Cluster CLI actions resolve --backend as flag > $OXO_FLOW_CLUSTER_BACKEND > slurm (see Monitoring Jobs); the profile block's backend key remains required and is validated the same strict way.

Cluster flag surface#

run --profile does not consume every run flag. The flags below are parsed but have no effect on the cluster path — each of the first three prints a warning and the run proceeds (-j is dropped silently):

run flag Cluster-path behavior
--retry Warned and dropped — there is no engine-side retry on the cluster path. Failed rules are recorded in the checkpoint and re-queued by resume / --resume-failed.
--timeout Warned and dropped — per-job walltime comes from the profile's [cluster] walltime key or the rule's own time_limit; the --walltime flag exists only on cluster submit.
--max-threads / --max-memory Warned and dropped — per-rule [resources] and the [cluster] profile drive scheduler limits; [resource_budget] performs the pre-submit check.
-j / --jobs Silently inapplicable — cluster concurrency is governed by max_submitted (the profile key or --max-submitted), clamped by [resource_budget] max_jobs.
--skip-env-setup, --ai-recover Accepted for command-line parity (warned, issue #136): environments are prepared inside each job script, and AI recovery does not run on the cluster path.

--skip-ref-build is accepted on the cluster path and is a no-op there by construction: the reference-build step runs only on the local execution path, and run --profile never builds references itself.

--provenance is honored on the cluster path (issue #1093): each rule's expanded outputs are hashed when the job settles and recorded into the checkpoint's checksums map, exactly like a local run — provenance verify and the freshness gate read the same entries from either execution path. The per-rule checksum = "sha256" | "md5" declaration works there too. A run that records nothing still warns, with the same wording as the local completion path.

--keep-going / -k is honored on the cluster path (issue #1093). It gates exactly one decision — what a required = true rule's failure does to in-flight work:

Mode Required rule fails required = false rule fails
default cancels every in-flight job, then exits non-zero never cancels; failure recorded, dependents blocked, run exits 0
-k / --keep-going in-flight jobs finish, ready independent instances keep dispatching until the queue drains, the failed rule's dependents stay blocked — then exits non-zero (a required rule failed) same as default

Interrupts (Ctrl-C / SIGTERM / SIGHUP) and driver-internal errors always cancel every in-flight job regardless of -k; the flag never softens the verdict — a run with a required failure exits non-zero either way.


Scheduling Fairness#

Cluster submission honors the same fairness contract as local execution:

  • Priority is honored: each round, ready rules are submitted in effective-priority order — declared priority plus priority aging (every round a ready rule spends beyond the max_submitted cap gains +1). A producer parked at the cap therefore outranks fresh high-priority rules after priority gap rounds — priority-inversion starvation cannot persist.
  • In-flight jobs are cancelled on a required failure by default: when a required rule fails, the driver cancels every job already submitted (scancel/qdel/bkill) before the run exits — no orphaned cluster jobs. With -k / --keep-going those jobs instead finish, ready independent instances keep dispatching until the queue drains, the failed rule's dependents stay blocked, and the run still exits non-zero (see Cluster flag surface).
  • Waits inside the scheduler queue are the scheduler's own ordering and are visible via squeue/qstat/bjobs (job ids are recorded per rule under the run directory). The local 60-second "waiting for resources:" diagnostics have no cluster counterpart by design — the pool lives on the scheduler, not in the engine.

Per-sample dispatch order#

Equal-priority ties are broken by instance name by default. Expanded instance names are <rule>_<group>_<sample>, so that tie-break groups every instance of one rule together: a per-sample pipeline submits all align instances, then all qc instances, then all merge instances. The first merge cannot run until the slowest sample's producers finish, and no per-sample result exists until the end of the run.

Set [engine] dispatch_order = "sample" to order ties by the sample binding first, then by rule declaration order:

[engine]
dispatch_order = "sample"

[[sample_groups]]
name = "cohort"
samples = ["S1", "S2"]

[[rules]]
name = "align"
output = ["out/{sample}/align.bam"]
shell = "align {sample} > out/{sample}/align.bam"

[[rules]]
name = "qc"
output = ["out/{sample}/qc.txt"]
shell = "qc {sample} > out/{sample}/qc.txt"

[[rules]]
name = "merge"
input = ["out/{sample}/align.bam", "out/{sample}/qc.txt"]
output = ["out/{sample}/merge.txt"]
shell = "merge out/{sample}/align.bam out/{sample}/qc.txt > out/{sample}/merge.txt"

Both producers read the same sample independently, and merge joins them. Under the default "rule" mode the ready list is align_S1, align_S2, qc_S1, qc_S2, …; under "sample" it is align_S1, qc_S1, align_S2, qc_S2, …. Priority aging remains the primary key: a sample-2 producer that has waited a round accrues +1 and can run before sample 1's newly-ready merge_S1. Sample mode therefore keeps each sample's stages adjacent among equal-priority ties and reduces phase-wise batching, but per-sample drain is best-effort, not guaranteed. Total work and the max_submitted cap are unchanged.

The setting is only a tie-break over currently ready rules: it never reorders dependencies, and it does not delay sample 2's independent producers behind sample 1's unfinished stages. priority remains the primary key — a rule whose declared (+ aging) priority is higher still jumps the queue, which is useful for a genuinely urgent one-off rule. Use priority for per-rule urgency and dispatch_order = "sample" for per-sample pipeline flow.

The same ordering is honored by local execution and by the cluster driver. The default "rule" mode is byte-identical to previous releases.


See Also#