Skip to content

Run on a Cluster#

This guide explains how to execute oxo-flow workflows on HPC and cloud backends: the schedulers SLURM, PBS, SGE, and LSF, plus Kubernetes, AWS Batch, and GCP Batch for cloud runs.


Overview#

oxo-flow's cluster module translates each rule into a cluster job submission. Resource requirements declared in the .oxoflow file (threads, memory, gpu, time_limit) are mapped to the appropriate scheduler directives. The disk field is not mapped to any scheduler directive — it only produces a local warning during oxo-flow run.

There are two ways to reach a scheduler, and they suit different jobs:

What it does Use when
run --profile <NAME> Submits, tracks jobs to completion, and updates the checkpoint Normal execution — you want the workflow to run
cluster submit Writes job scripts and stops You want to inspect or hand-edit scripts before submitting

run --profile is the everyday path; it inherits everything run does — wildcard expansion, checkpoint/resume, --samples, --rerun, config-change invalidation. See Cluster submission for the [cluster] profile block. cluster submit remains the escape hatch and is documented below.

Environment wrapping is applied automatically — conda, mamba, docker, singularity/apptainer, pixi, venv, and modules environments are wrapped in the generated scripts. Each rule applies one backend (the first declared in the resolver order mamba → conda → pixi → docker → singularity → venv → modules), so declaring e.g. both singularity and modules fails loudly instead of silently dropping the module loads.


Supported Schedulers#

Scheduler Status Maturity Directive prefix
SLURM Supported Production deployments + CI mock scheduler (operational evidence; not a per-commit live campaign) #SBATCH
PBS/Torque Supported Real-scheduler matrix, OpenPBS 23.06 (2026-09-22/23 campaign; not re-run per commit) #PBS
SGE Supported Real-scheduler matrix, SoGE 8.1.9 (2026-09-22/23 campaign; not re-run per commit) #$
LSF Supported Experimental — no live campaign; directives/status semantics cross-checked against IBM's bsub reference #BSUB
Kubernetes Supported (spec generation) Prototype — CLI wrapper (kubectl), mock/wire tests only; live campaign pending (CL-05 / #1158) k8s Job
AWS Batch Supported (spec generation) Prototype — CLI wrapper (aws), mock/wire tests only; live campaign pending (CL-05 / #1158) aws-batch
GCP Batch Supported (spec generation) Prototype — CLI wrapper (gcloud), mock/wire tests only; live campaign pending (CL-05 / #1158) gcp-batch

Per-element outcomes. Array elements settle on their own scheduler record whenever the backend exposes one: SLURM squeue element ids, Kubernetes Indexed Job completion indexes (status.completedIndexes / status.failedIndexes), AWS Batch child job ids ({parent}:{arrayIndex}), and GCP Batch task indexes (gcloud batch tasks list). One element failing never turns its successful siblings into failures, and a missing accounting record leaves the exit code unknown rather than reporting 0. PBS/LSF/SGE are polled by the array base id in this adapter; their per-instance records name that aggregate provenance in skip_reason and carry no base-level exit code or resource numbers, so an array-level verdict is never presented as an independently observed element outcome.

None of the cloud backends is qualified by a live campaign: their maturity label stays Prototype until the per-backend submit/queue/run/cancel/fail/retry/artifact campaign tracked by CL-05 (#1158) exists.


Declaring Resources#

Set resource requirements per rule:

[[rules]]
name = "align"
input = ["{sample}_R1.fastq.gz"]
output = ["aligned/{sample}.bam"]
environment = { singularity = "docker://biocontainers/bwa:0.7.17" }
shell = "bwa mem -t {threads} ref.fa {input} | samtools sort -o {output}"

[rules.resources]
threads = 16
memory = "32G"
time_limit = "24h"

Note on gpu: gpu = 0 means "no GPUs" — the engine omits the GPU directive entirely, exactly as if the field were absent. Only declare it when you need at least one GPU (gpu = 1 or higher).

Resource fields#

Field Type Example Description
threads Integer 16 Number of CPU cores
memory String "32G" RAM allocation
gpu Integer 1 Number of GPUs (simple count)
gpu_spec Table See below Detailed GPU specification
disk String "100G" Local disk space — local warning only, never emitted as a scheduler directive
time_limit String "24h" Wall-time limit

Memory and walltime conventions#

One engine-wide parser reads both memory and disk, so what you see in dry-run/run resource output is what the scheduler gets:

  • Bare numbers are megabytes (memory = "4096" = 4 GB) — on every backend. LSF's own default unit is KB, so oxo-flow converts the parsed figure ×1024 and submits an explicit KB number to -M / rusage[mem=…]; the same config cannot schedule 4 GB locally and 4 MB on the cluster.
  • Accepted forms: a bare number, or a single-letter K/M/G/T suffix ("4096", "8G", "16384M", "1T"). Fractional ("16.5G") and multi-letter-suffix ("4096MB", "2GB") figures parse to nothing engine-wide and are rejected at validate time — the same predicate check's E004 diagnostic applies, so a config that loads is a config the pool, the quota, and every scheduler render agree on. Because PBS mem= and SGE h_vmem= default to their own units (words/bytes), those two backends receive the figure with an explicit unit (mem=8192mb, h_vmem=16384M); SLURM's --mem and LSF's converted KB figure need no help (their defaults match or are converted).
  • time_limit accepts the suffix forms 90s, 30m, 24h, 2d and the scheduler-style colon forms 01:00:00, 24:00:00, 1-06:00:00 — one shared parser for the local executor and every backend, so the same value means the same limit everywhere. Bare numbers ("1500") are rejected at validate time: ambiguous between seconds and LSF minutes — write "1500s" or "25m".
  • A present but unparseable walltime is never silently replaced by a default: every backend renders it verbatim (with a warning in the log), so the scheduler rejects the script loudly instead of killing a long job at a hidden 1-hour limit. The local executor logs a warning and falls back to the global timeout.

GPU Specification#

For basic GPU requests, use the gpu field:

[rules.resources]
gpu = 2  # Request 2 GPUs

For advanced GPU configuration, use gpu_spec:

[rules.resources.gpu_spec]
count = 2
model = "a100"       # GPU model (optional — SLURM only)
memory_gb = 40       # Per-GPU memory in GB (optional — SLURM only)

count works on SLURM, PBS, and SGE; model and memory_gb only affect SLURM directives (LSF ignores gpu_spec entirely).

Different schedulers handle GPU requests differently:

Scheduler GPU Directive Notes
SLURM --gres=gpu:2 or --gres=gpu:a100:2:40g Full support for model and memory spec
PBS gpu=2 Basic count only; model selection varies by site
SGE -l gpu=2 Basic count only; requires queue configuration
LSF -gpu "num=2" Basic count only (rendered as a quoted num= run-options string)

SLURM Example#

oxo-flow generates SLURM job scripts automatically. For the align rule above with a sample group batch = ["S1", "S2", "S3"], the script generated for the first instance is:

#!/bin/bash
#SBATCH --chdir=/abs/path/to/workdir
#SBATCH --job-name=align_batch_S1
#SBATCH --cpus-per-task=16
#SBATCH --mem=32G
#SBATCH --time=1-00:00:00
#SBATCH --output=logs/align_batch_S1.out
#SBATCH --error=logs/align_batch_S1.err

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

Note: one script is generated per rule instance. Wildcards expand before the scripts are written, exactly as they do for run, so a 3-sample scatter produces align_batch_S1.sh, align_batch_S2.sh, and align_batch_S3.sh with concrete paths in each. The instance names match the ones dry-run plans, so a script maps back to a planned rule by name. The sh -c '…' sh '<command>' tail is the bash shim described under Environment Wrapping, and singularity is substituted when apptainer is absent.

Note the workdir pinning at the top of each script (--chdir / cd / -wd / -cwd): without it SLURM starts a job in the submission directory, PBS in $HOME, and SGE in a queue-configured directory, so a rule's relative output paths would silently land outside the run directory. The examples below show the pin for each backend.


PBS Example#

#!/bin/bash
#PBS -N align_batch_S1
#PBS -l nodes=1:ppn=16,mem=32768mb,walltime=1-00:00:00
#PBS -o logs/align_batch_S1.out
#PBS -e logs/align_batch_S1.err
cd /abs/path/to/workdir || exit 1

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

SGE Example#

#!/bin/bash
#$ -wd /abs/path/to/workdir
#$ -N align_batch_S1
#$ -pe smp 16
#$ -l h_vmem=32768M
#$ -l h_rt=1-00:00:00
#$ -o logs/align_batch_S1.out
#$ -e logs/align_batch_S1.err

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

LSF Example#

#!/bin/bash
#BSUB -cwd /abs/path/to/workdir
#BSUB -J align_batch_S1
#BSUB -n 16
#BSUB -R 'rusage[mem=33554432] span[hosts=1]'
#BSUB -M 33554432
#BSUB -W 24:00
#BSUB -o logs/align_batch_S1.out
#BSUB -e logs/align_batch_S1.err

set -e

mkdir -p logs
apptainer exec --bind /abs/path/to/workdir:/abs/path/to/workdir docker://biocontainers/bwa:0.7.17 sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi' sh 'bwa mem -t 16 ref.fa S1_R1.fastq.gz | samtools sort -o aligned/S1.bam'

The memory figure is explicit KB, and span[hosts=1] rides in the -R string — why both are emitted that way is explained once in the cluster command's LSF script section. -W 24:00 is the parsed form of the rule's time_limit = "24h" — LSF takes [HH:]MM. The workdir pin is explicit here too: the generated script carries #BSUB -cwd, the same pinned-directory directive the other backends get.


Environment Wrapping#

When generating cluster scripts, oxo-flow automatically wraps commands through the environment resolver (rules without an environment run the command directly). This table is the single source for the per-backend wrapping forms — other pages link here instead of restating them:

Backend Wrapping
Conda conda run --no-capture-output -n <env> bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; <command>'
Mamba <mamba\|micromamba\|conda> run -n <env> bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; <command>' (--no-capture-output added when the detected binary is conda)
Docker docker run --rm --user $(id -u):$(id -g) -v <workdir>:<workdir> -w <workdir> <image> sh -c '<shim>' sh '<command>'
Singularity / Apptainer <apptainer\|singularity> exec --bind <workdir>:<workdir> <image> sh -c '<shim>' sh '<command>'
Pixi pixi run --manifest-path <pixi.toml> bash -c '<command>'
Venv source <venv>/bin/activate && <command>
Modules module-init block, then module load <mod1> <mod2> && <command>

Three details worth knowing:

  • Container paths are absolute. docker -v and singularity --bind reject relative sources, and a scheduler may start a job from any directory, so oxo-flow renders the resolved workdir. The rule's command is handed to a shim inside the container — if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else exec sh -c "$1"; fi — because the image's entrypoint shell runs first, and multi-line or pipefail-using scripts need bash, which not every image ships.
  • Paths outside the workdir are bound read-only (Docker backend). A [config] reference genome or index living elsewhere on the shared filesystem is appended to the Docker mount list automatically (-v /data/ref:/data/ref:ro), so the rule can read it from inside the container. The Singularity/Apptainer backend does not add these mounts — if your site needs them, keep inputs inside the workdir or wrap the command yourself. (There is no [cluster.container] sub-table: the [cluster] profile block accepts exactly eight keys — account, backend, extra_args, max_array_size, max_submitted, partition, poll_interval, walltime.)
  • apptainer wins when both it and singularity are installed, so clusters that renamed the binary keep working without configuration.

Environment Examples#

Conda with GPU for deep learning:

[[rules]]
name = "train_model"
input = ["data/train.h5"]
output = ["models/trained.pt"]
environment = { conda = "envs/pytorch.yaml" }
shell = "python train.py --input {input} --output {output}"

[rules.resources]
threads = 8
memory = "64G"
gpu = 2
time_limit = "24h"

The gpu field controls the scheduler allocation only (e.g. #SBATCH --gres=gpu:2) — there is no {resources.gpu} shell placeholder.

Singularity (common on HPC):

[[rules]]
name = "variant_call"
input = ["aligned/{sample}.bam"]
output = ["variants/{sample}.vcf"]
environment = { singularity = "docker://broadinstitute/gatk:4.4.0.0" }
shell = "gatk HaplotypeCaller -I {input} -O {output}"

[rules.resources]
threads = 16
memory = "32G"

Only one backend applies per rule, so combining e.g. singularity with modules in the same rule is rejected outright (the module loads would be silently dropped) — use the module-based example below for module-only setups.

Pixi for reproducible environments:

[[rules]]
name = "qc_check"
input = ["{sample}.fastq.gz"]
output = ["qc/{sample}_fastqc.html"]
environment = { pixi = "envs/pixi.toml" }  # the manifest FILE, not an env name
shell = "fastqc -t {threads} -o qc/ {input}"

[rules.resources]
threads = 4

Pure Module-based (traditional HPC):

[[rules]]
name = "align"
input = ["reads/{sample}.fq"]
output = ["aligned/{sample}.bam"]
environment = { modules = ["bwa/0.7.17", "samtools/1.17", "gcc/11"] }
shell = "bwa mem -t {threads} ref.fa {input} | samtools sort -o {output}"

[rules.resources]
threads = 32
memory = "64G"

Pre-build environments on cluster nodes

Ensure your conda environments, docker images, or singularity containers are available on all cluster nodes before submitting jobs. Use --skip-env-setup when environments are pre-built.


Resource Enforcement#

Local Execution#

When running locally (oxo-flow run), resource constraints are enforced:

  • Check: Before execution, verify resources are available
  • Reserve: Reserve resources before starting the job
  • Release: Release resources after completion (or on failure/timeout)
# Limit to 16 threads and 32GB memory for local execution
oxo-flow run pipeline.oxoflow --max-threads 16 --max-memory 32768

Cluster Execution#

On clusters, the scheduler enforces resources based on the generated directives. oxo-flow does not manage resources during cluster execution — the scheduler handles that.


Best Practices#

Use Singularity on clusters

Most HPC clusters do not allow Docker. Use Singularity instead — oxo-flow handles the conversion automatically when you specify singularity = "docker://...".

Set realistic time limits

Generous wall-time limits prevent premature job termination but may lower scheduling priority. Profile your jobs first.

Failure handling on the cluster path

--keep-going / -k is honored on the cluster path (issue #1093). By default a required = true rule's failure cancels every in-flight job (scancel/qdel/bkill) and the run exits non-zero. With -k the failure is still recorded and still fails the run, but jobs already in flight run to completion and the driver keeps dispatching ready independent instances until the queue drains; dependents of the failed rule stay blocked (never submitted). Interrupts (Ctrl-C / SIGTERM / SIGHUP) and driver-internal errors always cancel every in-flight job regardless of -k. A required = false failure never cancels either way — mark salvageable steps required = false (best-effort QC, optional per-sample stages) when their failure should leave the exit code at 0: the failure is recorded, the run exits 0 with the failures listed, and re-running with resume (the standalone command that takes a checkpoint path) or run --resume-failed re-queues what failed.

Submission failure semantics

A job the scheduler has accepted is never dropped silently. Once sbatch/qsub/bsub-style submission returns a job id, the driver registers the job in memory and then writes its local records (jobs/<rule>/job.id, status.json, and index.json for arrays). If any of those writes fails — full disk, permissions, a directory in place of a file — the driver cancels every accepted scheduler job and fails the run with the scheduler id(s) named in the error. When the cancel itself cannot be confirmed, the error says so explicitly (failed to cancel N scheduler job(s) — they may still be running: <ids>) and never reports those jobs as cancelled; cancel them manually before re-running. Array submissions follow the same rule: elements are registered before the bookkeeping writes, and a failed registration cancels the whole array through its base id.

One window is outside the engine's reach: if the scheduler accepts a job but prints no parseable id, there is no handle to cancel. The run fails with the scheduler's own output quoted in the error so the job can be located and stopped by hand.

Aborted in-flight siblings are incomplete, not failed

When a run aborts, rules still in flight are killed and recorded as incomplete (neither completed nor failed) — resume re-runs them from their checkpoint, so no output is silently trusted from a killed job. A run stopped by SIGINT/SIGTERM/SIGHUP exits 128 + signum (130/143/129) and --json reports status "failed", even when every interrupted rule is required = false — an interrupt is never reported as a completed run.

What environment a cloud job sees

Cloud jobs (Kubernetes, AWS Batch, GCP Batch) never inherit the submitting host's environment. The generated manifest/spec carries the rendered rule command (and, when sensitive values are routed, a secretKeyRef env block referencing the operator's Secret), so the job process sees the container image's own ENV (or the VM image's environment for a GCP Batch script runnable) plus what the engine puts in the command: the rule's declared envvars (rendered as export K='V') and the OXO_FLOW_SENSITIVE_<n> references for sensitive = true values. Ambient variables from the submit shell — API tokens, cloud credentials, proxy settings — are never forwarded, and a regression test per backend pins that boundary (issue #1171). To get a host-side value onto the compute side, declare it in envvars or deliver it through the backend's secret channel; OXO_FLOW_ENV_PASSTHROUGH does not apply to cloud jobs because there is no ambient environment to re-admit. HPC job scripts (SLURM/PBS/SGE/LSF) are different: they do run a shell on nodes that the scheduler seeded from the submit environment, so they apply the env -i allowlist described in the run reference and in workflow format.

sensitive = true values need an operator-side secret on cloud backends

Cloud runs (run --profile <NAME> with [cluster] backend = "kubernetes", "aws-batch", or "gcp-batch", and cluster submit -b with the same backends) route every sensitive = true config value out of the job body before the manifest is generated: the body carries only an OXO_FLOW_SENSITIVE_<n> reference. Names are assigned longest-value-first (then lexically; values under 4 characters stay on the command line), and the same names are used across all rules and array elements. Supply the value through the backend's own secret channel — a Kubernetes Secret named oxoflow-secrets (one key per variable name), the AWS Batch job definition's containerProperties.secrets, or GCP Batch taskSpec.environment.secretVariables (the engine emits those entries for you; point each name at its Secret Manager path with OXO_FLOW_SECRET_REFS, a NAME=REFERENCE map — see workflow-format: [config]) — or the reference expands empty and the rule fails. HPC scheduler scripts (SLURM/PBS/SGE/LSF) route the same way: the driver writes a private 0o600 env file beside the run holding only the references each rule uses, and the generated script sources it inside the env -i wrapper, so the scheduler-visible script body carries $OXO_FLOW_SENSITIVE_<n> and never the value.

Check resource availability

Use sinfo (SLURM), pbsnodes (PBS), or qhost (SGE) to verify available resources before submitting.

Cache environment setup

Use --cache-dir to persist environment setup state across runs for faster startup.


Browser-Based Cluster Monitoring#

The web UI can also drive cluster execution without a terminal: register an SSH cluster connection on the Clusters page (or the platform config file's [[clusters]] section), then pick Execute on cluster in the Run dialog. The workdir is staged to the remote host, oxo-flow run is launched there detached, and logs/files/report views work exactly as for local runs once the results are pulled back. The remote host needs the oxo-flow CLI on PATH, tar, and non-interactive SSH access (key auth, BatchMode).

Result pulls are validated before a run can finalize

The pulled result stream is parsed by oxo-flow's own archive reader, not the host tar. A member whose path is absolute or contains .., a symlink or hard link whose target is absolute or contains any .. component (even one that would "cancel out" lexically — it can escape through a symlinked path segment; extraction targets are created empty or from regular files only, so no pre-existing symlink can be traversed), or a member type other than file/directory/link rejects the whole extraction and fails the run with the offending member named — nothing is written outside the local workdir. After extraction every member is re-checked on disk. The wrapper appends a run-specific completion record to the stream after the result tree, so a transfer that stops early cannot carry it: a truncated pull fails the run instead of reporting a partial tree as complete, and execution.log plus .exit-code are required as well. The failure is recorded in the run's artifacts_incomplete note and the run is never reported as completed. The pull verifies that the complete tree arrived — it does not re-derive the expanded output list locally; declared outputs are verified by the engine on the remote host before the run exits 0, and a non-zero remote exit always fails the run. Result trees that intentionally leave symlinks pointing outside the workdir (for example an externally rooted OXO_FLOW_ENV_STORE link, or any link target containing ..) are rejected by this check — pulled results must be self-contained.

Cluster connection management is admin-only outside personal mode. Details: Cluster Connections & Remote Execution in the REST API reference, and Deploy modes for the HPC serve mode.

Monitoring Jobs#

After submission, use your cluster's native tools:

# SLURM
squeue -u $USER

# PBS
qstat -u $USER

# SGE
qstat

# LSF
bjobs

Or use oxo-flow's status command with a checkpoint file:

oxo-flow status .oxo-flow/checkpoint.json

oxo-flow cluster status answers two questions. For a deployment that runs through run --profile <cluster>, it reads the newest run directory (.oxo-flow/runs/latest, or --run-dir <DIR>) and prints one row per rule instance — mapping scheduler ids back through jobs/<instance>/job.id — plus per-state counts:

$ oxo-flow cluster status
Cluster: run .oxo-flow/runs/latest
  completed 12   running 3    pending 47      failed 1
  instance            job      state    elapsed    exit
  align_all_9c4e12  1400_1   running  2:24:11    -

--json emits the same document for dashboards. For logs, oxo-flow cluster logs <id-or-instance> reads the engine-captured logs/<instance>.out/.err before the accounting store, so it works on sites without slurmdbd; --tail N trims the output.

With --all or no run directory it falls back to the raw scheduler view. With no ids it lists every job the scheduler currently reports for your user — the natural check right after a plain cluster submit:

$ oxo-flow cluster status --all
Cluster: Executing 'squeue -u alice --noheader -o %i|%T'...
Cluster: 2 queued jobs
  12345: running
  12346: pending

With ids (the ones submit/run printed) it answers exactly those, settling finished jobs from the scheduler's accounting store:

oxo-flow cluster status 12345 12346
oxo-flow cluster cancel 12345

Every cluster action resolves its backend the same way: the --backend / -b flag if given, then $OXO_FLOW_CLUSTER_BACKEND, then slurm. On a SLURM site you never type -b; on a PBS site set the environment variable once (export OXO_FLOW_CLUSTER_BACKEND=pbs). An explicit-but-typoed value is still rejected outright (unknown cluster backend '…' — expected slurm, pbs, sge, lsf, kubernetes (k8s), aws-batch, or gcp-batch) — the default never swallows a bad name.


Driver Defaults and the [cluster] Profile Block#

run --profile submits and tracks jobs through the backend driver, whose built-in defaults (max_submitted 50, max_array_size 1001, poll_interval 5s) are specified once in the DriverConfig table; the profile's [cluster] block overrides those three knobs. The full annotated TOML example (every key, the driver default each omission falls back to, and where a rule's own time_limit wins) lives once in Cluster submission — that copy is authoritative.

extra_args entries are emitted as scheduler directives verbatim and unvalidated — a typo reaches the scheduler as written (the same applies to cluster submit --extra-arg). Cluster CLI actions resolve --backend as flag > $OXO_FLOW_CLUSTER_BACKEND > slurm (see Monitoring Jobs); the profile block's backend key remains required and is validated the same strict way.

Cluster flag surface#

run --profile does not consume every run flag. The flags below are parsed but have no effect on the cluster path — each of the first three prints a warning and the run proceeds (-j is dropped silently):

run flag Cluster-path behavior
--retry Warned and dropped — there is no engine-side retry on the cluster path. Failed rules are recorded in the checkpoint and re-queued by resume (checkpoint path argument) or run --resume-failed.
--timeout Warned and dropped — per-job walltime comes from the profile's [cluster] walltime key or the rule's own time_limit; the --walltime flag exists only on cluster submit.
--max-threads / --max-memory Warned and dropped — per-rule [resources] and the [cluster] profile drive scheduler limits; [resource_budget] performs the pre-submit check.
-j / --jobs Silently inapplicable — cluster concurrency is governed by max_submitted (the profile key or --max-submitted), clamped by [resource_budget] max_jobs.
--skip-env-setup, --ai-recover Accepted for command-line parity (warned, issue #136): environments are prepared inside each job script, and AI recovery does not run on the cluster path.

--skip-ref-build is accepted on the cluster path and is a no-op there by construction: the reference-build step runs only on the local execution path, and run --profile never builds references itself.

--provenance is honored on the cluster path (issue #1093): each rule's expanded outputs are hashed when the job settles and recorded into the checkpoint's checksums map, exactly like a local run — provenance verify and the freshness gate read the same entries from either execution path. The per-rule checksum = "sha256" | "md5" declaration works there too. A run that records nothing still warns, with the same wording as the local completion path.

--keep-going / -k is honored on the cluster path (issue #1093). It gates exactly one decision — what a required = true rule's failure does to in-flight work:

Mode Required rule fails required = false rule fails
default cancels every in-flight job, then exits non-zero never cancels; failure recorded, dependents blocked, run exits 0
-k / --keep-going in-flight jobs finish, ready independent instances keep dispatching until the queue drains, the failed rule's dependents stay blocked — then exits non-zero (a required rule failed) same as default

Interrupts (Ctrl-C / SIGTERM / SIGHUP) and driver-internal errors always cancel every in-flight job regardless of -k; the flag never softens the verdict — a run with a required failure exits non-zero either way.


Scheduling Fairness#

Cluster submission honors the same fairness contract as local execution:

  • Priority is honored: each round, ready rules are submitted in effective-priority order — declared priority plus priority aging (every round a ready rule spends beyond the max_submitted cap gains +1). A producer parked at the cap therefore outranks fresh high-priority rules after priority gap rounds — priority-inversion starvation cannot persist.
  • In-flight jobs are cancelled on a required failure by default: when a required rule fails, the driver cancels every job already submitted (scancel/qdel/bkill) before the run exits — no orphaned cluster jobs. With -k / --keep-going those jobs instead finish, ready independent instances keep dispatching until the queue drains, the failed rule's dependents stay blocked, and the run still exits non-zero (see Cluster flag surface).
  • Waits inside the scheduler queue are the scheduler's own ordering and are visible via squeue/qstat/bjobs (job ids are recorded per rule under the run directory). The local 60-second "waiting for resources:" diagnostics have no cluster counterpart by design — the pool lives on the scheduler, not in the engine.

Per-sample dispatch order#

Equal-priority ties are broken by instance name by default. Expanded instance names are <rule>_<group>_<sample>, so that tie-break groups every instance of one rule together: a per-sample pipeline submits all align instances, then all qc instances, then all merge instances. The first merge cannot run until the slowest sample's producers finish, and no per-sample result exists until the end of the run.

Set [engine] dispatch_order = "sample" to order ties by the sample binding first, then by rule declaration order:

[engine]
dispatch_order = "sample"

[[sample_groups]]
name = "cohort"
samples = ["S1", "S2"]

[[rules]]
name = "align"
output = ["out/{sample}/align.bam"]
shell = "align {sample} > out/{sample}/align.bam"

[[rules]]
name = "qc"
output = ["out/{sample}/qc.txt"]
shell = "qc {sample} > out/{sample}/qc.txt"

[[rules]]
name = "merge"
input = ["out/{sample}/align.bam", "out/{sample}/qc.txt"]
output = ["out/{sample}/merge.txt"]
shell = "merge out/{sample}/align.bam out/{sample}/qc.txt > out/{sample}/merge.txt"

Both producers read the same sample independently, and merge joins them. Under the default "rule" mode the ready list is align_S1, align_S2, qc_S1, qc_S2, …; under "sample" it is align_S1, qc_S1, align_S2, qc_S2, …. Priority aging remains the primary key: a sample-2 producer that has waited a round accrues +1 and can run before sample 1's newly-ready merge_S1. Sample mode therefore keeps each sample's stages adjacent among equal-priority ties and reduces phase-wise batching, but per-sample drain is best-effort, not guaranteed. Total work and the max_submitted cap are unchanged.

The setting is only a tie-break over currently ready rules: it never reorders dependencies, and it does not delay sample 2's independent producers behind sample 1's unfinished stages. priority remains the primary key — a rule whose declared (+ aging) priority is higher still jumps the queue, which is useful for a genuinely urgent one-off rule. Use priority for per-rule urgency and dispatch_order = "sample" for per-sample pipeline flow.

The same ordering is honored by local execution and by the cluster driver. The default "rule" mode is byte-identical to previous releases.


See Also#