oxo-flow run#
Execute a workflow.
Usage#
Arguments#
| Argument | Description |
|---|---|
[WORKFLOW] |
Path to the .oxoflow workflow file. Optional — if not specified, auto-discovery searches for: (1) main.oxoflow in current directory, (2) alphabetically first *.oxoflow file in current directory. |
[KEY=VALUE]... |
Direct config overrides (see Config overrides) |
Config overrides#
KEY=VALUE and --KEY=VALUE work for any key; the --KEY VALUE space form
also works for any key present in [config] (plain key = "value" entries
included) — a key absent from [config] is a hard error (an unknown flag
would otherwise be indistinguishable from a mistyped option; a typo'd
--key=val after the workflow still sets a config var, so prefer the =
forms for undeclared keys).
An override key that differs from a declared key only by hyphens vs
underscores is canonicalized onto the declared spelling — --api-token=x
sets the declared key api_token (and inherits its sensitive masking);
keys matching nothing declared keep the literal spelling.
Options#
| Option | Short | Default | Description |
|---|---|---|---|
--jobs |
-j |
1 |
Maximum number of concurrent jobs |
--keep-going |
-k |
— | Continue execution when a job fails |
--json |
— | — | Emit a machine-readable JSON summary on stdout after the run (see JSON output) |
--workdir |
-d |
Workflow file's directory | Working directory for execution |
--checkpoint |
— | <workdir>/.oxo-flow/checkpoint.json |
Checkpoint file for this run. Use one path per workflow variant (smoke/test/full) that shares a workdir so each variant resumes independently (see Variant checkpoints) |
--log-file |
— | .oxo-flow/logs/oxo-flow.log in the workdir |
Write the run log to a custom path instead (relative paths resolve against the workdir; previous logs rotate to PATH.1 … PATH.9) |
--target |
-t |
All rules | Run only specific target rules |
--from-rule |
— | — | Re-run from RULE downstream: invalidates the transitive-dependent closure of RULE in the checkpoint, then runs RULE and everything downstream of it (issue #1017). Unlike --target (upstream closure), --from-rule propagates downstream — stale downstream outputs are invalidated before the freshness gate so they re-execute with the new upstream results |
--module |
— | — | Run one include module and the producers of its declared inputs (repeatable; unions with --target). Module names are the include's name field or its file stem (see Partial module runs) |
--retry |
-r |
0 |
Number of times to retry failed jobs |
--timeout |
— | 0 (disabled) |
Timeout per job in seconds, or a duration like 1h/30m |
--max-threads |
— | 0 (auto-detect) |
Maximum CPU threads available for execution. A workflow [resource_budget] max_threads caps the effective value (see Resource budgets) |
--max-memory |
— | 0 (auto-detect) |
Maximum memory in MB available for execution. Auto-detect is cgroup-aware on Linux (clamped to the tightest memory.max — see Notes); a workflow [resource_budget] max_memory also caps the effective value. Set it explicitly (e.g. --max-memory 430000 --retry 2) when the clamp misreads a deliberately generous slice |
--skip-env-setup |
— | — | Skip environment setup (assume environments are ready) |
--skip-ref-build |
— | — | Skip automatic reference/index building (assume pre-built) |
--cache-dir |
— | — | Directory for caching environment setup state (a small readiness manifest that rebuilds itself on demand). The retired post-run cache_max_age_days aging sweep warns and points at oxo-flow env gc, which reclaims the shared env store (OXO_FLOW_ENV_STORE) — not this directory |
--resume-failed |
— | — | Resume only failed rules from a previous run. Also clears stale checkpoint entries for rules whose when gate recorded false (they are skips, not failures); status reports them as skipped, not failed |
--profile |
— | — | Execution profile name, loaded from profiles/<NAME>.toml or profiles/<NAME>.oxoflow (see Execution profiles) |
--max-submitted |
— | profile default (50) |
Cluster jobs in flight at once (overrides the profile's max_submitted) |
--provenance |
— | — | Track output file checksums for later verification |
--arg |
— | — | Legacy form: set a workflow config value (KEY=VALUE). Repeatable. See [config] in workflow-format |
--bundle |
— | — | Execute from a published bundle (.tar.zst or .tar.gz). Extracts, verifies checksums, shows resource requirements, and prompts for confirmation |
--yes |
— | — | Skip the confirmation prompt when running a bundle (required in non-interactive sessions: CI, scripts, redirected input, or --json) |
--ai-recover |
— | — | Enable AI error recovery on rule failure |
--ai-max-retries |
— | 1 |
Maximum AI recovery attempts when a --ai-recover call fails (only a failed call is retried). This bounds recovery attempts; the generation-side --ai-max-retries on template (default 10, bounding orchestrator rounds) is a separate knob |
--samples |
— | all discovered samples | Sample selection: @path replaces the workflow's samples from a samplesheet, +@path appends (same-name groups merge, new groups added); names filter (or declare when the workflow ships no samples), first:N (pilot) and ready (samples whose entry inputs are complete) filter. Repeatable, comma-separated |
--rerun |
— | — | Force re-execution of this run's rules (ignore up-to-date checks). Checkpoint records for rules outside this run are kept |
--no-report-snapshot |
— | — | Skip the automatic report snapshot written after the run (see Report snapshots); resume has the same flag |
--background / --detach |
— | — | Detach the run into a background process and exit 0 immediately (see Background runs (--background)); --detach is an alias |
--allow-protected-drift |
— | — | Proceed despite protected-tree drift: when [engine] protected_data_trees baselines no longer match the tree on disk, run refuses to execute; this flag is the explicit operator override to run anyway (see Protected data trees) |
--strict-envs |
— | — | Hard-fail on environment setup errors instead of using the graceful pixi stale-lock fallback (see Stale pixi.lock fallback) — for GxP-style deployments that must never run on a re-solved environment |
--verbose |
-v |
— | Enable debug-level logging |
--quiet |
— | — | Suppress informational output — progress rendering, DAG listings, and per-rule completion lines are omitted; errors, warnings, and machine-readable output (--json) are unaffected |
--no-color |
— | — | Disable colored output (also honors the NO_COLOR environment variable) |
Flags the cluster driver does not honor
On run --profile, several run flags are parsed but not applied — the
engine warns once per flag you actually passed (full table:
Cluster flag surface).
--provenance and --keep-going are honored there: checksums are
recorded as each job settles, exactly like a local run; by default a
required rule's failure cancels every in-flight job, while -k lets
already-submitted independent jobs finish (dependents of the failed
rule stay blocked) and the run still exits non-zero — use
required = false for best-effort rules. Interrupts always cancel
every in-flight job. Set a per-job walltime with the rule's
time_limit (or the profile's walltime), and leave resource limits
to the per-rule [resources] block and the [cluster] profile.
JSON output#
The document carries command, status, results counts, and resources.
It is emitted on every exit path — completed, failed, and aborted
(preflight failures, budget breaches, cluster runs, plain-failure aborts) —
so --json consumers never get zero bytes. status is "completed" only
when the run finished with zero required failures and was not interrupted;
every other exit is "failed".
Human progress and result output on stderr is not suppressed — this flag appends the summary, it does not switch the run to machine-only mode.
resources lists every benchmarked rule (rule, status,
wall_time_secs, peak_rss_mb, cpu_seconds, retries) — the same data
as the report's Benchmarks table; it is empty on aborted paths and on
cluster runs (use the report's sacct-based Resource Accounting section
there).
Quick reference#
Common flags at a glance — every flag is documented in the Options table above and the sections below.
| Goal | Flags |
|---|---|
| Dry-run first / inspect the plan | dry-run before run |
| Parallelism and failure handling | --jobs, --keep-going, --retry, --timeout |
| Run part of the workflow | --target / -t, --from-rule, --module |
| Resume after failure | --resume-failed, or the resume command |
| Select samples | --samples (see Incremental data arrival) |
| Force re-execution | --rerun |
| Config values from the command line | [KEY=VALUE]... positionals, --arg KEY=VALUE (see Config overrides) |
| Execution profile / cluster | --profile, --max-submitted (see Cluster submission) |
| Resource budgets | --max-threads, --max-memory, [resource_budget] (see Resource budgets) |
| Machine-readable output | --json (see JSON output) |
| Output checksums | --provenance |
| Run detached / wait for it | --background / --detach, then oxo-flow wait |
| Quieter output | --quiet, --verbose, --no-color |
Output#
oxo-flow v0.31.0 — Rust-native bioinformatics pipeline engine
DAG: 5 rules in dependency order
1. fastqc
2. trim_reads
3. bwa_align
4. sort_bam
5. call_variants
⠋ [00:15] [████████████░░░░░░░░] 3/5 ETA:0:00:42 (executing sort_bam)
✓ fastqc (2.1s)
✓ trim_reads (15.0s)
✓ bwa_align (42.3s)
✓ sort_bam (10.2s)
✓ call_variants (8.4s)
Done: 5 succeeded, 0 skipped, 0 failed
✓ 5 output files verified (118.3MB total)
A progress bar shows execution progress with:
- Elapsed time
- Current position / total rules
- Estimated time remaining (ETA), plus the rule currently executing
- The final summary line reports succeeded / skipped / failed counts
Module-grouped output#
Workflows composed from [[include]] modules group the plan listing and the
--keep-going failure summary under module headers, so a composed pipeline
reads as module → rule instead of one long flat list:
DAG: 6 rules in dependency order
Preprocessing — 2 rules
1. fastqc
2. trim_reads
Alignment — 3 rules
3. bwa_align
4. sort_bam
5. markdup
Main workflow — 1 rule
6. report_summary
- Rules from a module file (
[[include]]) collect under their module's display title; rules declared in the main file land underMain workflow - Attribution is the same logic the metro graph export uses: a
module::rulenamespace prefix wins, then member matching; host rules that merely extend a member's name may land in that module's group (a display-only quirk — edges and execution are unaffected) - Group order follows first appearance in the execution order, and the per-rule numbers stay a single global sequence so plan position remains greppable
- Workflows without modules keep the flat listing above byte-for-byte — grouping is opt-in by structure, not a flag
The failure summary groups the same way:
Failed rules:
Germline (1):
✗ haplotype_caller — exit code 1
Blocked rules (did not run):
Germline (1):
⊘ joint_genotyper — depends on 'haplotype_caller'
Main workflow (1):
⊘ qc_report — depends on 'joint_genotyper'
Per-line shapes (✗ name — reason, ⊘ name — depends on 'dep') are
unchanged, so tooling that parses run logs keeps working.
Examples#
Run with auto-discovery (when only one .oxoflow file exists)#
Run with main.oxoflow (priority discovery)#
Run with explicit workflow#
Run a workflow from a repository (nextflow-style)#
A public git repository can be executed directly — no clone, no bundle:
# Default branch — the gh: prefix is optional for github.com repositories
oxo-flow run owner/pipeline
# Pinned to a git tag or branch (@ref selects a git ref — unlike `pull`,
# where @tag means a GitHub Release bundle)
oxo-flow run owner/pipeline@v0.31.0
# The explicit forms work the same way (gh: can always be kept)
oxo-flow run gh:owner/pipeline
oxo-flow run gh:owner/pipeline@v0.31.0
# Any git URL or local repository directory
oxo-flow run https://example.com/team/pipeline.git
# A local checkout via file:// (must be an existing directory)
oxo-flow run file:///path/to/pipeline
A bare owner/pipeline argument is interpreted as a GitHub repository
unless it names something real on the filesystem: a path that exists, one
starting with ., or one ending in .oxoflow is always treated as a
local workflow path. Only a single-slash owner/name — optionally with
@ref and a trailing .git — maps to https://github.com/owner/name.git.
(So a missing data/pipeline.oxoflow keeps its normal "workflow file not
found" error instead of attempting a clone.) An https://…/*.git URL runs
that repository directly; file://<dir> runs an existing local checkout.
The repository is checked out into .oxo-flow/repos/<host>-<owner>-<name>
under the current directory (e.g. .oxo-flow/repos/github.com-owner-pipeline;
reused on later runs — delete the directory to force a fresh clone; the host
and owner prefixes keep same-named repos of different owners or hosts from
sharing a checkout), and the workflow file is auto-discovered
(main.oxoflow first). Because the clone is a read-only cache, the working directory
defaults to the current directory for repository runs — outputs, the
checkpoint, and the workdir lock all land next to your data, not inside the
clone. All other run semantics apply unchanged: --samples, --rerun,
config overrides, --workdir, checkpoint-aware dry-run previews, etc.
For github.com clones the official URL is tried first, then the
ghfast.top / gh-proxy.com mirrors automatically (see
China Mirrors).
Custom working directory#
Relative paths in the workflow resolve against the workflow file's
directory — the single resolution base. When --workdir
differs from the workflow directory, the engine anchors workflow-shipped
files so they stay valid from the execution cwd:
- Environments: relative
conda/mamba/pixi/venv_requirementsfile specs resolve against the workflow directory. - Rule scripts: workflow-relative script files render absolute (workflow-rooted) so rule shells — which run with cwd = the workdir — can execute them.
- Workflow source files (zero duplication): declared rule inputs that
no rule produces, plus interpreter script paths (
python scripts/x.py), are materialized in the workdir as symlinks into the workflow directory at run start — the repo keeps the single copy of every source file. A pre-existing byte-identical workdir copy is replaced by a symlink (with a notice); a copy that differs is kept for the run and warned about (it may be a deliberate override). Repo paths a shell command mentions but no rule declares are reported as a note — declaring such a path as a rule input links it automatically. - Tool index sidecars (zero duplication): companion index files that
tools auto-derive from a linked file's path by convention —
samtools faidx/picard/GATK (ref.fa.fai,ref.fa.dict), bwa (ref.fa.amb/.ann/.bwt/.pac/.sa), minimap2 (.mmi), and the bgzip/tabix/htsfile index forms (.gzi,.tbi,.csi,.crai,.bai) — are linked alongside the file itself. Those paths never appear in a shell command, so static analysis alone cannot see them; the engine links every such sidecar that exists next to a linked workflow file, unless a rule produces the sidecar itself (a dedicated indexing rule keeps ownership). Tools driven through a bare index prefix (bowtie2-x index) should declare their index files as rule inputs directly. {input}and{log}: stay workdir-relative. Inputs are addressed from the run cwd (freshness/optional-input gates probe workdir-relative paths), and the engine creates log parent dirs under the workdir. Place workflow-consumed data under the workdir (or pass absolute paths).- input_groups disk scans: relative patterns scan the workflow
directory and emit workflow-absolute paths into
{input}. - Reference sources:
[references].sourceresolves against the workflow directory; reference outputs still land in the workdir. - Outputs: stay workdir-relative — they are run artifacts and land
under the workdir, matching the checkpoint and
.oxo-flowstate.
This keeps a workflow self-contained: for a shared, read-only workflow (a
fixed reference resource), point --workdir at the analysis directory
instead:
# Workflow lives in a central, read-only location; data and results
# belong to the current analysis directory
oxo-flow run /opt/pipelines/wgs.oxoflow --workdir .
# Outputs, .oxo-flow checkpoint, and lock all land in the workdir;
# dry-run/clean/report/resume accept --workdir too, and resume re-uses
# the workdir recorded in the checkpoint
Parallel execution#
Keep going on failure#
Retry failed jobs#
Run specific targets#
Target names may be prefixes: -t align matches every rule whose name
starts with align (including expanded instance names like
align_cohort_S1). The closure walks the instantiated DAG — it
includes each target plus every upstream rule that survives the plan-time
when gating, and skips when-gated variants entirely: a variant whose
when is false under the effective config never enters the execution set,
its upstream is not pulled in, and consumers of its (never-produced)
outputs are pruned too. Naming a when-gated-false target explicitly
reports it and aborts with "nothing to run" instead of planning a run
that cannot produce output.
The same when-aware closure applies to full runs (no --target) and
--module runs: a consumer left without any surviving producer of an
input group (all of its producers when-gated false, the files not
pre-built on disk, and the group not optional = "any") is dropped from
the plan with a note — rule '...' is unreachable — its upstream is
when-gated false; removed from the full-run execution set — instead of
failing mid-run on a file that was never produced.
The when-gated rule itself stays in the plan and is counted as skipped
exactly once by the executor. See
when-skipped instances
for how the consumer's baked expand_inputs list is cleaned in the same
sweep.
Set a per-job timeout#
Limit resource usage#
# Use only 16 threads and 32GB memory
oxo-flow run pipeline.oxoflow --max-threads 16 --max-memory 32768
A workflow can carry the same caps in [resource_budget] so every run of the
file is bounded without an external wrapper (issue #1080):
Cache environment setup#
# Cache environment setup state for faster subsequent runs
oxo-flow run pipeline.oxoflow --cache-dir .oxo-flow/cache
The cache is not aged automatically — --cache-dir only selects where
environment setup state lives, and that state is a small readiness manifest
that rebuilds itself on the next run if entries go missing. To reclaim the
bulk of environment disk usage, target the shared env store instead: run
oxo-flow env gc (see env), which collects
OXO_FLOW_ENV_STORE directories under its --min-age-days gate (default 90)
so entries that recent runs may still reference survive.
Removed: cache_max_age_days
Older versions swept this cache after every run, keyed off the workflow
config key cache_max_age_days. That mechanism was retired in favor of
the explicit, store-aware oxo-flow env gc command. A
workflow that still declares the key now triggers a one-time warning
pointing at env gc; the key has no effect.
Skip environment setup (when environments are pre-built)#
Shared environment store (pixi/venv, opt-in)#
Pixi and venv environments can be materialized once per content key in a
shared store (opt-in: set $OXO_FLOW_ENV_STORE to a non-empty path —
location IS activation; unset keeps envs self-contained inside each workflow
tree, exactly like before the store existed). When enabled, the workflow tree only holds a
symlink into the store: ten checkouts of the same pipeline share one
environment instead of re-downloading it ten times, and a deleted store dir
heals itself — the next run detects the broken link and rebuilds. Clean up
unreferenced environments with oxo-flow env gc. Conda,
Docker, and Singularity environments keep their native cache layouts and are
unaffected — the store is pixi/venv-only by design (see the env
command for the rationale: conda name envs already share a
central prefix, conda prefixes are not relocatable, containers use their own
registries, modules come from the cluster).
Backend preflight: fail fast on a missing environment backend#
After DAG construction and before any rule executes, run validates every
unique environment spec among the pending rules — once per spec (rule
templates fan out one spec across N samples, so the spec set is tiny even
for large cohorts). When a required backend binary is missing
(conda/mamba/pixi/docker/singularity/modules), the run aborts
before anything executes:
2 environment backends required by pending rules are unavailable; no rules were run:
- conda: conda is not installed or not in PATH — required by 3 pending rules, e.g. `bwa_align` (environment: envs/alignment.yaml); install conda and ensure it is on PATH
- pixi: pixi is not installed or not in PATH — required by 1 pending rule, e.g. `report` (environment: pixi.toml); install pixi (https://pixi.sh) and ensure it is on PATH
Rules sharing one broken spec, and distinct specs failing with the
identical backend-missing message, collapse into a single entry — one root
cause per line, with the pending-rule count, an example rule, the primary
spec, and a mode-aware remediation hint. Nothing ran, so the --json
summary still reports the run as failed with zero counts.
See Troubleshooting — Environment Issues
for per-backend checks.
The remediation depends on why the spec failed:
- Manifest file not found (pixi declared by relative path): the spec
is resolved against the workflow directory (the same base the executor
uses), and the hint says to create the manifest at the
resolved path or correct the rule's
pixipath — installing the backend would not help when the file itself is absent. Other backends keep the install advice (a missing conda YAML surfaces only at task execution). - Backend binary missing (the manifest exists, or the backend cannot be
located for another reason): the hint is the per-backend install advice
(
install pixi (https://pixi.sh) and ensure it is on PATH, and so on).
This is the run-time safety net. validate and plan/dry-run preflight
pixi manifests up front (error
E019) so the failure surfaces before you
commit to a run; run keeps the check because manifests can also be
created between validate and run.
Stale pixi.lock fallback#
A committed pixi.lock pins exact package builds. When an upstream channel
later removes one of those builds (bioconda does this occasionally — old
builds disappear), a lock-based pixi install fails with a fetch error
(failed to fetch … 404 Not Found) and every run using that environment
fails until the lock is refreshed by hand.
run recovers from this automatically (issue #1011). When a pixi setup
fails with that fetch signature and a committed pixi.lock sits beside the
manifest, the executor:
- Deletes the committed lock and re-runs the setup command, so pixi re-solves from the manifest.
- Parks the fresh solution as
pixi.lock.newbeside the committed lock (never silently overwriting it). - Restores the committed
pixi.lockbyte-identical. - Warns loudly with the package-level diff (old lock vs new solve) and
marks the run's environment reproducibility as degraded — the report
environment section carries an
Environment Reproducibility: degraded …disclosure (see Report).
The run proceeds on the re-solved environment: rule commands wrap with
pixi run --as-is, which skips re-validating the committed lock against the
upstream channels — the restored lock still pins the vanished build, so the
default wrap would fail the rule with the same 404. The degraded marker
persists in the environment cache (a pixi_degraded.json sidecar beside
environment_cache.json), so later runs keep wrapping with --as-is: they
skip setup entirely (the re-solved environment is already materialized and
cache-ready) but still disclose degraded reproducibility in the report.
To adopt the fresh lock, replace pixi.lock with pixi.lock.new and commit
it. The environment's cache key folds the lock content, so the stale degraded
marker stops matching and the next run returns to the normal lock-validated
wrap — nothing to clean up by hand.
--strict-envs disables the fallback entirely: a stale-lock failure keeps
today's hard error with a hint explaining how to refresh the lock by hand.
Use it for GxP-style deployments that must never run on a re-solved
environment.
What the fallback does not do
A deterministic solver conflict (failed to solve requirements) is
not recoverable — re-solving would land on the same conflict — so
it still fails immediately. The fallback fires only for fetch failures
(HTTP 404/410 or equivalent) with a committed lock present.
Pass config values#
Every key in the [config] section of the workflow (including declarative
key = { default = …, required = …, … } entries) can be overridden from the
CLI — no extra flags to declare:
# Direct flag form
oxo-flow run pipeline.oxoflow --database refs/nt
# Attached form
oxo-flow run pipeline.oxoflow --database=refs/nt
# Key=value form directly after the workflow file
oxo-flow run pipeline.oxoflow database=refs/nt threshold=1e-3
# Legacy --arg form (still supported)
oxo-flow run pipeline.oxoflow --arg database=refs/nt --arg threshold=1e-3
# Empty string = explicit derive-if-empty override
oxo-flow run pipeline.oxoflow gene_bed=
oxo-flow run pipeline.oxoflow --gene_bed ""
Did-you-mean hints on mistyped overrides and rules
Typos in --KEY VALUE overrides and --rule targets are diagnosed,
not just rejected. An override key that is not declared in [config]
and never referenced as {config.KEY} suggests the closest declared
key by edit distance:
oxo-flow run pipeline.oxoflow --samplename S1
Warning: override 'samplename' is not a declared [config] key and the
workflow never references {config.samplename} — did you mean
'sample_name'? It changes nothing.
The same heuristic powers --rule suggestions (oxo-flow debug
workflow.oxoflow --rule <typo> lists the nearest rule names) and the
engine's rule-not-found errors, which list the available rules
(sorted, capped at ten) when a when/input reference misses. The
hint logic lives once in the core (edit_distance) — the CLI and the
engine always agree on what "closest" means.
Empty values select the derive-if-empty branch
An empty value is accepted and overrides the config key with the
empty string. Pipelines that auto-derive a value branch on
[ -n "{config.gene_bed}" ] — passing gene_bed= forces that branch
off (e.g. to re-derive from a new reference instead of reusing a
declared default path). Only the key must be non-empty; =value
alone is a syntax error.
Ordering: run flags before overrides
run flags (--json, --rerun, --samples, -j, …) must come
before positional KEY=VALUE overrides — the override list is
trailing, so a flag typed after it is reported with actionable guidance:
oxo-flow run pipeline.oxoflow min_quality=30 --json
'--json' is a command flag, not a config override.
Command flags must come before KEY=VALUE overrides, e.g.:
oxo-flow run <workflow.oxoflow> --json min_quality=30
For a config key that itself starts with dashes, use --arg KEY=VALUE
(also placed before positional overrides).
Execution profiles#
--profile <NAME> applies a reusable config supplement to a run. Profiles
are plain TOML (or .oxoflow) files placed in the workflow's own
profiles/ directory; .toml is tried before .oxoflow:
The name is just a filename — it carries no built-in meaning. What the
profile contains decides its effect: a [config]/[defaults] block
supplements config, a [cluster] block additionally routes execution to
a scheduler (see Cluster submission). Naming a
profile conda or slurm is a convention, not a mechanism — compute
environments are declared per rule via [rules.environment] and are
independent of profiles.
The profile's [config] table fills in keys the workflow does not set
itself by default — values already present in the workflow are never
overwritten. This lets one workflow carry per-environment fallback
defaults (laptop, cluster, cloud). Declare profile_mode = "override"
in the workflow's [workflow] section when a profile should instead
replace workflow values (deep-merged: nested tables merge recursively,
scalars and arrays are replaced) — the classic "cluster profile switches
threads/memory" case:
If the named profile file does not exist, run (and dry-run) fail
loudly with a nonzero exit and an error listing every available profile
in the profiles/ directory — a typo'd --profile must never silently
run with the workflow's own config. The same hard error applies when the
profile file exists but its TOML fails to parse or merge.
Cluster submission#
A profile that also carries a [cluster] block makes run --profile submit
to a scheduler instead of executing locally: the run expands rules into
per-instance jobs, submits at most max_submitted at a time (grouped into
scheduler arrays where possible), tracks them to completion, and writes the
checkpoint exactly as a local run does. Both conditions are required — a
[cluster] block in effect and a profile named; adding either one alone
never changes an existing local run.
The full story — the [cluster] TOML reference and driver-default fallbacks,
profile lookup rules, TMPDIR exports on job scripts, the scratch = true
plan-time rejection, job arrays, the run directory layout, what status.json
records, and interrupt/re-attach semantics — lives in
Cluster submission on the
cluster page. run flags that are parsed but dropped on that
path (--retry, --timeout,
--max-threads/--max-memory) are tabulated in
Cluster flag surface
(--provenance and --keep-going are honored there — checksums are
recorded as jobs settle, and -k keeps in-flight independents running
past a required failure).
Cluster scheduler submission (SLURM, PBS, SGE, LSF, Kubernetes, AWS Batch,
GCP Batch) is configured
separately — see the cluster command.
Partial module runs (--module)#
A composed workflow (clinical pipelines like clindet: several
[[include]] modules chained together) can run just ONE module and the
producers of its declared inputs:
# run only the germline module (rules/20_germline.oxoflow) of a composed
# clinical pipeline — the module name defaults to the include's file stem
oxo-flow run main.oxoflow --module 20_germline
# multiple modules + rule targets union together
oxo-flow run main.oxoflow --module 20_germline --module 30_vcf_norm -t report
The closure is: the module's rules + every host rule producing one of its
declared concrete contract inputs; upstream DAG dependents come through
the regular target machinery. The closure is when-aware:
two rules sharing one output path behind complementary when gates
resolve to the satisfiable one — a rule whose template when evaluates
false under the effective config never claims a producer slot, and a
member whose wildcard-gated instances were all pruned at fan-out is
dropped from the closure (its name has no node in the instantiated DAG).
Unknown module names fail with the known module list. dry-run --module
previews the same set.
The same when-awareness covers the preflight source gate on -t/--module
runs: a missing input whose only remaining producer instance was vetoed by
its when at fan-out does not gate the run. Expansion records those vetoed
instances' declared outputs in when_skipped_outputs, and the gate
exempts any missing pattern found there — config-expanded at compare time,
so {config.*}-bearing recorded patterns match their expanded gate forms.
The file belongs to a rule that will never run in this cohort, not to a
dead source of the selected closure.
Run directory layout#
Cluster runs leave their job scripts, scheduler ids, and per-job status
under .oxo-flow/runs/<timestamp>/ — see
Run directory layout on the
cluster page.
What a finished job records#
Cluster jobs record the scheduler's accounting data (queue wait, exit code,
wall time, peak RSS, CPU seconds) into per-job status.json files; fields
the site's accounting store does not report are omitted rather than guessed.
See What a finished job records
for the record format, the accounting-variance caveats, and the
interrupt/re-attach semantics.
Failed-rule output rollback#
A failed rule's declared outputs are rolled back: files created by the
attempt are deleted, and pre-existing files are moved aside as
*.oxo-failed (protected outputs declared with protected_output are
never touched). The failure message names the rolled-back files, so a rule
that created one of two declared outputs and then failed shows both the
missing file and the one that was removed:
[oxo-flow] output validation failed: 1 declared output(s) not found: …/merge.vcf
[oxo-flow] declared outputs of the failed rule are rolled back
(deleted or moved aside as *.oxo-failed): …/merge.vcf.idx
Placeholder validation (strict by default)#
Every executing rule's rendered command is scanned for engine placeholders
that survived rendering — {sample}, {group}, {pair_id},
{experiment}, {control}, {tumor}, {normal}, {chunks}, {log},
{log[0]}, {threads}, {memory}, {effective_threads},
{effective_memory_mb},
{meta.<column>}, {config.<key>}, {params.<key>}, {input…},
{output…}, {wildcard.*}, and {values.*}. Such a token has
no value bound to it, so the shell would receive the literal brace text and
the tool would fail on a path that looks valid. The rule fails before the
shell spawns:
✗ rule 'align_reads' failed: unresolved placeholder(s) {meta.bam} reached
the rendered command — … Fix: give the rule a sample-domain binding, or
add `singleton = true` for an intentional constant-metadata read. …
The fix hints are class-specific (declare the [config] key, check an
{input[N]} index, add the sample-domain source, …). Two escape hatches
exist while downstream workflows migrate:
[engine] strict_placeholders = falsein the workflow, orOXO_FLOW_STRICT_PLACEHOLDERS=0in the environment (wins per invocation).
Both restore the historical warning-only behavior. singleton = true
silences the static W041 diagnosis for an intentional one-instance
{meta.*} read but does not soften this runtime gate: a {meta.*}
token that survives rendering (more than one metadata row and no
sample-like binding) is still fatal. A one-row metadata table is the
supported constant read; use {config.<key>} for a workflow constant.
{sample}/{group} survivors are fatal in every case. Declared output
paths are checked too: an unresolved placeholder there is fatal even for
a singleton — a literal file named after the token is never a valid
artifact. The strict_placeholders switches above also restore the
warning-only behavior for that namespaced output-path check (the
bare-wildcard output gate is a separate, always-on correctness check).
Shell ${...} parameter expansion (${#parts[@]}, ${ch##*.clones_}) is
not engine syntax and the gate never treats it as a placeholder.
Template rendering is textual, however: a bound name written inside
${...} is substituted like any other token (${sample} renders as
$<sample value>), so give shell variables names that do not collide with
fan-out variables. Brace groups ({ echo hi; }) and brace
expansion ({1..10}, {a,b}) are likewise shell text. Two realistic
false-positive shapes need an escape: a Python f-string that prints an
engine token (python3 -c 'print(f"{sample}")' — write f"{{sample}}"
instead) and a shell literal (\{sample\}). Escape the opening brace in
the host language (\{ in bash, {{ in Python f-strings) or build the
literal from parts; $-prefixed and backslash-escaped braces are never
flagged. lint reports the static
diagnosis as
W041 — a rule that references
the sample domain without a binding source.
Execute a published bundle#
# Run from a bundle (extracts, verifies checksums, shows resource requirements,
# then prompts for confirmation — requires an interactive terminal on both
# stdin and stderr)
oxo-flow run --bundle pipeline-bundle.tar.zst -j 16
# Skip confirmation for CI/scripts (also required when stdin is redirected
# or with --json, which is always non-interactive)
oxo-flow run --bundle pipeline-bundle.tar.zst -j 16 --yes
# .tar.gz format also supported
oxo-flow run --bundle pipeline-bundle.tar.gz -j 16 --yes
# Pull from remote and execute in one step
oxo-flow pull gh:user/repo@v1 && oxo-flow run --bundle repo-bundle.tar.zst -j 16 --yes
Bundles published with publish --sign-key carry an ed25519 manifest
signature. run --bundle verifies it after the checksums and aborts before
execution if the manifest or signature was altered; unsigned bundles keep
the checksums-only behavior (see publish → Bundle Signing).
Pilot runs and scale-up#
For a large cohort, run a fast pilot on a subset first; the checkpoint
then skips the pilot samples when you scale up. --samples needs a
workflow that has a sample dimension (sample_pattern,
[[sample_groups]], sample_groups_file, or [[pairs]]): a
workflow with none rejects the flag with
--samples matched no samples in this workflow.
# Pilot: run the full pipeline on the first 2 samples only
oxo-flow run pipeline.oxoflow --samples first:2
# Or name the pilot samples explicitly (combines with first:N)
oxo-flow run pipeline.oxoflow --samples first:2,NA12899
# Scale up: no flag needed — completed samples are skipped automatically
oxo-flow run pipeline.oxoflow
# Preview the pilot plan without executing — with a checkpoint present,
# dry-run also predicts what will actually re-run (and what stays
# protected); see dry-run's "Checkpoint-Aware Rerun Preview"
oxo-flow dry-run pipeline.oxoflow --samples first:2
# Fix a config bug found by the pilot — only affected rules re-run
# automatically (see "Config changes" below); use --rerun to force everything
oxo-flow run pipeline.oxoflow --rerun
--samples filters every sample source (sample groups, sample_pattern
discovery, and experiment/control pairs). --rerun re-executes the rules
selected for this run even when outputs are up to date — combine it with
--samples to re-run just the pilot subset while keeping the other
samples' completed records.
After a --samples run, a pilot summary is printed: samples run,
wall time, per-sample time, and a linear projection for the full cohort.
Scientific-preflight findings are reported as distinct findings with
their instance counts — one template rule appearing once per sample shows
as 1 distinct finding (3 instance(s)), not 3 finding(s). When the
workflow enables [ai], a plain-language pilot report (health assessment
and scale-up advice) is appended automatically.
Incremental data arrival: --samples ready#
Sequencing centers deliver data in batches, so a cohort's fastq files trickle in over days. Instead of waiting for every sample, run the analysis as data arrives:
# Which samples can be processed right now, and what is still missing?
oxo-flow dry-run pipeline.oxoflow
# Sample readiness: 87/100 complete, 13 waiting
# ⏳ NA12891 (missing: data/NA12891_R2.fastq.gz), …
# Run only the samples whose entry inputs are complete
oxo-flow run pipeline.oxoflow --samples ready
# … more data arrives …
oxo-flow run pipeline.oxoflow --samples ready
# Done: 5 succeeded, 87 skipped — the checkpoint skips completed samples
A sample is ready when every external input belonging to it exists —
that is, every rule input (after wildcard and {config.x} expansion) that
the workflow itself does not produce. Files declared in a rule's
protected_output count as workflow-produced (a rule may declare expensive
secondary artifacts only there, and downstream expand_inputs consumers
read them during the run). Intermediate products are never
checked (producing them is the DAG's job), and optional = true rules do
not block readiness (the executor skips them when their inputs are absent).
Relative paths resolve against the workflow file's directory (or
--workdir) — the same place rules actually run from.
Semantics:
readyis a special--samplesvalue, not a new syntax: it resolves to the names of ready samples and combines withfirst:Nand explicit names as a union. Because it is reserved, a sample literally namedreadycannot be selected by name.- With zero ready samples
runaborts and lists the waiting samples;dry-run --samples readyinstead reports the full cohort state. - For
[[pairs]]workflows, a pair is kept only when both experiment and control inputs are complete; otherwise it is skipped with a note. Missing files that belong to no specific sample (shared references) are reported as workflow-level inputs. - The report is also available as JSON via
dry-run --json(thesamplesblock), and--samples readyworks intest --runas well.
Background runs (--background)#
run --background (and resume --background; --detach is an alias)
detaches execution into a background process and returns to the shell
immediately:
# Foreground prints one line and exits 0; the run continues in the background
oxo-flow run pipeline.oxoflow --background
# started in background (pid 48291) · log: .oxo-flow/logs/oxo-flow.log ·
# monitor: oxo-flow status .oxo-flow/checkpoint.json ·
# wait: oxo-flow wait --workdir /data/run --checkpoint …/checkpoint.json ·
# stop: kill 48291
# Detach a resumed run the same way
oxo-flow resume .oxo-flow/checkpoint.json --background
# Block from a script until the detached run finishes:
oxo-flow wait --workdir /data/run # exit 0 ok, 1 failed, 3 timeout
Mechanics:
- The foreground process spawns a detached child that re-runs the same
command with the detach flag (
--background/--detach) removed — every other flag (-j,--profile,--samples,--rerun,--log-file,--checkpoint, …) passes through verbatim, so a background run behaves exactly like a foreground one: checkpoint, workdir lock, report snapshots, and resume semantics are unchanged. - The child's stdout and stderr are redirected to the run log
(
--log-fileif given, otherwise.oxo-flow/logs/oxo-flow.login the workdir). In this mode the tracing tee is skipped on purpose — the redirect already captures everything, and a second writer would duplicate every line. - The run log archives BOTH the structured tracing stream AND the
user-facing progress narrative (
Running:/✓/Done:) plus one JSON line per execution event (workflow_started,rule_started,rule_completed,rule_failed,workflow_completed) — the whole run is replayable from the single file. Failed rules emit"event":"rule_failed"(compact serde_json, no space after:) carryingrule,exit_code,signal,retries, andduration_ms, sogrep -c 'rule_failed' run.launch.logcounts failures without parsing the human-facing✗narration;"signal":9after an OOM-kill is the smoking gun distinguishing a memory-capped environment from a tool bug.duration_msspans all attempts including retry delays, not just the final one. Staging failures (environment setup, security checks) never start a process, so they surface as"event":"rule_completed"with"status":"failed"and no exit code/signal instead. - The child's pid is written to
.oxo-flow/background.pidin the workdir (also shown in the summary) while the run is in flight; the file is removed when the child exits — including the signal-termination path, so a stale pid never survives a finished run. - The run's own terminal outcome is written to
<checkpoint>.status(beside the checkpoint;runningonce the workdir lock is held, thensucceeded/failedwith the error text). Per-checkpoint placement keeps variant runs in one workdir from aliasing each other.oxo-flow waituses it to distinguish a preflight refusal from a previous clean checkpoint. - The child gets its own process group, so it survives terminal close and Ctrl-C at the shell — it keeps running until it finishes or is killed.
- The foreground exits 0 once the child is spawned; it does not wait. If the spawn fails (for example because another run already holds the workdir lock), the foreground exits nonzero with the reason.
Which commands accept --background:
runandresume— the only long-lived terminal-bound commands. Cluster runs go throughrun --profile <name>and therefore detach the same way:run --profile slurm --backgroundsubmits and polls the cluster jobs entirely inside the background process.dry-runintentionally does not: a preview is fast and read-only — keep it in the foreground.- Other commands (
pull,test,export,ai, …) are either quick or one-shot; run them before the background run and keep them foreground.
Monitoring a background run works exactly like a foreground one: poll
oxo-flow status .oxo-flow/checkpoint.json (or status --timing), read
the run log, and check the report snapshots in .oxo-flow/reports/. For
scripted supervision, oxo-flow wait blocks until the run
finishes and returns a structured exit status (0 success, 1 failed or
incomplete, 3 timeout, 4 no run found) — the native replacement for a
poll-for-output-files shell loop. Stop a background run with kill <pid>
(Unix) or taskkill /PID <pid> — the pid is in the summary line and
(while the run is in flight) in .oxo-flow/background.pid.
Notes:
- The global
--jsonis rejected with--background: the foreground only launches the detached child, so there is no run summary to emit. Drop--jsonor run in the foreground to get the machine-readable summary. - Bundle runs need an explicit
--workdir(the bundle's extracted directory is created per-process, so it cannot be predicted from the foreground invocation) and--yes— a detached child cannot prompt.--workdiris also what bundle discovery consults as a fallback: asample_pattern/pairs_patternthat matches nothing inside the extraction is retried against the workdir. - The detach flag is a launcher flag, not a workflow setting: the child
re-parses the remaining argv, so flag ordering rules (command flags
before
KEY=VALUEoverrides) apply to the whole command line as usual.
Checkpointing and Resuming#
oxo-flow automatically persists execution state to a checkpoint file after every rule completion. This enables:
- Resuming failed runs: If a job fails or is interrupted, simply run
oxo-flow runagain. The engine will skip all rules already marked ascompletedand resume from the first pending task. - State Inspection: Use the
oxo-flow statuscommand to view the progress of a run from its checkpoint file.
A rule whose latest execution record is a non-terminal running state
(a crash or kill while the rule was in flight) never passes the freshness
gate: on restart the engine re-runs it instead of trusting the
partially-written outputs, so a killed job's half-finished outputs are not
mistaken for finished work (issue #1083). Rules with no checkpoint entry at
all keep their historical freshness semantics — output existence alone
decides, as before (issue #77).
Variant checkpoints (--checkpoint)#
The default checkpoint is per workdir (<workdir>/.oxo-flow/checkpoint.json),
so running a pipeline.smoke.oxoflow variant next to pipeline.oxoflow in
the same directory shares one file. The engine detects the workflow
identity mismatch and ignores the other workflow's completed/failed records
— correct, but the variant runs are then silently non-reusable. Pass one
checkpoint per variant to make each first-class:
oxo-flow run pipeline.smoke.oxoflow --checkpoint .oxo-flow/smoke.checkpoint.json
oxo-flow resume .oxo-flow/smoke.checkpoint.json # resume the smoke variant
oxo-flow status .oxo-flow/smoke.checkpoint.json # inspect it
An explicit path wins over the workdir-derived default for load and
save. Relative paths resolve against the process working directory, like
every other --checkpoint-style argument.
Metadata Directory#
By default, checkpoints are saved in a hidden .oxo-flow/ directory located in the same folder as the workflow file.
- Filename:
checkpoint.json(the name is always the same regardless of workflow name)
Rule run records in the checkpoint#
Each executed rule carries a rule_runs entry in checkpoint.json with the
expanded command, the process exit code, a bounded stdout/stderr tail, the
resolved
report caption, and (since the cancellation-semantics change) the terminal
status ("failed", "cancelled", "timed_out", …), the
skip_reason when the rule did not run to completion, and the
terminating signal when the process died to one. The three newer
fields (status, skip_reason, signal) are optional and absent in
checkpoints written by older engines — consumers must infer legacy records
as before (failure when the rule is in failed_rules). Each entry also
persists the execution started_at and finished_at
timestamps, so status consumers can report a node's start alongside its
duration for both completed and failed rules
(see the web API's Run Status);
both are absent in legacy checkpoints and for rules whose execution never
started.
Timestamp semantics: the engine stamps
enqueued_at when an execution attempt is submitted and stamps
started_at only after the rule has acquired its resources — the
moment actual execution begins. Their difference is the time the rule
spends waiting in the resource pool (same split as the cluster driver's
queue_wait_secs), and all wall-time reporting (duration_ms, the
report's Benchmarks Wall Time column, benchmarks.wall_time_secs) is
measured from started_at, so a rule parked behind busy resources never
inflates its reported runtime. The checkpoint persists these timestamps in
the rule_runs entry itself (alongside the resulting wall time via
benchmarks.wall_time_secs), and cluster runs additionally persist the
split per job as queue_wait_secs in jobs/<rule>/status.json (see
What a finished job records). enqueued_at
remains in-memory only.
The tails are truncated to the last 64 KiB per stream by default and
prefixed with a … marker when cut. Set OXO_FLOW_OUTPUT_TAIL_BYTES to
change the size (bytes; an invalid or zero value falls back to the default
with a warning), e.g. OXO_FLOW_OUTPUT_TAIL_BYTES=10485760 (10 MiB) for
error-heavy tools whose message scrolls past the tail. The in-RAM capture
window widens to match when the setting exceeds its 1 MiB floor — budget
roughly 2 × (1 MiB + tail) of RAM per running job. Tails are
bytes-bounded, so a multi-byte UTF-8 cut lands on the nearest character
boundary (rounding down); older checkpoints that only carry the legacy
2048-character tails remain readable (the new size applies from the next
recorded run onward).
Declared log files vs captured tails#
A rule's declared log path is separate from this capture: the engine
creates the log's parent directory, but the file only gains content when
the rule's shell redirects into it (2> {log}) and the tool writes to
stderr. Many bioinformatics tools self-log to their own location instead
(e.g. a logs/<date>.txt file they manage themselves), leaving the
declared log empty — an empty declared log never means no diagnostics
were captured; the tails above are persisted in rule_runs regardless.
When a rule succeeds, its declared log is empty (0-byte file or empty
directory), and both captured tails are empty too, the run emits one
Info-level hint that the tool likely self-logs elsewhere — quiet tools
with output in any of the three stay silent.
Workflow version in the checkpoint#
When the workflow file lives inside a git repository, oxo-flow records the
repository's HEAD commit SHA in the checkpoint (workflow_git_sha) at
run start. Every result set is therefore auditable to the exact workflow
version that produced it: oxo-flow provenance verify prints the SHA, and
report snapshots carry it through the checkpoint they embed.
The lookup is best-effort — running outside a git repository simply omits the field and never fails the run. Commit your workflow before running to get fully version-audited results. See Workflow Versioning for the full model.
Include resolutions in the checkpoint#
Every [[include]] that pins a git repository (repo + ref) records
the exact commit SHA its checkout resolved to at parse time, in the
checkpoint (include_resolutions, keyed repo@ref path → SHA). Branch
and tag pins re-fetch on every activation — the workflow file alone cannot
prove which module content a past run actually used — so this per-include
SHA is the evidence a run/resume pair needs that the modules did not drift
between the two. The field is absent for workflows without repo includes,
and recording it is best-effort (an unreadable checkout logs a warning and
records nothing, never failing the run). The run report's Provenance
section lists one row per include. See
Workflow Versioning for the full model.
Run logs#
Every run — and resume, which re-enters the same path — archives its own
log under the workdir:
.oxo-flow/logs/oxo-flow.logis always the latest run; on each new run the previous log rotates to.oxo-flow/logs/oxo-flow.log.1, then.2, … up to.9(the oldest backup is deleted).--log-file PATHoverrides the location; the same rotation applies toPATH.- The log header names the exact workflow version that produced the record:
timestamp, oxo-flow version, full command line, workflow path,
workflow_name,workflow_version,git_sha(the repository HEAD recorded by the run — see Workflow Versioning), and workdir. - The engine's tracing stream (rule start/finish verdicts, warnings, errors) is written into the log alongside stderr. Logging is best-effort: if the file cannot be opened or written, the run continues with stderr-only logging and a warning. Dry-run remains stderr-only.
Report snapshots#
After every run (and resume) — unless --no-report-snapshot is given —
oxo-flow writes a JSON report snapshot of the final checkpoint:
.oxo-flow/reports/report-<UTC timestamp>.json— the full report data model, so a run leaves a machine-readable report behind with no reporting step needed (a-Nsuffix is used when two snapshots land in the same second).oxo-flow/reports/index.json— a JSON array of{generated_at, workflow, checkpoint, report}entries, kept sorted bygenerated_at; the last entry is the newest snapshot
Snapshot failures are warnings — a reporting hiccup never fails a run.
oxo-flow report with no -o writes auto-discovered reports into the
same .oxo-flow/reports/ directory.
Config changes and precise invalidation#
The checkpoint records a snapshot of the effective config values and a structural fingerprint of every completed rule. On each run, oxo-flow compares them against the current workflow:
- Changed config keys invalidate exactly the rules that reference them plus their DAG downstream; unrelated completed rules keep their checkpoint records. How a rule references a key decides the verdict:
- A key interpolated into
shell,script,input/outputpaths,envvars, orparams({config.<key>}) always invalidates — the new value bakes straight into the command or paths. - A key referenced only inside a
whencondition invalidates only when the condition's truth value actually flips under the new config. A toggle that leaves the gate true (or false) — e.g.(config.refine_bins || config.run_checkm)switching which term is true — reuses the completed rules instead of re-running hours-long chains whose inputs and commands are identical. Flipping a gate in either direction still invalidates: the engine pre-marks completed rules without re-evaluatingwhen, so only plan-time detection can retire a now-skipped producer or cascade a newly-activated one to its consumers. - Edited rule definitions (shell, inputs, outputs, environment, …) are
caught by the per-rule fingerprint and invalidate the same way. A
file-backed environment spec (pixi manifest, conda/mamba YAML, venv
requirements) is fingerprinted by content, not just by the path the
rule names — pixi specs also cover the sibling
pixi.lock, since a lock re-pin changes exactly which packages the environment materializes. Editing any of these files in place invalidates the rules that reference them. Checkpoints written before the content tagging lack these digests, so the first run after that upgrade re-runs rules with file-backed environments once (the same safe-but-costly bootstrap as the config-tracking baseline below). samples_list/samples_<group>are engine-injected and never trigger invalidation, so toggling--samplesbetween runs stays cheap. This covers rules whose input list is baked from an injected key (expand_inputsoverconfig.samples_list, a gather-rule pattern): their rule fingerprint and input manifest change with the selection, and oxo-flow treats that change as non-invalidating. When such a rule is skipped because of a selection change,runprints a warning naming the rule — its outputs still reflect the previous run's full sample set, and--rerunis the documented way to regenerate them under the new selection. The exclusion is narrow and verified: the rule's other definition fields must be byte-identical, and the current input file set must be exactly reproducible from the injected value with unchanged content — any genuine edit still invalidates. Checkpoints written by older binaries lack the input-excluded fingerprint, so the first subset run after an upgrade may re-run such a rule once (safe default).
# min_quality is interpolated by fastp_trim; only it and its downstream re-run.
# A flag toggled in a when-only rule whose gate stays true re-runs nothing —
# the run reports those rules as reused instead.
oxo-flow run pipeline.oxoflow min_quality=30
Config change:
min_quality: 20 → 30
when-condition unchanged, reused: bin_classify
→ invalidated 3 (1 directly affected), re-running 3/10 this run, skipping 7
⊝ index_reference (already completed)
Running: fastp_trim
…
When a when gate closes at execution time, the rule is reported under
Skipped (when condition false) with a ⊘ marker (or ⊝ in the
config-change reuse view) — a gated-off rule is a skip, never a failure
(see status for the checkpoint-side
display, including legacy checkpoints that carry stale failed_rules
entries).
Rules changed by a transformed split set (transform.split.values_from)
are detected through their baked input lists, so chunked rules re-combine
correctly when the split values change.
Limitations (correctness-first, documented deliberately):
- Changing
threads/memory(performance knobs) does not invalidate results; changing anything else in a rule — including shell comments — does. - A checkpoint written before config tracking was introduced is adopted as
a one-time baseline: the first post-upgrade run reuses everything and
records the snapshot; changes made before that baseline cannot be
detected. The same applies to the per-rule
whenverdicts behind gate-aware reuse: a pre-verdict checkpoint invalidates when-referencing rules once on the first changed-key run, records their verdicts, and reuses stably from then on. - Config values declared
sensitiveare stored in the snapshot as SHA-256 digests, never as plaintext. - Concurrent
oxo-flow runinvocations on the same workdir are prevented by an exclusive lock on.oxo-flow/lock(held for the whole run). The second invocation fails with a clear error instead of silently racing oncheckpoint.json; the lock releases automatically when the first process exits (even on a crash).cleanrefuses to delete while a run is active unless--forceis given.
Input changes and manifest invalidation#
The checkpoint also records an input manifest for every completed rule: the file set its inputs resolved to at completion time, with each file's path, size, and modification time — plus a content hash for files up to 64 MiB. Entries also record whether the file is a symlink and, if so, its recorded target. On every run, oxo-flow re-resolves the inputs and compares:
- Literal glob inputs (
data/*.txt) detect added and removed files — a new file matching the glob rebuilds the rule (and its DAG downstream). - Directory inputs (e.g.
input = ["results"]— a path that resolves to a directory) are listed recursively, so files added or changed anywhere inside invalidate the rule.Dirinputs with apatternfilter track only matching files. - Plain file inputs are content-addressed up to 64 MiB: a same-size
rewrite is detected even when the mtime is preserved, and a mere
touchno longer invalidates. Larger files compare size and mtime, closing the same stale-reuse hole for ordinary files. expand_inputs-baked inputs whose producer iswhen-gated off under the active config are tolerated as absent: the manifest records the existing subset instead of erroring, so a no-op rerun does not invalidate the rule and cascade its producers. If a later config change produces the missing files, the manifest then mismatches and the rule invalidates — the flip is still caught.
When an invalidation is detected, the run output spells out WHAT changed,
one line per affected file. Each line reads
path: <old size> @<old mtime> → <new size> @<new mtime> followed by
these labels:
(content <h1> → <h2>)— the content hash moved (files up to 64 MiB are hashed); shown alone when a same-size/same-mtime rewrite is caught by hash comparison.(TRUNCATED)— the file collapsed to 0 B: the vendor-data incident shape a write through a symlinked input path produces (the shell truncates the target before the command's bytes land).(removed)/(added)— the file disappeared from, or appeared in, the tracked set.[symlink]— the path resolves through a symlink on the recorded or the current side (recorded symlink facts are display-only and never invalidate by themselves).
For example (from the verification of this feature):
· raw/S1.fq: 22 B @2026-10-27T03:08:41Z → 0 B @2026-10-27T03:08:42Z (content sha256:b29f0d4b37f… → sha256:e3b0c44298f…) (TRUNCATED) [symlink]
A pure symlink-target flip with unchanged size/mtime/hash produces no diff line at all (target facts are display-only); the target surfaces only in the first-seen notice below.
A run also prints one first-seen notice per symlinked input —
ℹ input <path> is a symlink → <target> — collected across all completed
rules, deduplicated and sorted, whether or not anything invalidates. A
rewrite of identical size/mtime through a symlink leaves the manifests
matching (so nothing re-runs) but still names the path, because writing
to a symlinked input path writes THROUGH to its target — the exact
hazard behind irreversible data loss in real pipelines. The notice fires
once per run; it is not gated on the file changing.
- Missing external source inputs fail fast before any work starts:
a required input that no rule in the DAG produces and whose concrete
files are absent aborts the run before scheduling with
workflow is missing required source input(s) — no rule produces them, naming each affected rule and pattern (exit non-zero). Previously a fully-completed checkpoint plus a deleted raw data file could exit 0 from stale checkpoint credit ("re-running 0 producer rules"); that silent-success hole is closed. The gate never fires for generated intermediates (they have producers and are governed by the manifest cascade below),[[references]]outputs (built later in the same run), inputs discoverable through the workflow'ssample_patterndirectory, when-gated-off rules, zero-fan-out scatter rules, oroptionalrules.
Invalidated rules bypass the executor's freshness gate and re-execute even when their outputs exist and look recent; downstream rules that consumed their outputs rebuild in the same run.
Limitations (correctness-first, documented deliberately):
- Detection is a hybrid policy: files up to 64 MiB are content-hashed
(
sha256), so a same-size rewrite is detected even when the mtime is preserved — and a meretouchno longer invalidates. Larger files (multi-gigabyte intermediates) keep the size+mtime policy, likemakeand Snakemake's default mode: hashing every input on every run would cost O(total input bytes), prohibitive for bioinformatics-scale files. A large file rewritten with identical size and a deliberately preserved mtime is not detected; use--rerunto force. Legacy checkpoints (entries written before hashing) keep comparing size+mtime instead of invalidating everything once. - Only one glob level is tracked per input pattern; brace-expansion
patterns (
*.{txt,log}) are not expanded by the manifest scanner, so their matched set is not tracked (the shell still expands them at runtime). - Symlinked directories are recorded as single entries, never traversed (cycle-safe), so changes inside a symlinked directory are tracked at the symlink's own metadata level. Symlinked files are recorded with their symlink-ness and lexical target (see the first-seen notice above); the comparison follows the link's resolved content — a retarget to an identical-content file keeps the rule valid, while a write through the link invalidates like any content change.
- Rules that clean up their inputs at the end of a successful run
(
transform.cleanup = truechunk consumers) are not manifest-tracked: their inputs are engine-managed intermediates governed by the upstream-cascade invalidation instead. - Optional rules with permanently-absent inputs (
optional = trueoroptional = "any"): when a declared input cannot be resolved because its producer was never instantiated (aninput_groupspattern that matched no files, or an endedness-filtered branch that never ran), the snapshot skips that entry and records a manifest for the inputs that do exist — the rule stays skipped/completed across runs instead of cascading a full re-run every invocation. The same grace applies during invalidation detection: a missing input with no producer in the DAG is absent by design for an optional rule, while a required rule with a missing input still invalidates (a deleted raw data file must re-run). A failed manifest snapshot is reported on stderr and in the run log (⚠ could not snapshot inputs of rule …) instead of being silently dropped. - A checkpoint written before input tracking was introduced adopts the current file set as a one-time baseline (the first post-upgrade run reuses everything); changes made before that baseline cannot be detected.
Temporary rules (temporary = true)#
Pipeline intermediates are usually the largest files in a run (per-sample
BAMs, unsorted alignments) and often useless once every downstream rule has
consumed them. Marking a rule temporary = true deletes its outputs after
a fully successful run — once every dependent has completed — and
records a tombstone in the checkpoint:
This is checkpoint-aware, not a blind delete:
- A plain re-run skips the rule like any completed rule — the output is not regenerated just because it is missing.
- When a dependent actually needs the outputs again (its own outputs were deleted, its inputs changed, or the config changed), the producer is regenerated first (lazy cascade-up), then the dependent runs — the same order as the original run.
- Failed runs keep the outputs (the deletion only happens when nothing failed), so debugging a broken run never has to re-fetch intermediates.
- Leaf rules (no dependents) keep their outputs — there is no downstream work for them to enable.
- With an
[engine]section present, the deletion is disk-gated: it only fires when free space at the workdir is belowreclaim_free_diskat run end. With disk comfortable, the outputs are kept — so the next run skips the completed producer instead of re-materializing it — and one line explains the choice:
ℹ free space 39876544MB ≥ reclaim_free_disk 5120000MB — temporary outputs kept (disk pressure alone triggers re-materialization; remove the [engine] thresholds to always prune)
If free space cannot be measured (non-Unix platforms), the outputs are
kept as a precaution. Workflows without [engine] thresholds keep the
historical unconditional deletion.
↻ temporary outputs needed again — re-running 1 producer rule: trim_S1
…
⊘ temporary outputs deleted for 'trim_S1' (1 file, regenerated on demand)
Dry-run predicts exactly when a tombstoned rule will regenerate
([rerun: upstream of X]) — see
Checkpoint-aware rerun preview.
Disk-pressure monitoring ([engine])#
Large pipelines can fill a disk mid-run. Declaring an [engine] section in
the workflow makes the engine measure the working directory's free space at
startup, at every scheduling round, and every 60 s while work is parked:
[engine]
min_free_disk = "50G" # floor — entering Hold below it
reclaim_free_disk = "100G" # optional reclaim trigger; defaults to min_free_disk
The response ladder:
- Reclaim (
min ≤ free < reclaim): completed rules' temporary outputs are tombstoned and unlinked mid-run — the same checkpoint-aware reclaim astemporary = truecleanup, but while the run is still going. Only when every dependent is completed and its own outputs still exist (a dependent whose outputs are missing will re-run and needs its inputs). The narration line is⊘ reclaimed temporary outputs of '<rule>' …. The same floor gates the success-path deletion oftemporary = trueoutputs (see Temporary rules) — with disk comfortable at run end they are kept, not pruned. - Hold (
free < min): new rule dispatches are withheld; in-flight rules continue. If nothing is in flight and nothing is reclaimable, the run parks and re-measures every 60 s. - Abort (Hold persists ~2 parked rounds ≈ 2 min): the run checkpoints
and fails. Free space manually (delete intermediates or
oxo-flow clean), thenoxo-flow resume— or lowermin_free_disk.
Protected outputs (protected_output) are never reclaimed, and the final
(leaf) rule's outputs always survive. See
[engine] in the workflow format reference
for the full field table and edge cases.
Tuning knobs (both fall back to their defaults when unset, invalid, or
0, so a typo can never disable the abort):
OXO_FLOW_DISK_TICK_SECS— seconds between measurements while parked (default 60).OXO_FLOW_DISK_HOLD_ROUNDS— grace park rounds under Hold before the run aborts (default 2).
Known boundaries:
- Mid-run reclaim of fan-out producers (
output_patternrules with per-instance values) fires only with full instance attribution: the recorded instance combos fromcheckpoint.output_pattern_domainsare expanded to concrete paths, and every combo must resolve and every consumer (detected structurally — pattern producers register no DAG edges) must be completed before any deletion. If any instance combo cannot be attributed, the whole rule is skipped with a once-per-rule warning — no partial tombstone is written. The post-run sweep covers them regardless (see below). - Free space is measured at the working directory's filesystem, which is volume-level (statvfs): thresholds apply to the whole volume, not to a quota.
- Keep scratch-heavy tools off the measured volume only if they are not cleaned by their rules; outputs declared by rules are always accounted for by the ladder.
Protected data trees (protected_data_trees)#
Vendor-delivered inputs (a sequencer drop, a reference bundle) must not change silently while a pipeline is running against them. Declaring
makes the engine treat those workdir-relative trees as protected:
- First observation (first run/plan/dry-run after the key appears) records a baseline manifest in the checkpoint — per-file size + mtime, plus a sha256 content hash for files up to 64 MiB.
- Every later plan/dry-run/run diffs the tree against the baseline and
prints a loud
✗ PROTECTED TREE DRIFT detected in N file(s):section with one line per changed/removed/added file (a truncated file is labeled(TRUNCATED), content edits carry the old→new hash, symlink changes are labeled[symlink]). runrefuses to execute on drift:
Error: protected data tree drift detected (1 file(s) changed since baseline):
raw/S1.fq: 4.0 B @2026-10-27 10:00:00 → 0 B @2026-10-27 10:05:00 (TRUNCATED)
Restore the files, or pass --allow-protected-drift to run anyway (the
drift is still printed; the baseline is NOT updated — a later clean run
re-reports it until the tree is restored or the checkpoint's baselines are
reset by deleting the entry). To re-baseline after an intentional change,
remove the protected_tree_baselines entry from .oxo-flow/checkpoint.json
(or the checkpoint file itself) — the next invocation adopts the current
tree content as the new baseline.
- Trees that do not exist yet are adopted only once they appear — populating
a tree is normal data arrival, not drift.
- dry-run reports the same section read-only (it never persists the
baseline and never fails on drift — only run gates).
This is a detection surface, not an access control: out-of-band writes are
noticed within one invocation, not prevented. Combined with
--provenance checksums it catches damaged vendor data before any rule
consumes it. For a one-shot audit of a whole tree against the vendor's own
md5.txt sidecar (post-incident audits, post-recovery verification), use
oxo-flow verify-data. On a web-served instance the same
drift state is available per run as a read-only HTTP preview:
GET /api/runs/{id}/protected-drift (see
Web API reference) —
it never records baselines, so it is safe to poll before overriding a gate
with allow_protected_drift on POST /api/runs.
Forcing Execution#
To bypass checkpoints and re-execute rules that have already completed, use
oxo-flow clean to remove outputs and the checkpoint file
before running, or --rerun to re-execute this run's rules while keeping
checkpoint records for rules outside the run.
Checkpoint re-entry (checkpoint = true)#
A checkpoint rule discovers new values at runtime: after it completes, the
engine reads its checkpoint_manifest, merges the declared samples, and
executes the newly created rule instances in the same run. The checkpoint
records each re-entry (reentries), so a resume replays it and reconstructs
the same plan; invalidating the checkpoint rule revokes its samples until it
re-runs. A missing or unparsable manifest fails the checkpoint rule. See
Workflow Format for
the config surface.
Exit codes#
| Code | Meaning |
|---|---|
0 |
Success — the run finished with zero required-rule failures |
1 |
Validation or parse failure — the workflow file could not be read, parsed, or validated |
1 |
Run failure — a required rule failed (a plain --keep-going failure summary still exits non-zero; only required = false failures leave the exit code at 0) |
130 / 143 / 129 |
Interrupted — SIGINT / SIGTERM / SIGHUP stopped the run. In-flight rules are recorded as interrupted and --json reports status "failed"; this holds on the local and run --profile cluster paths, and even when every interrupted rule is required = false (an interrupt is never reported as success) |
Validation and run failures both exit 1; the distinction is in the stderr
diagnostics (a parse error names the workflow file, a run failure lists the
failed rules). Machine consumers that need to separate the two should use
--json and read status. Failure-specific exit codes on
other commands are documented per page — see report
and test.
Notes#
- The workflow file is optional; if not specified, auto-discovery searches for
main.oxoflowfirst, then any*.oxoflowfile alphabetically - If no
.oxoflowfile is found, an error message suggests runningoxo-flow initto create one - The DAG is built and validated before any rules execute
- Rules are executed in topological order; independent rules may run in parallel up to the
-jlimit - If
--keep-goingis not set, execution stops at the first failure --keep-goingchanges scheduling, never the verdict: it runs the remaining rules past a failure, but the run still exits non-zero when arequiredrule failed — scripts and the web UI (which classifies by exit code) see the truth. Onlyrequired = falsefailures leave the exit code at 0.- The
--retryflag re-runs failed jobs up to N times before marking them as failed - A timeout of
0means no timeout - Resource constraints (
threads,memory) in rules are checked against available resources before execution - Pre-flight budget check: When
--max-threads/--max-memoryis explicitly set — or the workflow declares[resource_budget] max_threads/max_memory(issue #1080) — the engine checks all rules before execution starts. Any rule whose requirements exceed the effective cap is reported immediately (fast-fail), preventing mid-pipeline failures. The workflow budget is a hard, versioned cap: it clamps a larger CLI request, a tighter CLI request still wins, and a silent CLI inherits it. - Backend preflight: before any rule executes, every unique environment spec of the pending rules is validated once — a missing backend (
conda,mamba,pixi,docker,singularity,modules) aborts the run with one collapsed entry per root cause and "no rules were run" (see Backend preflight). - Auto-detected capacity is soft: Rules whose declared requests exceed the machine's detected threads/memory are not rejected — the declared request is the tool's upper bound (often an upstream HPC label), not a scheduling requirement. The engine warns and clamps the pool reservation, so an over-capacity rule runs alone (serialized) instead of blocking the workflow.
- CPU-oversubscription warning: the engine compares each scheduled parallel wave's worst-case demand (
min(-j, wave width) × heaviest rule threads, both clamped to the effective pool ceiling) with the detected CPUs and warns at ≥ 80 % — the analogue of the memory clamp warning. The estimate covers only the rules the run will execute (-t/--modulefiltering included). A ceiling that leaves headroom silences it; a cap at or above the host still warns. See CPU Oversubscription Warning. - Deadlock detection: If pending rules remain while nothing is running and none can become ready (typically an upstream failure), the engine reports
Deadlock detected: N rules stuckwith the stuck rule names. Resource waits cannot deadlock: over-capacity requests are clamped, and explicit budget violations fail fast before any rule runs. - Target-aware execution: The
-tflag supports prefix matching —-t almatches all rules whose names start with "al". Use this to run a subset of the workflow, similar tomake <target>. Only the named targets and their transitive upstream dependencies are executed; downstream rules are excluded. - Default targets: rules marked
target = trueare whatrun/dry-runbuild when neither-tnor--moduleselects anything — the marked rules and their transitive producers. With no marked rule the full DAG runs (backward compatible); an explicit-t/--modulealways wins. - Abort cancellation semantics: with fail-fast aborts, rules that were merely killed by the abort (signal death, no exit code) are recorded as cancelled, not failed — they are excluded from
failed_rulesand the failure counts, and narrated like skips. A sibling that failed on its own merit (exit code present) keeps its failure record. The checkpoint's per-rulerule_runsentries distinguish the two via theirstatus/skip_reason/signalfields. - Setting
--max-threads 0or--max-memory 0auto-detects system resources - Cgroup-aware auto-detection: on Linux the auto-detected memory ceiling
is clamped to the tightest cgroup v2
memory.maxapplying to the engine process — a container cap, a systemd user slice (user@.servicewithMemoryMax=), or asystemd-run --user -p MemoryMax=…scope. Host RAM figures ignore these limits: on a 1007G box whose user slice caps sessions at 512G, scheduling against 1007G commits more memory than the kernel can back inside the cgroup and the OOM-killer tears the run down (exit 137 cascading into a fail-fast teardown of healthy rules). The engine logs acgroup memory limit is below the detected host memorywarning with both figures when the clamp bites. The same clamp applies to the{effective_memory_mb}placeholder and to container--memoryceilings. For long cohort runs that park at the cap for hours, pair this with--retryso a transient OOM does not kill the whole run. The CPU side is cgroup-aware too: the detector (num_cpus) honours affinity/cpuset and cgroup v1/v2 CPU quotas (container --cpus, systemdCPUQuota=,cpu.max), so the auto-detected thread ceiling,{effective_threads}, and the oversubscription warning already reflect a quota (issue #1080). - Environment setup is performed automatically before first use of each environment (conda, pixi, docker, singularity, venv)
- Use
--skip-env-setupwhen environments are pre-built to avoid redundant setup - Use
--cache-dirto persist environment setup state across runs for faster startup (reclaim stale entries withoxo-flow env gc— the old post-run aging sweep was retired)