Skip to content

oxo-flow run#

Execute a workflow.


Usage#

oxo-flow run [OPTIONS] [WORKFLOW] [KEY=VALUE]...

Arguments#

Argument Description
[WORKFLOW] Path to the .oxoflow workflow file. Optional — if not specified, auto-discovery searches for: (1) main.oxoflow in current directory, (2) alphabetically first *.oxoflow file in current directory.
[KEY=VALUE]... Direct config overrides (see Config overrides)

Config overrides#

KEY=VALUE and --KEY=VALUE work for any key; the --KEY VALUE space form also works for any key present in [config] (plain key = "value" entries included) — a key absent from [config] is a hard error (an unknown flag would otherwise be indistinguishable from a mistyped option; a typo'd --key=val after the workflow still sets a config var, so prefer the = forms for undeclared keys).

An override key that differs from a declared key only by hyphens vs underscores is canonicalized onto the declared spelling — --api-token=x sets the declared key api_token (and inherits its sensitive masking); keys matching nothing declared keep the literal spelling.


Options#

Option Short Default Description
--jobs -j 1 Maximum number of concurrent jobs
--keep-going -k — Continue execution when a job fails
--json — — Emit a machine-readable JSON summary on stdout after the run (see JSON output)
--workdir -d Workflow file's directory Working directory for execution
--checkpoint — <workdir>/.oxo-flow/checkpoint.json Checkpoint file for this run. Use one path per workflow variant (smoke/test/full) that shares a workdir so each variant resumes independently (see Variant checkpoints)
--log-file — .oxo-flow/logs/oxo-flow.log in the workdir Write the run log to a custom path instead (relative paths resolve against the workdir; previous logs rotate to PATH.1 … PATH.9)
--target -t All rules Run only specific target rules
--from-rule — — Re-run from RULE downstream: invalidates the transitive-dependent closure of RULE in the checkpoint, then runs RULE and everything downstream of it (issue #1017). Unlike --target (upstream closure), --from-rule propagates downstream — stale downstream outputs are invalidated before the freshness gate so they re-execute with the new upstream results
--module — — Run one include module and the producers of its declared inputs (repeatable; unions with --target). Module names are the include's name field or its file stem (see Partial module runs)
--retry -r 0 Number of times to retry failed jobs
--timeout — 0 (disabled) Timeout per job in seconds, or a duration like 1h/30m
--max-threads — 0 (auto-detect) Maximum CPU threads available for execution. A workflow [resource_budget] max_threads caps the effective value (see Resource budgets)
--max-memory — 0 (auto-detect) Maximum memory in MB available for execution. Auto-detect is cgroup-aware on Linux (clamped to the tightest memory.max — see Notes); a workflow [resource_budget] max_memory also caps the effective value. Set it explicitly (e.g. --max-memory 430000 --retry 2) when the clamp misreads a deliberately generous slice
--skip-env-setup — — Skip environment setup (assume environments are ready)
--skip-ref-build — — Skip automatic reference/index building (assume pre-built)
--cache-dir — — Directory for caching environment setup state (a small readiness manifest that rebuilds itself on demand). The retired post-run cache_max_age_days aging sweep warns and points at oxo-flow env gc, which reclaims the shared env store (OXO_FLOW_ENV_STORE) — not this directory
--resume-failed — — Resume only failed rules from a previous run. Also clears stale checkpoint entries for rules whose when gate recorded false (they are skips, not failures); status reports them as skipped, not failed
--profile — — Execution profile name, loaded from profiles/<NAME>.toml or profiles/<NAME>.oxoflow (see Execution profiles)
--max-submitted — profile default (50) Cluster jobs in flight at once (overrides the profile's max_submitted)
--provenance — — Track output file checksums for later verification
--arg — — Legacy form: set a workflow config value (KEY=VALUE). Repeatable. See [config] in workflow-format
--bundle — — Execute from a published bundle (.tar.zst or .tar.gz). Extracts, verifies checksums, shows resource requirements, and prompts for confirmation
--yes — — Skip the confirmation prompt when running a bundle (required in non-interactive sessions: CI, scripts, redirected input, or --json)
--ai-recover — — Enable AI error recovery on rule failure (see AI error recovery)
--no-ai-recover — — Disable AI error recovery for this invocation, including the workflow's [ai] enabled auto-detect. The verified-repair loop passes this flag to its verification rerun so a fixed run can never start another repair cycle (see AI error recovery)
--ai-max-retries — 1 Maximum AI recovery attempts when a --ai-recover call fails (only a failed call is retried). This bounds recovery attempts; the generation-side --ai-max-retries on template (default 10, bounding orchestrator rounds) is a separate knob
--samples — all discovered samples Sample selection: @path replaces the workflow's samples from a samplesheet, +@path appends (same-name groups merge, new groups added); names filter (or declare when the workflow ships no samples), first:N (pilot) and ready (samples whose entry inputs are complete) filter. Repeatable, comma-separated
--rerun — — Force re-execution of this run's rules (ignore up-to-date checks). Checkpoint records for rules outside this run are kept
--no-report-snapshot — — Skip the automatic report snapshot written after the run (see Report snapshots); resume has the same flag
--background / --detach — — Detach the run into a background process and exit 0 immediately (see Background runs (--background)); --detach is an alias
--allow-protected-drift — — Proceed despite protected-tree drift: when [engine] protected_data_trees baselines no longer match the tree on disk, run refuses to execute; this flag is the explicit operator override to run anyway (see Protected data trees)
--strict-envs — — Hard-fail on environment setup errors instead of using the graceful pixi stale-lock fallback (see Stale pixi.lock fallback) — for GxP-style deployments that must never run on a re-solved environment
--verbose -v — Enable debug-level logging
--quiet — — Suppress informational output — progress rendering, DAG listings, and per-rule completion lines are omitted; errors, warnings, and machine-readable output (--json) are unaffected
--no-color — — Disable colored output (also honors the NO_COLOR environment variable)

Flags the cluster driver does not honor

On run --profile, several run flags are parsed but not applied — the engine warns once per flag you actually passed (full table: Cluster flag surface). --provenance and --keep-going are honored there: checksums are recorded as each job settles, exactly like a local run; by default a required rule's failure cancels every in-flight job, while -k lets already-submitted independent jobs finish (dependents of the failed rule stay blocked) and the run still exits non-zero — use required = false for best-effort rules. Interrupts always cancel every in-flight job. Set a per-job walltime with the rule's time_limit (or the profile's walltime), and leave resource limits to the per-rule [resources] block and the [cluster] profile.

JSON output#

The document carries command, status, results counts, and resources. It is emitted on every exit path — completed, failed, and aborted (preflight failures, budget breaches, cluster runs, plain-failure aborts) — so --json consumers never get zero bytes. status is "completed" only when the run finished with zero required failures and was not interrupted; every other exit is "failed".

Human progress and result output on stderr is not suppressed — this flag appends the summary, it does not switch the run to machine-only mode.

resources lists every benchmarked rule (rule, status, wall_time_secs, peak_rss_mb, cpu_seconds, retries) — the same data as the report's Benchmarks table; it is empty on aborted paths and on cluster runs (use the report's sacct-based Resource Accounting section there).

When the verified-repair loop applied an AI fix and ran its verification rerun (see AI error recovery), the document also carries a repair object (applied, session_id, rerun_attempted, rerun_succeeded). On a verified repair (rerun_succeeded: true) the status is the normal "completed" — the rerun finished the workflow with zero failures — with the counters describing the final state (failed 0, blocked 0); repair is what distinguishes a verified-repair run from a clean pass.

Quick reference#

Common flags at a glance — every flag is documented in the Options table above and the sections below.

Goal Flags
Dry-run first / inspect the plan dry-run before run
Parallelism and failure handling --jobs, --keep-going, --retry, --timeout
Run part of the workflow --target / -t, --from-rule, --module
Resume after failure --resume-failed, or the resume command
Select samples --samples (see Incremental data arrival)
Force re-execution --rerun
Config values from the command line [KEY=VALUE]... positionals, --arg KEY=VALUE (see Config overrides)
Execution profile / cluster --profile, --max-submitted (see Cluster submission)
Resource budgets --max-threads, --max-memory, [resource_budget] (see Resource budgets)
Machine-readable output --json (see JSON output)
Output checksums --provenance
Run detached / wait for it --background / --detach, then oxo-flow wait
Quieter output --quiet, --verbose, --no-color

Output#

oxo-flow v0.32.0 — Rust-native bioinformatics pipeline engine
DAG: 5 rules in dependency order
  1. fastqc
  2. trim_reads
  3. bwa_align
  4. sort_bam
  5. call_variants
⠋ [00:15] [████████████░░░░░░░░] 3/5 ETA:0:00:42 (executing sort_bam)
  ✓ fastqc (2.1s)
  ✓ trim_reads (15.0s)
  ✓ bwa_align (42.3s)
  ✓ sort_bam (10.2s)
  ✓ call_variants (8.4s)

Done: 5 succeeded, 0 skipped, 0 failed
✓ 5 output files verified (118.3MB total)

A progress bar shows execution progress with:

  • Elapsed time
  • Current position / total rules
  • Estimated time remaining (ETA), plus the rule currently executing
  • The final summary line reports succeeded / skipped / failed counts

Module-grouped output#

Workflows composed from [[include]] modules group the plan listing and the --keep-going failure summary under module headers, so a composed pipeline reads as module → rule instead of one long flat list:

DAG: 6 rules in dependency order
  Preprocessing — 2 rules
    1. fastqc
    2. trim_reads
  Alignment — 3 rules
    3. bwa_align
    4. sort_bam
    5. markdup
  Main workflow — 1 rule
    6. report_summary
  • Rules from a module file ([[include]]) collect under their module's display title; rules declared in the main file land under Main workflow
  • Attribution is the same logic the metro graph export uses: a module::rule namespace prefix wins, then member matching; host rules that merely extend a member's name may land in that module's group (a display-only quirk — edges and execution are unaffected)
  • Group order follows first appearance in the execution order, and the per-rule numbers stay a single global sequence so plan position remains greppable
  • Workflows without modules keep the flat listing above byte-for-byte — grouping is opt-in by structure, not a flag

The failure summary groups the same way:

Failed rules:
  Germline (1):
    ✗ haplotype_caller — exit code 1

Blocked rules (did not run):
  Germline (1):
    ⊘ joint_genotyper — depends on 'haplotype_caller'
  Main workflow (1):
    ⊘ qc_report — depends on 'joint_genotyper'

Per-line shapes (✗ name — reason, ⊘ name — depends on 'dep') are unchanged, so tooling that parses run logs keeps working.


Examples#

Run with auto-discovery (when only one .oxoflow file exists)#

# No need to specify the workflow file
oxo-flow run

Run with main.oxoflow (priority discovery)#

# If main.oxoflow exists, it's automatically used
oxo-flow run

Run with explicit workflow#

oxo-flow run pipeline.oxoflow

Run a workflow from a repository (nextflow-style)#

A public git repository can be executed directly — no clone, no bundle:

# Default branch — the gh: prefix is optional for github.com repositories
oxo-flow run owner/pipeline

# Pinned to a git tag or branch (@ref selects a git ref — unlike `pull`,
# where @tag means a GitHub Release bundle)
oxo-flow run owner/pipeline@v0.32.0

# The explicit forms work the same way (gh: can always be kept)
oxo-flow run gh:owner/pipeline
oxo-flow run gh:owner/pipeline@v0.32.0

# Any git URL or local repository directory
oxo-flow run https://example.com/team/pipeline.git

# A local checkout via file:// (must be an existing directory)
oxo-flow run file:///path/to/pipeline

A bare owner/pipeline argument is interpreted as a GitHub repository unless it names something real on the filesystem: a path that exists, one starting with ., or one ending in .oxoflow is always treated as a local workflow path. Only a single-slash owner/name — optionally with @ref and a trailing .git — maps to https://github.com/owner/name.git. (So a missing data/pipeline.oxoflow keeps its normal "workflow file not found" error instead of attempting a clone.) An https://…/*.git URL runs that repository directly; file://<dir> runs an existing local checkout.

The repository is checked out into .oxo-flow/repos/<host>-<owner>-<name> under the current directory (e.g. .oxo-flow/repos/github.com-owner-pipeline; reused on later runs — delete the directory to force a fresh clone; the host and owner prefixes keep same-named repos of different owners or hosts from sharing a checkout), and the workflow file is auto-discovered (main.oxoflow first). Because the clone is a read-only cache, the working directory defaults to the current directory for repository runs — outputs, the checkpoint, and the workdir lock all land next to your data, not inside the clone. All other run semantics apply unchanged: --samples, --rerun, config overrides, --workdir, checkpoint-aware dry-run previews, etc.

For github.com clones the official URL is tried first, then the ghfast.top / gh-proxy.com mirrors automatically (see China Mirrors).

Custom working directory#

Relative paths in the workflow resolve against the workflow file's directory — the single resolution base. When --workdir differs from the workflow directory, the engine anchors workflow-shipped files so they stay valid from the execution cwd:

  • Environments: relative conda / mamba / pixi / venv_requirements file specs resolve against the workflow directory.
  • Rule scripts: workflow-relative script files render absolute (workflow-rooted) so rule shells — which run with cwd = the workdir — can execute them.
  • Workflow source files (zero duplication): declared rule inputs that no rule produces, plus interpreter script paths (python scripts/x.py), are materialized in the workdir as symlinks into the workflow directory at run start — the repo keeps the single copy of every source file. A pre-existing byte-identical workdir copy is replaced by a symlink (with a notice); a copy that differs is kept for the run and warned about (it may be a deliberate override). Repo paths a shell command mentions but no rule declares are reported as a note — declaring such a path as a rule input links it automatically.
  • Tool index sidecars (zero duplication): companion index files that tools auto-derive from a linked file's path by convention — samtools faidx/picard/GATK (ref.fa.fai, ref.fa.dict), bwa (ref.fa.amb/.ann/.bwt/.pac/.sa), minimap2 (.mmi), and the bgzip/tabix/htsfile index forms (.gzi, .tbi, .csi, .crai, .bai) — are linked alongside the file itself. Those paths never appear in a shell command, so static analysis alone cannot see them; the engine links every such sidecar that exists next to a linked workflow file, unless a rule produces the sidecar itself (a dedicated indexing rule keeps ownership). Tools driven through a bare index prefix (bowtie2 -x index) should declare their index files as rule inputs directly.
  • {input} and {log}: stay workdir-relative. Inputs are addressed from the run cwd (freshness/optional-input gates probe workdir-relative paths), and the engine creates log parent dirs under the workdir. Place workflow-consumed data under the workdir (or pass absolute paths).
  • input_groups disk scans: relative patterns scan the workflow directory and emit workflow-absolute paths into {input}.
  • Reference sources: [references].source resolves against the workflow directory; reference outputs still land in the workdir.
  • Outputs: stay workdir-relative — they are run artifacts and land under the workdir, matching the checkpoint and .oxo-flow state.

This keeps a workflow self-contained: for a shared, read-only workflow (a fixed reference resource), point --workdir at the analysis directory instead:

# Workflow lives in a central, read-only location; data and results
# belong to the current analysis directory
oxo-flow run /opt/pipelines/wgs.oxoflow --workdir .

# Outputs, .oxo-flow checkpoint, and lock all land in the workdir;
# dry-run/clean/report/resume accept --workdir too, and resume re-uses
# the workdir recorded in the checkpoint

Parallel execution#

oxo-flow run pipeline.oxoflow -j 8

Keep going on failure#

oxo-flow run pipeline.oxoflow -j 4 -k

Retry failed jobs#

oxo-flow run pipeline.oxoflow -j 8 -r 2

Run specific targets#

oxo-flow run pipeline.oxoflow -t align -t sort

Target names may be prefixes: -t align matches every rule whose name starts with align (including expanded instance names like align_cohort_S1). The closure walks the instantiated DAG — it includes each target plus every upstream rule that survives the plan-time when gating, and skips when-gated variants entirely: a variant whose when is false under the effective config never enters the execution set, its upstream is not pulled in, and consumers of its (never-produced) outputs are pruned too. Naming a when-gated-false target explicitly reports it and aborts with "nothing to run" instead of planning a run that cannot produce output.

The same when-aware closure applies to full runs (no --target) and --module runs: a consumer left without any surviving producer of an input group (all of its producers when-gated false, the files not pre-built on disk, and the group not optional = "any") is dropped from the plan with a note — rule '...' is unreachable — its upstream is when-gated false; removed from the full-run execution set — instead of failing mid-run on a file that was never produced. The when-gated rule itself stays in the plan and is counted as skipped exactly once by the executor. See when-skipped instances for how the consumer's baked expand_inputs list is cleaned in the same sweep.

Set a per-job timeout#

# 3600 seconds, or equivalently:
oxo-flow run pipeline.oxoflow --timeout 1h

Limit resource usage#

# Use only 16 threads and 32GB memory
oxo-flow run pipeline.oxoflow --max-threads 16 --max-memory 32768

A workflow can carry the same caps in [resource_budget] so every run of the file is bounded without an external wrapper (issue #1080):

[resource_budget]
max_threads = 16
max_memory = "32G"

Cache environment setup#

# Cache environment setup state for faster subsequent runs
oxo-flow run pipeline.oxoflow --cache-dir .oxo-flow/cache

The cache is not aged automatically — --cache-dir only selects where environment setup state lives, and that state is a small readiness manifest that rebuilds itself on the next run if entries go missing. To reclaim the bulk of environment disk usage, target the shared env store instead: run oxo-flow env gc (see env), which collects OXO_FLOW_ENV_STORE directories under its --min-age-days gate (default 90) so entries that recent runs may still reference survive.

Removed: cache_max_age_days

Older versions swept this cache after every run, keyed off the workflow config key cache_max_age_days. That mechanism was retired in favor of the explicit, store-aware oxo-flow env gc command. A workflow that still declares the key now triggers a one-time warning pointing at env gc; the key has no effect.

Skip environment setup (when environments are pre-built)#

oxo-flow run pipeline.oxoflow --skip-env-setup

Shared environment store (pixi/venv, opt-in)#

Pixi and venv environments can be materialized once per content key in a shared store (opt-in: set $OXO_FLOW_ENV_STORE to a non-empty path — location IS activation; unset keeps envs self-contained inside each workflow tree, exactly like before the store existed). When enabled, the workflow tree only holds a symlink into the store: ten checkouts of the same pipeline share one environment instead of re-downloading it ten times, and a deleted store dir heals itself — the next run detects the broken link and rebuilds. Clean up unreferenced environments with oxo-flow env gc. Conda, Docker, and Singularity environments keep their native cache layouts and are unaffected — the store is pixi/venv-only by design (see the env command for the rationale: conda name envs already share a central prefix, conda prefixes are not relocatable, containers use their own registries, modules come from the cluster).

Backend preflight: fail fast on a missing environment backend#

After DAG construction and before any rule executes, run validates every unique environment spec among the pending rules — once per spec (rule templates fan out one spec across N samples, so the spec set is tiny even for large cohorts). When a required backend binary is missing (conda/mamba/pixi/docker/singularity/modules), the run aborts before anything executes:

2 environment backends required by pending rules are unavailable; no rules were run:
  - conda: conda is not installed or not in PATH — required by 3 pending rules, e.g. `bwa_align` (environment: envs/alignment.yaml); install conda and ensure it is on PATH
  - pixi: pixi is not installed or not in PATH — required by 1 pending rule, e.g. `report` (environment: pixi.toml); install pixi (https://pixi.sh) and ensure it is on PATH

Rules sharing one broken spec, and distinct specs failing with the identical backend-missing message, collapse into a single entry — one root cause per line, with the pending-rule count, an example rule, the primary spec, and a mode-aware remediation hint. Nothing ran, so the --json summary still reports the run as failed with zero counts. See Troubleshooting — Environment Issues for per-backend checks.

The remediation depends on why the spec failed:

  • Manifest file not found (pixi declared by relative path): the spec is resolved against the workflow directory (the same base the executor uses), and the hint says to create the manifest at the resolved path or correct the rule's pixi path — installing the backend would not help when the file itself is absent. Other backends keep the install advice (a missing conda YAML surfaces only at task execution).
  • Backend binary missing (the manifest exists, or the backend cannot be located for another reason): the hint is the per-backend install advice (install pixi (https://pixi.sh) and ensure it is on PATH, and so on).

This is the run-time safety net. validate and plan/dry-run preflight pixi manifests up front (error E019) so the failure surfaces before you commit to a run; run keeps the check because manifests can also be created between validate and run.

Stale pixi.lock fallback#

A committed pixi.lock pins exact package builds. When an upstream channel later removes one of those builds (bioconda does this occasionally — old builds disappear), a lock-based pixi install fails with a fetch error (failed to fetch … 404 Not Found) and every run using that environment fails until the lock is refreshed by hand.

run recovers from this automatically (issue #1011). When a pixi setup fails with that fetch signature and a committed pixi.lock sits beside the manifest, the executor:

  1. Deletes the committed lock and re-runs the setup command, so pixi re-solves from the manifest.
  2. Parks the fresh solution as pixi.lock.new beside the committed lock (never silently overwriting it).
  3. Restores the committed pixi.lock byte-identical.
  4. Warns loudly with the package-level diff (old lock vs new solve) and marks the run's environment reproducibility as degraded — the report environment section carries an Environment Reproducibility: degraded … disclosure (see Report).

The run proceeds on the re-solved environment: rule commands wrap with pixi run --as-is, which skips re-validating the committed lock against the upstream channels — the restored lock still pins the vanished build, so the default wrap would fail the rule with the same 404. The degraded marker persists in the environment cache (a pixi_degraded.json sidecar beside environment_cache.json), so later runs keep wrapping with --as-is: they skip setup entirely (the re-solved environment is already materialized and cache-ready) but still disclose degraded reproducibility in the report.

To adopt the fresh lock, replace pixi.lock with pixi.lock.new and commit it. The environment's cache key folds the lock content, so the stale degraded marker stops matching and the next run returns to the normal lock-validated wrap — nothing to clean up by hand.

--strict-envs disables the fallback entirely: a stale-lock failure keeps today's hard error with a hint explaining how to refresh the lock by hand. Use it for GxP-style deployments that must never run on a re-solved environment.

What the fallback does not do

A deterministic solver conflict (failed to solve requirements) is not recoverable — re-solving would land on the same conflict — so it still fails immediately. The fallback fires only for fetch failures (HTTP 404/410 or equivalent) with a committed lock present.

Pass config values#

Every key in the [config] section of the workflow (including declarative key = { default = …, required = …, … } entries) can be overridden from the CLI — no extra flags to declare:

# Direct flag form
oxo-flow run pipeline.oxoflow --database refs/nt

# Attached form
oxo-flow run pipeline.oxoflow --database=refs/nt

# Key=value form directly after the workflow file
oxo-flow run pipeline.oxoflow database=refs/nt threshold=1e-3

# Legacy --arg form (still supported)
oxo-flow run pipeline.oxoflow --arg database=refs/nt --arg threshold=1e-3

# Empty string = explicit derive-if-empty override
oxo-flow run pipeline.oxoflow gene_bed=
oxo-flow run pipeline.oxoflow --gene_bed ""

Did-you-mean hints on mistyped overrides and rules

Typos in --KEY VALUE overrides and --rule targets are diagnosed, not just rejected. An override key that is not declared in [config] and never referenced as {config.KEY} suggests the closest declared key by edit distance:

oxo-flow run pipeline.oxoflow --samplename S1
Warning: override 'samplename' is not a declared [config] key and the
  workflow never references {config.samplename} — did you mean
  'sample_name'? It changes nothing.

The same heuristic powers --rule suggestions (oxo-flow debug workflow.oxoflow --rule <typo> lists the nearest rule names) and the engine's rule-not-found errors, which list the available rules (sorted, capped at ten) when a when/input reference misses. The hint logic lives once in the core (edit_distance) — the CLI and the engine always agree on what "closest" means.

Empty values select the derive-if-empty branch

An empty value is accepted and overrides the config key with the empty string. Pipelines that auto-derive a value branch on [ -n "{config.gene_bed}" ] — passing gene_bed= forces that branch off (e.g. to re-derive from a new reference instead of reusing a declared default path). Only the key must be non-empty; =value alone is a syntax error.

Ordering: run flags before overrides

run flags (--json, --rerun, --samples, -j, …) must come before positional KEY=VALUE overrides — the override list is trailing, so a flag typed after it is reported with actionable guidance:

oxo-flow run pipeline.oxoflow min_quality=30 --json
'--json' is a command flag, not a config override.
  Command flags must come before KEY=VALUE overrides, e.g.:
  oxo-flow run <workflow.oxoflow> --json min_quality=30
  For a config key that itself starts with dashes, use --arg KEY=VALUE
  (also placed before positional overrides).

Execution profiles#

--profile <NAME> applies a reusable config supplement to a run. Profiles are plain TOML (or .oxoflow) files placed in the workflow's own profiles/ directory; .toml is tried before .oxoflow:

<workflow-dir>/profiles/<NAME>.toml     # preferred
<workflow-dir>/profiles/<NAME>.oxoflow

The name is just a filename — it carries no built-in meaning. What the profile contains decides its effect: a [config]/[defaults] block supplements config, a [cluster] block additionally routes execution to a scheduler (see Cluster submission). Naming a profile conda or slurm is a convention, not a mechanism — compute environments are declared per rule via [rules.environment] and are independent of profiles.

# profiles/cluster.toml
[config]
threads = "32"
memory_mb = "128000"
oxo-flow run pipeline.oxoflow --profile cluster

The profile's [config] table fills in keys the workflow does not set itself by default — values already present in the workflow are never overwritten. This lets one workflow carry per-environment fallback defaults (laptop, cluster, cloud). Declare profile_mode = "override" in the workflow's [workflow] section when a profile should instead replace workflow values (deep-merged: nested tables merge recursively, scalars and arrays are replaced) — the classic "cluster profile switches threads/memory" case:

[workflow]
name = "rnaseq"
profile_mode = "override"   # fill | override (default: fill)

If the named profile file does not exist, run (and dry-run) fail loudly with a nonzero exit and an error listing every available profile in the profiles/ directory — a typo'd --profile must never silently run with the workflow's own config. The same hard error applies when the profile file exists but its TOML fails to parse or merge.

Cluster submission#

A profile that also carries a [cluster] block makes run --profile submit to a scheduler instead of executing locally: the run expands rules into per-instance jobs, submits at most max_submitted at a time (grouped into scheduler arrays where possible), tracks them to completion, and writes the checkpoint exactly as a local run does. Both conditions are required — a [cluster] block in effect and a profile named; adding either one alone never changes an existing local run.

The full story — the [cluster] TOML reference and driver-default fallbacks, profile lookup rules, TMPDIR exports on job scripts, the scratch = true plan-time rejection, job arrays, the run directory layout, what status.json records, and interrupt/re-attach semantics — lives in Cluster submission on the cluster page. run flags that are parsed but dropped on that path (--retry, --timeout, --max-threads/--max-memory) are tabulated in Cluster flag surface (--provenance and --keep-going are honored there — checksums are recorded as jobs settle, and -k keeps in-flight independents running past a required failure).

Cluster scheduler submission (SLURM, PBS, SGE, LSF, Kubernetes, AWS Batch, GCP Batch) is configured separately — see the cluster command.

Partial module runs (--module)#

A composed workflow (clinical pipelines like clindet: several [[include]] modules chained together) can run just ONE module and the producers of its declared inputs:

# run only the germline module (rules/20_germline.oxoflow) of a composed
# clinical pipeline — the module name defaults to the include's file stem
oxo-flow run main.oxoflow --module 20_germline

# multiple modules + rule targets union together
oxo-flow run main.oxoflow --module 20_germline --module 30_vcf_norm -t report

The closure is: the module's rules + every host rule producing one of its declared concrete contract inputs; upstream DAG dependents come through the regular target machinery. The closure is when-aware: two rules sharing one output path behind complementary when gates resolve to the satisfiable one — a rule whose template when evaluates false under the effective config never claims a producer slot, and a member whose wildcard-gated instances were all pruned at fan-out is dropped from the closure (its name has no node in the instantiated DAG). Unknown module names fail with the known module list. dry-run --module previews the same set.

The same when-awareness covers the preflight source gate on -t/--module runs: a missing input whose only remaining producer instance was vetoed by its when at fan-out does not gate the run. Expansion records those vetoed instances' declared outputs in when_skipped_outputs, and the gate exempts any missing pattern found there — config-expanded at compare time, so {config.*}-bearing recorded patterns match their expanded gate forms. The file belongs to a rule that will never run in this cohort, not to a dead source of the selected closure.

Run directory layout#

Cluster runs leave their job scripts, scheduler ids, and per-job status under .oxo-flow/runs/<timestamp>/ — see Run directory layout on the cluster page.

What a finished job records#

Cluster jobs record the scheduler's accounting data (queue wait, exit code, wall time, peak RSS, CPU seconds) into per-job status.json files; fields the site's accounting store does not report are omitted rather than guessed. See What a finished job records for the record format, the accounting-variance caveats, and the interrupt/re-attach semantics.

Failed-rule output rollback#

A failed rule's declared outputs are rolled back: files created by the attempt are deleted, and pre-existing files are moved aside as *.oxo-failed (protected outputs declared with protected_output are never touched). The failure message names the rolled-back files, so a rule that created one of two declared outputs and then failed shows both the missing file and the one that was removed:

[oxo-flow] output validation failed: 1 declared output(s) not found: …/merge.vcf
[oxo-flow] declared outputs of the failed rule are rolled back
  (deleted or moved aside as *.oxo-failed): …/merge.vcf.idx

Placeholder validation (strict by default)#

Every executing rule's rendered command is scanned for engine placeholders that survived rendering — {sample}, {group}, {pair_id}, {experiment}, {control}, {tumor}, {normal}, {chunks}, {log}, {log[0]}, {threads}, {memory}, {effective_threads}, {effective_memory_mb}, {meta.<column>}, {config.<key>}, {params.<key>}, {input…}, {output…}, {wildcard.*}, and {values.*}. Such a token has no value bound to it, so the shell would receive the literal brace text and the tool would fail on a path that looks valid. The rule fails before the shell spawns:

✗ rule 'align_reads' failed: unresolved placeholder(s) {meta.bam} reached
  the rendered command — … Fix: give the rule a sample-domain binding, or
  add `singleton = true` for an intentional constant-metadata read. …

The fix hints are class-specific (declare the [config] key, check an {input[N]} index, add the sample-domain source, …). Two escape hatches exist while downstream workflows migrate:

  • [engine] strict_placeholders = false in the workflow, or
  • OXO_FLOW_STRICT_PLACEHOLDERS=0 in the environment (wins per invocation).

Both restore the historical warning-only behavior. singleton = true silences the static W041 diagnosis for an intentional one-instance {meta.*} read but does not soften this runtime gate: a {meta.*} token that survives rendering (more than one metadata row and no sample-like binding) is still fatal. A one-row metadata table is the supported constant read; use {config.<key>} for a workflow constant. {sample}/{group} survivors are fatal in every case. Declared output paths are checked too: an unresolved placeholder there is fatal even for a singleton — a literal file named after the token is never a valid artifact. The strict_placeholders switches above also restore the warning-only behavior for that namespaced output-path check (the bare-wildcard output gate is a separate, always-on correctness check).

Shell ${...} parameter expansion (${#parts[@]}, ${ch##*.clones_}) is not engine syntax and the gate never treats it as a placeholder. Template rendering is textual, however: a bound name written inside ${...} is substituted like any other token (${sample} renders as $<sample value>), so give shell variables names that do not collide with fan-out variables. Brace groups ({ echo hi; }) and brace expansion ({1..10}, {a,b}) are likewise shell text. Two realistic false-positive shapes need an escape: a Python f-string that prints an engine token (python3 -c 'print(f"{sample}")' — write f"{{sample}}" instead) and a shell literal (\{sample\}). Escape the opening brace in the host language (\{ in bash, {{ in Python f-strings) or build the literal from parts; $-prefixed and backslash-escaped braces are never flagged. lint reports the static diagnosis as W041 — a rule that references the sample domain without a binding source.

Undefined config variables fail at plan time#

Independent of the runtime gate above, run and dry-run hard-fail before any rule is planned when a rule references a {config.<key>} that is defined nowhere — not in [config], not supplied on the CLI, and not one of the engine-injected keys (samples_list, paired_pairs_count, …). The scan covers shell commands, script paths, input/output paths, log, benchmark, output_pattern, lifecycle hooks, and when conditions:

Error: workflow references undefined config variables — fix before running:
  rule 'align_reads': shell command references undefined config variable
  'refence' (define 'refence' in the [config] section)

This plan-time gate is not controlled by [engine] strict_placeholders or OXO_FLOW_STRICT_PLACEHOLDERS=0 — those switches only soften the execution-time rendered-command scan above. An undefined key has no value to warn about, so the run aborts before the DAG is built. validate and lint surface the same problem as the static E005 diagnostic, so the documented validate → dry-run → run path catches it first.

Execute a published bundle#

# Run from a bundle (extracts, verifies checksums, shows resource requirements,
# then prompts for confirmation — requires an interactive terminal on both
# stdin and stderr)
oxo-flow run --bundle pipeline-bundle.tar.zst -j 16

# Skip confirmation for CI/scripts (also required when stdin is redirected
# or with --json, which is always non-interactive)
oxo-flow run --bundle pipeline-bundle.tar.zst -j 16 --yes

# .tar.gz format also supported
oxo-flow run --bundle pipeline-bundle.tar.gz -j 16 --yes

# Pull from remote and execute in one step
oxo-flow pull gh:user/repo@v1 && oxo-flow run --bundle repo-bundle.tar.zst -j 16 --yes

Bundles published with publish --sign-key carry an ed25519 manifest signature. run --bundle verifies it after the checksums and aborts before execution if the manifest or signature was altered; unsigned bundles keep the checksums-only behavior (see publish → Bundle Signing).

Pilot runs and scale-up#

For a large cohort, run a fast pilot on a subset first; the checkpoint then skips the pilot samples when you scale up. --samples needs a workflow that has a sample dimension (sample_pattern, [[sample_groups]], sample_groups_file, or [[pairs]]): a workflow with none rejects the flag with --samples matched no samples in this workflow.

# Pilot: run the full pipeline on the first 2 samples only
oxo-flow run pipeline.oxoflow --samples first:2

# Or name the pilot samples explicitly (combines with first:N)
oxo-flow run pipeline.oxoflow --samples first:2,NA12899

# Scale up: no flag needed — completed samples are skipped automatically
oxo-flow run pipeline.oxoflow

# Preview the pilot plan without executing — with a checkpoint present,
# dry-run also predicts what will actually re-run (and what stays
# protected); see dry-run's "Checkpoint-Aware Rerun Preview"
oxo-flow dry-run pipeline.oxoflow --samples first:2

# Fix a config bug found by the pilot — only affected rules re-run
# automatically (see "Config changes" below); use --rerun to force everything
oxo-flow run pipeline.oxoflow --rerun

--samples filters every sample source (sample groups, sample_pattern discovery, and experiment/control pairs). --rerun re-executes the rules selected for this run even when outputs are up to date — combine it with --samples to re-run just the pilot subset while keeping the other samples' completed records.

After a --samples run, a pilot summary is printed: samples run, wall time, per-sample time, and a linear projection for the full cohort. Scientific-preflight findings are reported as distinct findings with their instance counts — one template rule appearing once per sample shows as 1 distinct finding (3 instance(s)), not 3 finding(s). When the workflow enables [ai], a plain-language pilot report (health assessment and scale-up advice) is appended automatically.

Incremental data arrival: --samples ready#

Sequencing centers deliver data in batches, so a cohort's fastq files trickle in over days. Instead of waiting for every sample, run the analysis as data arrives:

# Which samples can be processed right now, and what is still missing?
oxo-flow dry-run pipeline.oxoflow
#   Sample readiness: 87/100 complete, 13 waiting
#     ⏳ NA12891 (missing: data/NA12891_R2.fastq.gz), …

# Run only the samples whose entry inputs are complete
oxo-flow run pipeline.oxoflow --samples ready

# … more data arrives …
oxo-flow run pipeline.oxoflow --samples ready
#   Done: 5 succeeded, 87 skipped — the checkpoint skips completed samples

A sample is ready when every external input belonging to it exists — that is, every rule input (after wildcard and {config.x} expansion) that the workflow itself does not produce. Files declared in a rule's protected_output count as workflow-produced (a rule may declare expensive secondary artifacts only there, and downstream expand_inputs consumers read them during the run). Intermediate products are never checked (producing them is the DAG's job), and optional = true rules do not block readiness (the executor skips them when their inputs are absent). Relative paths resolve against the workflow file's directory (or --workdir) — the same place rules actually run from.

Semantics:

  • ready is a special --samples value, not a new syntax: it resolves to the names of ready samples and combines with first:N and explicit names as a union. Because it is reserved, a sample literally named ready cannot be selected by name.
  • With zero ready samples run aborts and lists the waiting samples; dry-run --samples ready instead reports the full cohort state.
  • For [[pairs]] workflows, a pair is kept only when both experiment and control inputs are complete; otherwise it is skipped with a note. Missing files that belong to no specific sample (shared references) are reported as workflow-level inputs.
  • The report is also available as JSON via dry-run --json (the samples block), and --samples ready works in test --run as well.

Background runs (--background)#

run --background (and resume --background; --detach is an alias) detaches execution into a background process and returns to the shell immediately:

# Foreground prints one line and exits 0; the run continues in the background
oxo-flow run pipeline.oxoflow --background
#   started in background (pid 48291) · log: .oxo-flow/logs/oxo-flow.log ·
#   monitor: oxo-flow status .oxo-flow/checkpoint.json ·
#   wait: oxo-flow wait --workdir /data/run --checkpoint …/checkpoint.json ·
#   stop: kill 48291

# Detach a resumed run the same way
oxo-flow resume .oxo-flow/checkpoint.json --background

# Block from a script until the detached run finishes:
oxo-flow wait --workdir /data/run            # exit 0 ok, 1 failed, 3 timeout

Mechanics:

  • The foreground process spawns a detached child that re-runs the same command with the detach flag (--background / --detach) removed — every other flag (-j, --profile, --samples, --rerun, --log-file, --checkpoint, …) passes through verbatim, so a background run behaves exactly like a foreground one: checkpoint, workdir lock, report snapshots, and resume semantics are unchanged.
  • The child's stdout and stderr are redirected to the run log (--log-file if given, otherwise .oxo-flow/logs/oxo-flow.log in the workdir). In this mode the tracing tee is skipped on purpose — the redirect already captures everything, and a second writer would duplicate every line.
  • The run log archives BOTH the structured tracing stream AND the user-facing progress narrative (Running: / ✓ / Done:) plus one JSON line per execution event (workflow_started, rule_started, rule_completed, rule_failed, workflow_completed) — the whole run is replayable from the single file. Failed rules emit "event":"rule_failed" (compact serde_json, no space after :) carrying rule, exit_code, signal, retries, and duration_ms, so grep -c 'rule_failed' run.launch.log counts failures without parsing the human-facing ✗ narration; "signal":9 after an OOM-kill is the smoking gun distinguishing a memory-capped environment from a tool bug. duration_ms spans all attempts including retry delays, not just the final one. Staging failures (environment setup, security checks) never start a process, so they surface as "event":"rule_completed" with "status":"failed" and no exit code/signal instead.
  • The child's pid is written to .oxo-flow/background.pid in the workdir (also shown in the summary) while the run is in flight; the file is removed when the child exits — including the signal-termination path, so a stale pid never survives a finished run.
  • The run's own terminal outcome is written to <checkpoint>.status (beside the checkpoint; running once the workdir lock is held, then succeeded/failed with the error text). Per-checkpoint placement keeps variant runs in one workdir from aliasing each other. oxo-flow wait uses it to distinguish a preflight refusal from a previous clean checkpoint.
  • The child gets its own process group, so it survives terminal close and Ctrl-C at the shell — it keeps running until it finishes or is killed.
  • The foreground exits 0 once the child is spawned; it does not wait. If the spawn fails (for example because another run already holds the workdir lock), the foreground exits nonzero with the reason.

Which commands accept --background:

  • run and resume — the only long-lived terminal-bound commands. Cluster runs go through run --profile <name> and therefore detach the same way: run --profile slurm --background submits and polls the cluster jobs entirely inside the background process.
  • dry-run intentionally does not: a preview is fast and read-only — keep it in the foreground.
  • Other commands (pull, test, export, ai, …) are either quick or one-shot; run them before the background run and keep them foreground.

Monitoring a background run works exactly like a foreground one: poll oxo-flow status .oxo-flow/checkpoint.json (or status --timing), read the run log, and check the report snapshots in .oxo-flow/reports/. For scripted supervision, oxo-flow wait blocks until the run finishes and returns a structured exit status (0 success, 1 failed or incomplete, 3 timeout, 4 no run found) — the native replacement for a poll-for-output-files shell loop. Stop a background run with kill <pid> (Unix) or taskkill /PID <pid> — the pid is in the summary line and (while the run is in flight) in .oxo-flow/background.pid.

Notes:

  • The global --json is rejected with --background: the foreground only launches the detached child, so there is no run summary to emit. Drop --json or run in the foreground to get the machine-readable summary.
  • Bundle runs need an explicit --workdir (the bundle's extracted directory is created per-process, so it cannot be predicted from the foreground invocation) and --yes — a detached child cannot prompt. --workdir is also what bundle discovery consults as a fallback: a sample_pattern/pairs_pattern that matches nothing inside the extraction is retried against the workdir.
  • The detach flag is a launcher flag, not a workflow setting: the child re-parses the remaining argv, so flag ordering rules (command flags before KEY=VALUE overrides) apply to the whole command line as usual.

Checkpointing and Resuming#

oxo-flow automatically persists execution state to a checkpoint file after every rule completion. This enables:

  1. Resuming failed runs: If a job fails or is interrupted, simply run oxo-flow run again. The engine will skip all rules already marked as completed and resume from the first pending task.
  2. State Inspection: Use the oxo-flow status command to view the progress of a run from its checkpoint file.

A rule whose latest execution record is a non-terminal running state (a crash or kill while the rule was in flight) never passes the freshness gate: on restart the engine re-runs it instead of trusting the partially-written outputs, so a killed job's half-finished outputs are not mistaken for finished work (issue #1083). Rules with no checkpoint entry at all keep their historical freshness semantics — output existence alone decides, as before (issue #77).

Variant checkpoints (--checkpoint)#

The default checkpoint is per workdir (<workdir>/.oxo-flow/checkpoint.json), so running a pipeline.smoke.oxoflow variant next to pipeline.oxoflow in the same directory shares one file. The engine detects the workflow identity mismatch and ignores the other workflow's completed/failed records — correct, but the variant runs are then silently non-reusable. Pass one checkpoint per variant to make each first-class:

oxo-flow run pipeline.smoke.oxoflow --checkpoint .oxo-flow/smoke.checkpoint.json
oxo-flow resume .oxo-flow/smoke.checkpoint.json   # resume the smoke variant
oxo-flow status .oxo-flow/smoke.checkpoint.json   # inspect it

An explicit path wins over the workdir-derived default for load and save. Relative paths resolve against the process working directory, like every other --checkpoint-style argument.

Metadata Directory#

By default, checkpoints are saved in a hidden .oxo-flow/ directory located in the same folder as the workflow file.

  • Filename: checkpoint.json (the name is always the same regardless of workflow name)

Rule run records in the checkpoint#

Each executed rule carries a rule_runs entry in checkpoint.json with the expanded command, the process exit code, a bounded stdout/stderr tail, the resolved report caption, and (since the cancellation-semantics change) the terminal status ("failed", "cancelled", "timed_out", …), the skip_reason when the rule did not run to completion, and the terminating signal when the process died to one. The three newer fields (status, skip_reason, signal) are optional and absent in checkpoints written by older engines — consumers must infer legacy records as before (failure when the rule is in failed_rules). Each entry also persists the execution started_at and finished_at timestamps, so status consumers can report a node's start alongside its duration for both completed and failed rules (see the web API's Run Status); both are absent in legacy checkpoints and for rules whose execution never started.

Timestamp semantics: the engine stamps enqueued_at when an execution attempt is submitted and stamps started_at only after the rule has acquired its resources — the moment actual execution begins. Their difference is the time the rule spends waiting in the resource pool (same split as the cluster driver's queue_wait_secs), and all wall-time reporting (duration_ms, the report's Benchmarks Wall Time column, benchmarks.wall_time_secs) is measured from started_at, so a rule parked behind busy resources never inflates its reported runtime. The checkpoint persists these timestamps in the rule_runs entry itself (alongside the resulting wall time via benchmarks.wall_time_secs), and cluster runs additionally persist the split per job as queue_wait_secs in jobs/<rule>/status.json (see What a finished job records). enqueued_at remains in-memory only.

The tails are truncated to the last 64 KiB per stream by default and prefixed with a … marker when cut. Set OXO_FLOW_OUTPUT_TAIL_BYTES to change the size (bytes; an invalid or zero value falls back to the default with a warning), e.g. OXO_FLOW_OUTPUT_TAIL_BYTES=10485760 (10 MiB) for error-heavy tools whose message scrolls past the tail. The in-RAM capture window widens to match when the setting exceeds its 1 MiB floor — budget roughly 2 × (1 MiB + tail) of RAM per running job. Tails are bytes-bounded, so a multi-byte UTF-8 cut lands on the nearest character boundary (rounding down); older checkpoints that only carry the legacy 2048-character tails remain readable (the new size applies from the next recorded run onward).

Declared log files vs captured tails#

A rule's declared log path is separate from this capture: the engine creates the log's parent directory, but the file only gains content when the rule's shell redirects into it (2> {log}) and the tool writes to stderr. Many bioinformatics tools self-log to their own location instead (e.g. a logs/<date>.txt file they manage themselves), leaving the declared log empty — an empty declared log never means no diagnostics were captured; the tails above are persisted in rule_runs regardless. When a rule succeeds, its declared log is empty (0-byte file or empty directory), and both captured tails are empty too, the run emits one Info-level hint that the tool likely self-logs elsewhere — quiet tools with output in any of the three stay silent.

Workflow version in the checkpoint#

When the workflow file lives inside a git repository, oxo-flow records the repository's HEAD commit SHA in the checkpoint (workflow_git_sha) at run start. Every result set is therefore auditable to the exact workflow version that produced it: oxo-flow provenance verify prints the SHA, and report snapshots carry it through the checkpoint they embed.

The lookup is best-effort — running outside a git repository simply omits the field and never fails the run. Commit your workflow before running to get fully version-audited results. See Workflow Versioning for the full model.

Include resolutions in the checkpoint#

Every [[include]] that pins a git repository (repo + ref) records the exact commit SHA its checkout resolved to at parse time, in the checkpoint (include_resolutions, keyed repo@ref path → SHA). Branch and tag pins re-fetch on every activation — the workflow file alone cannot prove which module content a past run actually used — so this per-include SHA is the evidence a run/resume pair needs that the modules did not drift between the two. The field is absent for workflows without repo includes, and recording it is best-effort (an unreadable checkout logs a warning and records nothing, never failing the run). The run report's Provenance section lists one row per include. See Workflow Versioning for the full model.

Run logs#

Every run — and resume, which re-enters the same path — archives its own log under the workdir:

  • .oxo-flow/logs/oxo-flow.log is always the latest run; on each new run the previous log rotates to .oxo-flow/logs/oxo-flow.log.1, then .2, … up to .9 (the oldest backup is deleted). --log-file PATH overrides the location; the same rotation applies to PATH.
  • The log header names the exact workflow version that produced the record: timestamp, oxo-flow version, full command line, workflow path, workflow_name, workflow_version, git_sha (the repository HEAD recorded by the run — see Workflow Versioning), and workdir.
  • The engine's tracing stream (rule start/finish verdicts, warnings, errors) is written into the log alongside stderr. Logging is best-effort: if the file cannot be opened or written, the run continues with stderr-only logging and a warning. Dry-run remains stderr-only.

Report snapshots#

After every run (and resume) — unless --no-report-snapshot is given — oxo-flow writes a JSON report snapshot of the final checkpoint:

  • .oxo-flow/reports/report-<UTC timestamp>.json — the full report data model, so a run leaves a machine-readable report behind with no reporting step needed (a -N suffix is used when two snapshots land in the same second)
  • .oxo-flow/reports/index.json — a JSON array of {generated_at, workflow, checkpoint, report} entries, kept sorted by generated_at; the last entry is the newest snapshot

Snapshot failures are warnings — a reporting hiccup never fails a run. oxo-flow report with no -o writes auto-discovered reports into the same .oxo-flow/reports/ directory.

Config changes and precise invalidation#

The checkpoint records a snapshot of the effective config values and a structural fingerprint of every completed rule. On each run, oxo-flow compares them against the current workflow:

  • Changed config keys invalidate exactly the rules that reference them plus their DAG downstream; unrelated completed rules keep their checkpoint records. How a rule references a key decides the verdict:
  • A key interpolated into shell, script, input/output paths, envvars, or params ({config.<key>}) always invalidates — the new value bakes straight into the command or paths.
  • A key referenced only inside a when condition invalidates only when the condition's truth value actually flips under the new config. A toggle that leaves the gate true (or false) — e.g. (config.refine_bins || config.run_checkm) switching which term is true — reuses the completed rules instead of re-running hours-long chains whose inputs and commands are identical. Flipping a gate in either direction still invalidates: the engine pre-marks completed rules without re-evaluating when, so only plan-time detection can retire a now-skipped producer or cascade a newly-activated one to its consumers.
  • Edited rule definitions (shell, inputs, outputs, environment, …) are caught by the per-rule fingerprint and invalidate the same way. A file-backed environment spec (pixi manifest, conda/mamba YAML, venv requirements) is fingerprinted by content, not just by the path the rule names — pixi specs also cover the sibling pixi.lock, since a lock re-pin changes exactly which packages the environment materializes. Editing any of these files in place invalidates the rules that reference them. Checkpoints written before the content tagging lack these digests, so the first run after that upgrade re-runs rules with file-backed environments once (the same safe-but-costly bootstrap as the config-tracking baseline below).
  • samples_list / samples_<group> are engine-injected and never trigger invalidation, so toggling --samples between runs stays cheap. This covers rules whose input list is baked from an injected key (expand_inputs over config.samples_list, a gather-rule pattern): their rule fingerprint and input manifest change with the selection, and oxo-flow treats that change as non-invalidating. When such a rule is skipped because of a selection change, run prints a warning naming the rule — its outputs still reflect the previous run's full sample set, and --rerun is the documented way to regenerate them under the new selection. The exclusion is narrow and verified: the rule's other definition fields must be byte-identical, and the current input file set must be exactly reproducible from the injected value with unchanged content — any genuine edit still invalidates. Checkpoints written by older binaries lack the input-excluded fingerprint, so the first subset run after an upgrade may re-run such a rule once (safe default).
# min_quality is interpolated by fastp_trim; only it and its downstream re-run.
# A flag toggled in a when-only rule whose gate stays true re-runs nothing —
# the run reports those rules as reused instead.
oxo-flow run pipeline.oxoflow min_quality=30
Config change:
  min_quality: 20 → 30
  when-condition unchanged, reused: bin_classify
  → invalidated 3 (1 directly affected), re-running 3/10 this run, skipping 7
  ⊝ index_reference (already completed)
  Running: fastp_trim
  …

When a when gate closes at execution time, the rule is reported under Skipped (when condition false) with a ⊘ marker (or ⊝ in the config-change reuse view) — a gated-off rule is a skip, never a failure (see status for the checkpoint-side display, including legacy checkpoints that carry stale failed_rules entries).

Rules changed by a transformed split set (transform.split.values_from) are detected through their baked input lists, so chunked rules re-combine correctly when the split values change.

Limitations (correctness-first, documented deliberately):

  • Changing threads/memory (performance knobs) does not invalidate results; changing anything else in a rule — including shell comments — does.
  • A checkpoint written before config tracking was introduced is adopted as a one-time baseline: the first post-upgrade run reuses everything and records the snapshot; changes made before that baseline cannot be detected. The same applies to the per-rule when verdicts behind gate-aware reuse: a pre-verdict checkpoint invalidates when-referencing rules once on the first changed-key run, records their verdicts, and reuses stably from then on.
  • Config values declared sensitive are stored in the snapshot as SHA-256 digests, never as plaintext.
  • Concurrent oxo-flow run invocations on the same workdir are prevented by an exclusive lock on .oxo-flow/lock (held for the whole run). The second invocation fails with a clear error instead of silently racing on checkpoint.json; the lock releases automatically when the first process exits (even on a crash). clean refuses to delete while a run is active unless --force is given.

Input changes and manifest invalidation#

The checkpoint also records an input manifest for every completed rule: the file set its inputs resolved to at completion time, with each file's path, size, and modification time — plus a content hash for files up to 64 MiB. Entries also record whether the file is a symlink and, if so, its recorded target. On every run, oxo-flow re-resolves the inputs and compares:

  • Literal glob inputs (data/*.txt) detect added and removed files — a new file matching the glob rebuilds the rule (and its DAG downstream).
  • Directory inputs (e.g. input = ["results"] — a path that resolves to a directory) are listed recursively, so files added or changed anywhere inside invalidate the rule. Dir inputs with a pattern filter track only matching files.
  • Plain file inputs are content-addressed up to 64 MiB: a same-size rewrite is detected even when the mtime is preserved, and a mere touch no longer invalidates. Larger files compare size and mtime, closing the same stale-reuse hole for ordinary files.
  • expand_inputs-baked inputs whose producer is when-gated off under the active config are tolerated as absent: the manifest records the existing subset instead of erroring, so a no-op rerun does not invalidate the rule and cascade its producers. If a later config change produces the missing files, the manifest then mismatches and the rule invalidates — the flip is still caught.

When an invalidation is detected, the run output spells out WHAT changed, one line per affected file. Each line reads path: <old size> @<old mtime> → <new size> @<new mtime> followed by these labels:

  • (content <h1> → <h2>) — the content hash moved (files up to 64 MiB are hashed); shown alone when a same-size/same-mtime rewrite is caught by hash comparison.
  • (TRUNCATED) — the file collapsed to 0 B: the vendor-data incident shape a write through a symlinked input path produces (the shell truncates the target before the command's bytes land).
  • (removed) / (added) — the file disappeared from, or appeared in, the tracked set.
  • [symlink] — the path resolves through a symlink on the recorded or the current side (recorded symlink facts are display-only and never invalidate by themselves).

For example (from the verification of this feature):

  · raw/S1.fq: 22 B @2026-10-27T03:08:41Z → 0 B @2026-10-27T03:08:42Z (content sha256:b29f0d4b37f… → sha256:e3b0c44298f…) (TRUNCATED) [symlink]

A pure symlink-target flip with unchanged size/mtime/hash produces no diff line at all (target facts are display-only); the target surfaces only in the first-seen notice below.

A run also prints one first-seen notice per symlinked input — ℹ input <path> is a symlink → <target> — collected across all completed rules, deduplicated and sorted, whether or not anything invalidates. A rewrite of identical size/mtime through a symlink leaves the manifests matching (so nothing re-runs) but still names the path, because writing to a symlinked input path writes THROUGH to its target — the exact hazard behind irreversible data loss in real pipelines. The notice fires once per run; it is not gated on the file changing.

  • Missing external source inputs fail fast before any work starts: a required input that no rule in the DAG produces and whose concrete files are absent aborts the run before scheduling with workflow is missing required source input(s) — no rule produces them, naming each affected rule and pattern (exit non-zero). Previously a fully-completed checkpoint plus a deleted raw data file could exit 0 from stale checkpoint credit ("re-running 0 producer rules"); that silent-success hole is closed. The gate never fires for generated intermediates (they have producers and are governed by the manifest cascade below), [[references]] outputs (built later in the same run), inputs discoverable through the workflow's sample_pattern directory, when-gated-off rules, zero-fan-out scatter rules, or optional rules.
# a new file appears in data/ after the last run
oxo-flow run pipeline.oxoflow
  ↻ input changes invalidated 2 rules: gather, report
  Running: gather
  Running: report
  …

Invalidated rules bypass the executor's freshness gate and re-execute even when their outputs exist and look recent; downstream rules that consumed their outputs rebuild in the same run.

Limitations (correctness-first, documented deliberately):

  • Detection is a hybrid policy: files up to 64 MiB are content-hashed (sha256), so a same-size rewrite is detected even when the mtime is preserved — and a mere touch no longer invalidates. Larger files (multi-gigabyte intermediates) keep the size+mtime policy, like make and Snakemake's default mode: hashing every input on every run would cost O(total input bytes), prohibitive for bioinformatics-scale files. A large file rewritten with identical size and a deliberately preserved mtime is not detected; use --rerun to force. Legacy checkpoints (entries written before hashing) keep comparing size+mtime instead of invalidating everything once.
  • Only one glob level is tracked per input pattern; brace-expansion patterns (*.{txt,log}) are not expanded by the manifest scanner, so their matched set is not tracked (the shell still expands them at runtime).
  • Symlinked directories are recorded as single entries, never traversed (cycle-safe), so changes inside a symlinked directory are tracked at the symlink's own metadata level. Symlinked files are recorded with their symlink-ness and lexical target (see the first-seen notice above); the comparison follows the link's resolved content — a retarget to an identical-content file keeps the rule valid, while a write through the link invalidates like any content change.
  • Rules that clean up their inputs at the end of a successful run (transform.cleanup = true chunk consumers) are not manifest-tracked: their inputs are engine-managed intermediates governed by the upstream-cascade invalidation instead.
  • Optional rules with permanently-absent inputs (optional = true or optional = "any"): when a declared input cannot be resolved because its producer was never instantiated (an input_groups pattern that matched no files, or an endedness-filtered branch that never ran), the snapshot skips that entry and records a manifest for the inputs that do exist — the rule stays skipped/completed across runs instead of cascading a full re-run every invocation. The same grace applies during invalidation detection: a missing input with no producer in the DAG is absent by design for an optional rule, while a required rule with a missing input still invalidates (a deleted raw data file must re-run). A failed manifest snapshot is reported on stderr and in the run log (⚠ could not snapshot inputs of rule …) instead of being silently dropped.
  • A checkpoint written before input tracking was introduced adopts the current file set as a one-time baseline (the first post-upgrade run reuses everything); changes made before that baseline cannot be detected.

Temporary rules (temporary = true)#

Pipeline intermediates are usually the largest files in a run (per-sample BAMs, unsorted alignments) and often useless once every downstream rule has consumed them. Marking a rule temporary = true deletes its outputs after a fully successful run — once every dependent has completed — and records a tombstone in the checkpoint:

[[rules]]
name = "align"
output = ["aligned/{sample}.bam"]
temporary = true

This is checkpoint-aware, not a blind delete:

  • A plain re-run skips the rule like any completed rule — the output is not regenerated just because it is missing.
  • When a dependent actually needs the outputs again (its own outputs were deleted, its inputs changed, or the config changed), the producer is regenerated first (lazy cascade-up), then the dependent runs — the same order as the original run.
  • Failed runs keep the outputs (the deletion only happens when nothing failed), so debugging a broken run never has to re-fetch intermediates.
  • Leaf rules (no dependents) keep their outputs — there is no downstream work for them to enable.
  • With an [engine] section present, the deletion is disk-gated: it only fires when free space at the workdir is below reclaim_free_disk at run end. With disk comfortable, the outputs are kept — so the next run skips the completed producer instead of re-materializing it — and one line explains the choice:
  ℹ free space 39876544MB ≥ reclaim_free_disk 5120000MB — temporary outputs kept (disk pressure alone triggers re-materialization; remove the [engine] thresholds to always prune)

If free space cannot be measured (non-Unix platforms), the outputs are kept as a precaution. Workflows without [engine] thresholds keep the historical unconditional deletion.

  ↻ temporary outputs needed again — re-running 1 producer rule: trim_S1
  …
  ⊘ temporary outputs deleted for 'trim_S1' (1 file, regenerated on demand)

Dry-run predicts exactly when a tombstoned rule will regenerate ([rerun: upstream of X]) — see Checkpoint-aware rerun preview.

Disk-pressure monitoring ([engine])#

Large pipelines can fill a disk mid-run. Declaring an [engine] section in the workflow makes the engine measure the working directory's free space at startup, at every scheduling round, and every 60 s while work is parked:

[engine]
min_free_disk = "50G"        # floor — entering Hold below it
reclaim_free_disk = "100G"   # optional reclaim trigger; defaults to min_free_disk

The response ladder:

  • Reclaim (min ≤ free < reclaim): completed rules' temporary outputs are tombstoned and unlinked mid-run — the same checkpoint-aware reclaim as temporary = true cleanup, but while the run is still going. Only when every dependent is completed and its own outputs still exist (a dependent whose outputs are missing will re-run and needs its inputs). The narration line is ⊘ reclaimed temporary outputs of '<rule>' …. The same floor gates the success-path deletion of temporary = true outputs (see Temporary rules) — with disk comfortable at run end they are kept, not pruned.
  • Hold (free < min): new rule dispatches are withheld; in-flight rules continue. If nothing is in flight and nothing is reclaimable, the run parks and re-measures every 60 s.
  • Abort (Hold persists ~2 parked rounds ≈ 2 min): the run checkpoints and fails. Free space manually (delete intermediates or oxo-flow clean), then oxo-flow resume — or lower min_free_disk.

Protected outputs (protected_output) are never reclaimed, and the final (leaf) rule's outputs always survive. See [engine] in the workflow format reference for the full field table and edge cases.

Tuning knobs (both fall back to their defaults when unset, invalid, or 0, so a typo can never disable the abort):

  • OXO_FLOW_DISK_TICK_SECS — seconds between measurements while parked (default 60).
  • OXO_FLOW_DISK_HOLD_ROUNDS — grace park rounds under Hold before the run aborts (default 2).

Known boundaries:

  • Mid-run reclaim of fan-out producers (output_pattern rules with per-instance values) fires only with full instance attribution: the recorded instance combos from checkpoint.output_pattern_domains are expanded to concrete paths, and every combo must resolve and every consumer (detected structurally — pattern producers register no DAG edges) must be completed before any deletion. If any instance combo cannot be attributed, the whole rule is skipped with a once-per-rule warning — no partial tombstone is written. The post-run sweep covers them regardless (see below).
  • Free space is measured at the working directory's filesystem, which is volume-level (statvfs): thresholds apply to the whole volume, not to a quota.
  • Keep scratch-heavy tools off the measured volume only if they are not cleaned by their rules; outputs declared by rules are always accounted for by the ladder.

Rule execution environment (issue #1108)#

Rule shells, pre_exec hooks, rule-level on_success/on_failure hooks, and workflow-level on_complete/on_error hooks run under a cleared environment: the child starts from an empty environment plus a fixed allowlist, never the engine's ambient environment. The allowlist is the rule's own declared envvars (applied last, so they win), PATH, HOME, LANG, every LC_* variable, and the TMPDIR/TMP/TEMP triple from [engine] tmp_dir / OXO_FLOW_TMPDIR. This prevents a rule that declares no environment from reading the engine's API keys and cloud credentials out of its own process environment.

A workflow that genuinely needs an ambient variable opts it back in per invocation:

OXO_FLOW_ENV_PASSTHROUGH=SLURM_JOB_ID,USER oxo-flow run workflow.oxoflow

The value is a comma-separated name list (whitespace around each name is trimmed; empty entries ignored); only the names listed are re-admitted — everything else stays hidden. It is an environment variable rather than an [engine] key because the need is host/invocation-specific (a variable present on one cluster but not another), mirroring OXO_FLOW_TMPDIR.

Engine tooling commands (conda/pixi/venv environment setup, teardown, and env_verify) spawn through a different code path and keep the ambient environment, so tool discovery and proxy settings still work.

Two CLI-side shell spawns run through this same allowlist policy (issue

1233, RB-15), so they cannot read the operator's ambient secrets either:#

  • The reference build shell ([[references]] build, including its bash-first / sh-fallback) — a reference whose environment is a docker/singularity kind gets the container-daemon admission below, exactly like a container rule.
  • The oxo-flow batch sugar shell (sh -c <template>) — it has no [engine] section, so its scratch routing comes from OXO_FLOW_TMPDIR alone.

Local container rules (docker/singularity)#

A container rule runs docker run / apptainer exec from inside that cleared shell, so daemon and cache configuration the operator relies on would otherwise be invisible to it. Rules whose environment kind is docker or singularity therefore automatically re-admit these ambient variables when they are set (issue #1173):

  • Docker CLI: DOCKER_HOST, DOCKER_CONFIG, DOCKER_CERT_PATH, DOCKER_TLS_VERIFY, DOCKER_BUILDKIT
  • Apptainer/Singularity: APPTAINER_CACHEDIR, SINGULARITY_CACHEDIR, APPTAINER_BIND, SINGULARITY_BIND, APPTAINER_TMPDIR, SINGULARITY_TMPDIR, APPTAINER_LOCALCACHEDIR, SINGULARITY_LOCALCACHEDIR

The list is exact-name (never a DOCKER_*/APPTAINER_* prefix, so an unknown credential-bearing variable is not admitted by accident) and applies only to container rules — the bare (non-container) path's allowlist is unchanged. Declared envvars still win over the ambient copy. Any other variable a container command needs — a proxy setting, a private registry token, a site variable — must be declared in the rule's envvars or listed in OXO_FLOW_ENV_PASSTHROUGH. No podman backend is wired, so PODMAN_* is not admitted.

HPC cluster submissions#

The same allowlist is applied to the job scripts oxo-flow cluster submit generates for SLURM, PBS/Torque, SGE, and LSF (issue #1108, RB-15). A scheduler forwards the submitting shell's environment to every job by default — SLURM's --export=ALL, PBS/SGE -V, LSF -env — which would hand the credentials on the submit host to every rule running on the node. The generated script therefore wraps both the rule command and the array dispatch line in an explicit env -i prefix that admits only the allowlisted names, the operator's OXO_FLOW_ENV_PASSTHROUGH list, and — for a rule running through a local container runtime — the exact container-daemon names in CONTAINER_DAEMON_ENV_VARS (the same admission the local executor makes, issue #1173), so the isolated environment a rule sees is the same on the cluster as it is locally. Declared per-rule envvars are re-exported by the command itself and are unaffected.

This is done in the script body rather than through a submit flag because SLURM rejects --export=NONE combined with an explicit list, and the other schedulers spell the equivalent differently (-V/-v/-env); one body-level mechanism covers all four HPC backends with a single shared implementation. Two caveats to note:

  • A configured [engine] shell prelude rides inside the wrapped command (the same path the local executor uses), so it executes inside the isolated bash -c and sees the same allowlisted environment as the rule body.
  • Remote dispatch paths (Kubernetes, AWS Batch, GCP Batch) do not forward the submit host's environment into a job at all: a cloud job's environment is the container image's own ENV, the declared envvars, and the OXO_FLOW_SENSITIVE_<n> secret references — never the operator's ambient environment (pinned per backend by the cloud-dispatch boundary tests, issue #1171). A future change that adds env forwarding to any of them must re-admit the declared variables through this same policy rather than passing the ambient environment through. Locally executed docker/singularity rules do run under the allowlist (their CLI wrapper executes inside the rule shell) — see Local container rules for the narrow daemon-variable admission.

Protected data trees (protected_data_trees)#

Vendor-delivered inputs (a sequencer drop, a reference bundle) must not change silently while a pipeline is running against them. Declaring

[engine]
protected_data_trees = ["raw/", "reference/"]

makes the engine treat those workdir-relative trees as protected:

  • First observation (first run/plan/dry-run after the key appears) records a baseline manifest in the checkpoint — per-file size + mtime, plus a sha256 content hash for files up to 64 MiB.
  • Every later plan/dry-run/run diffs the tree against the baseline and prints a loud ✗ PROTECTED TREE DRIFT detected in N file(s): section with one line per changed/removed/added file (a truncated file is labeled (TRUNCATED), content edits carry the old→new hash, symlink changes are labeled [symlink]).
  • run refuses to execute on drift:
Error: protected data tree drift detected (1 file(s) changed since baseline):
  raw/S1.fq: 4.0 B @2026-10-27 10:00:00 → 0 B @2026-10-27 10:05:00 (TRUNCATED)

Restore the files, or pass --allow-protected-drift to run anyway (the drift is still printed; the baseline is NOT updated — a later clean run re-reports it until the tree is restored or the checkpoint's baselines are reset by deleting the entry). To re-baseline after an intentional change, remove the protected_tree_baselines entry from .oxo-flow/checkpoint.json (or the checkpoint file itself) — the next invocation adopts the current tree content as the new baseline. - Trees that do not exist yet are adopted only once they appear — populating a tree is normal data arrival, not drift. - dry-run reports the same section read-only (it never persists the baseline and never fails on drift — only run gates).

This is a detection surface, not an access control: out-of-band writes are noticed within one invocation, not prevented. Combined with --provenance checksums it catches damaged vendor data before any rule consumes it. For a one-shot audit of a whole tree against the vendor's own md5.txt sidecar (post-incident audits, post-recovery verification), use oxo-flow verify-data. On a web-served instance the same drift state is available per run as a read-only HTTP preview: GET /api/runs/{id}/protected-drift (see Web API reference) — it never records baselines, so it is safe to poll before overriding a gate with allow_protected_drift on POST /api/runs.

Forcing Execution#

To bypass checkpoints and re-execute rules that have already completed, use oxo-flow clean to remove outputs and the checkpoint file before running, or --rerun to re-execute this run's rules while keeping checkpoint records for rules outside the run.

Checkpoint re-entry (checkpoint = true)#

A checkpoint rule discovers new values at runtime: after it completes, the engine reads its checkpoint_manifest, merges the declared samples, and executes the newly created rule instances in the same run. The checkpoint records each re-entry (reentries), so a resume replays it and reconstructs the same plan; invalidating the checkpoint rule revokes its samples until it re-runs. A missing or unparsable manifest fails the checkpoint rule. See Workflow Format for the config surface.

AI error recovery (--ai-recover)#

--ai-recover turns a rule failure into a bounded diagnose → apply → verify loop. AI can also activate implicitly: a workflow with [ai] enabled = true runs recovery on failure without the flag. The write authorization comes from [ai] auto_fix (see workflow-format); in every mode the model's own safety verdict and full workflow validation gate the write, and an unauthorized or refused fix never touches the file.

When a required rule aborts the run and recovery is active:

  1. The failure is diagnosed (--ai-max-retries bounds failed diagnosis calls) and a fix is proposed.
  2. The resolved auto_fix mode decides whether the fix is written: always applies safe fixes automatically; ask prompts y/N on an interactive terminal and is denied headless; never only reports. The generated TOML must still validate as a complete workflow before it lands.
  3. When the fix is applied, the engine prints REPAIR_APPLIED session_id=<id> — one line, on stderr and in the run log. These REPAIR_* prefixes are a stable machine-readable contract (automation and eval harnesses parse them verbatim; they are never colored or reworded).
  4. The engine then attempts one verification rerun with the updated workflow: a subprocess oxo-flow run <workflow> --resume-failed --no-ai-recover against the same checkpoint — succeeded rules stay skipped, failed rules re-run, logs interleave with the parent's. The rerun's outcome is narrated as REPAIR_RERUN_SUCCEEDED <detail>, REPAIR_RERUN_FAILED <detail> (exit code included), or REPAIR_RERUN_SKIPPED <reason> (e.g. the loop guard, or the binary could not be resolved).
  5. The outcome is recorded in the checkpoint's repair_record: {applied, session_id, rerun_attempted, rerun_succeeded, detail, timestamp}. Legacy checkpoints without the field load unchanged, and the report's ai-repair section surfaces the record (see report).

Outcome semantics: a rerun that completes the workflow exits 0 — the run finishes successfully (✓ repair verified — rerun succeeded, and the --json summary carries "status": "completed" with a repair object); a failed rerun keeps the ordinary failure path (non-zero exit, REPAIR_RERUN_FAILED).

The loop is bounded by construction: at most one verification rerun per run invocation, enforced by the --no-ai-recover flag the rerun child receives and an internal OXO_FLOW_REPAIR_RERUN guard variable. A rerun therefore never diagnoses, never rewrites the workflow, and never starts another rerun — it only verifies the fix that was already authorized. --no-ai-recover also disables recovery for a normal invocation (including the [ai] enabled auto-detect).

Exit codes#

Code Meaning
0 Success — the run finished with zero required-rule failures
1 Validation or parse failure — the workflow file could not be read, parsed, or validated
1 Run failure — a required rule failed (a plain --keep-going failure summary still exits non-zero; only required = false failures leave the exit code at 0)
130 / 143 / 129 Interrupted — SIGINT / SIGTERM / SIGHUP stopped the run. In-flight rules are recorded as interrupted and --json reports status "failed"; this holds on the local and run --profile cluster paths, and even when every interrupted rule is required = false (an interrupt is never reported as success)

Validation and run failures both exit 1; the distinction is in the stderr diagnostics (a parse error names the workflow file, a run failure lists the failed rules). Machine consumers that need to separate the two should use --json and read status. Failure-specific exit codes on other commands are documented per page — see report and test.


Notes#

  • The workflow file is optional; if not specified, auto-discovery searches for main.oxoflow first, then any *.oxoflow file alphabetically
  • If no .oxoflow file is found, an error message suggests running oxo-flow init to create one
  • The DAG is built and validated before any rules execute
  • Rules are executed in topological order; independent rules may run in parallel up to the -j limit
  • If --keep-going is not set, execution stops at the first failure
  • --keep-going changes scheduling, never the verdict: it runs the remaining rules past a failure, but the run still exits non-zero when a required rule failed — scripts and the web UI (which classifies by exit code) see the truth. Only required = false failures leave the exit code at 0.
  • The --retry flag re-runs failed jobs up to N times before marking them as failed
  • A timeout of 0 means no timeout
  • Resource constraints (threads, memory) in rules are checked against available resources before execution
  • Pre-flight budget check: When --max-threads/--max-memory is explicitly set — or the workflow declares [resource_budget] max_threads/max_memory (issue #1080) — the engine checks all rules before execution starts. Any rule whose requirements exceed the effective cap is reported immediately (fast-fail), preventing mid-pipeline failures. The workflow budget is a hard, versioned cap: it clamps a larger CLI request, a tighter CLI request still wins, and a silent CLI inherits it.
  • Backend preflight: before any rule executes, every unique environment spec of the pending rules is validated once — a missing backend (conda, mamba, pixi, docker, singularity, modules) aborts the run with one collapsed entry per root cause and "no rules were run" (see Backend preflight).
  • Auto-detected capacity is soft: Rules whose declared requests exceed the machine's detected threads/memory are not rejected — the declared request is the tool's upper bound (often an upstream HPC label), not a scheduling requirement. The engine warns and clamps the pool reservation, so an over-capacity rule runs alone (serialized) instead of blocking the workflow.
  • CPU-oversubscription warning: the engine compares each scheduled parallel wave's worst-case demand (min(-j, wave width) × heaviest rule threads, both clamped to the effective pool ceiling) with the detected CPUs and warns at ≥ 80 % — the analogue of the memory clamp warning. The estimate covers only the rules the run will execute (-t/--module filtering included). A ceiling that leaves headroom silences it; a cap at or above the host still warns. See CPU Oversubscription Warning.
  • Deadlock detection: If pending rules remain while nothing is running and none can become ready (typically an upstream failure), the engine reports Deadlock detected: N rules stuck with the stuck rule names. Resource waits cannot deadlock: over-capacity requests are clamped, and explicit budget violations fail fast before any rule runs.
  • Target-aware execution: The -t flag supports prefix matching — -t al matches all rules whose names start with "al". Use this to run a subset of the workflow, similar to make <target>. Only the named targets and their transitive upstream dependencies are executed; downstream rules are excluded.
  • Default targets: rules marked target = true are what run/dry-run build when neither -t nor --module selects anything — the marked rules and their transitive producers. With no marked rule the full DAG runs (backward compatible); an explicit -t/--module always wins.
  • Abort cancellation semantics: with fail-fast aborts, rules that were merely killed by the abort (signal death, no exit code) are recorded as cancelled, not failed — they are excluded from failed_rules and the failure counts, and narrated like skips. A sibling that failed on its own merit (exit code present) keeps its failure record. The checkpoint's per-rule rule_runs entries distinguish the two via their status/skip_reason/signal fields.
  • Setting --max-threads 0 or --max-memory 0 auto-detects system resources
  • Cgroup-aware auto-detection: on Linux the auto-detected memory ceiling is clamped to the tightest cgroup v2 memory.max applying to the engine process — a container cap, a systemd user slice (user@.service with MemoryMax=), or a systemd-run --user -p MemoryMax=… scope. Host RAM figures ignore these limits: on a 1007G box whose user slice caps sessions at 512G, scheduling against 1007G commits more memory than the kernel can back inside the cgroup and the OOM-killer tears the run down (exit 137 cascading into a fail-fast teardown of healthy rules). The engine logs a cgroup memory limit is below the detected host memory warning with both figures when the clamp bites. The same clamp applies to the {effective_memory_mb} placeholder and to container --memory ceilings. For long cohort runs that park at the cap for hours, pair this with --retry so a transient OOM does not kill the whole run. The CPU side is cgroup-aware too: the detector (num_cpus) honours affinity/cpuset and cgroup v1/v2 CPU quotas (container --cpus, systemd CPUQuota=, cpu.max), so the auto-detected thread ceiling, {effective_threads}, and the oversubscription warning already reflect a quota (issue #1080).
  • Environment setup is performed automatically before first use of each environment (conda, pixi, docker, singularity, venv)
  • Use --skip-env-setup when environments are pre-built to avoid redundant setup
  • Use --cache-dir to persist environment setup state across runs for faster startup (reclaim stale entries with oxo-flow env gc — the old post-run aging sweep was retired)