Skip to content

Glossary#

Key terms and concepts used in oxo-flow documentation.


Core Concepts#

DAG (Directed Acyclic Graph)#

A mathematical representation of a workflow where each node is a rule (task) and edges represent dependencies. "Acyclic" means there are no circular dependencies — a rule cannot depend on itself through a chain of other rules. oxo-flow automatically builds a DAG from input/output file relationships.

Example: If rule A produces aligned.bam and rule B needs aligned.bam as input, there's an edge A → B in the DAG.


Rule#

A single processing step in a workflow. Each rule defines:

  • Input: Files needed to run the step
  • Output: Files produced by the step
  • Shell/Script: The command to execute
  • Environment: The software environment to use

Example: A bwa_align rule takes FASTQ files as input and produces a BAM file as output.


Workflow#

The complete pipeline definition in a .oxoflow TOML file. A workflow contains:

  • Metadata (name, version, author)
  • Configuration variables
  • Default settings
  • One or more rules

Wildcard#

A placeholder pattern like {sample} that expands to concrete values based on input files or explicit configuration. Wildcards enable writing one rule template that processes many samples.

Example: input = ["raw/{sample}.fastq.gz"] expands to raw/S001.fastq.gz, raw/S002.fastq.gz, etc.


Execution#

Topological Sort#

An ordering of DAG nodes where every node appears before any nodes that depend on it. This determines the execution order — dependencies must run before the rules that need them.

Example: If A → B → C, execution order is A, B, C.


Parallel Groups#

Sets of rules that can run simultaneously because they have no dependencies on each other. oxo-flow's -j flag controls how many rules from a parallel group can run at once.

Example: fastqc and fastp_trim can run in parallel if neither depends on the other.


Checkpoint#

A persistent record of which rules have completed. Checkpoints enable resuming workflows after failures without re-running successful rules.

Besides rule status, a checkpoint stores config snapshots, rule fingerprints, and input file manifests — the engine compares them on every run to invalidate rules whose config, definition, or input file set changed. It also records tombstones: rules marked temporary = true whose outputs were deleted after a successful run (once their dependents completed, and — with [engine] disk thresholds configured — only when free space was below reclaim_free_disk), so later runs skip them unless a dependent needs the outputs again. For provenance it stores the workflow git repo's HEAD SHA (workflow_git_sha) and, per pinned git [[include]], the checkout's resolved HEAD SHA (include_resolutions) — evidence that module content did not drift between a run and its resume.

File location: .oxo-flow/checkpoint.json


Fingerprint#

A hash of a rule's config snapshot, definition, and input manifest recorded in the checkpoint. On every run the engine compares the stored fingerprint against the current state to decide whether a completed rule can be skipped or must re-execute.


Tombstone#

A checkpoint record for a rule marked temporary = true whose outputs were deleted after a successful run (once their dependents completed, and — with [engine] disk thresholds configured — only when free space was below reclaim_free_disk). Later runs skip tombstoned rules unless a dependent needs the outputs again.


Environments#

Environment Backend#

A system for managing software dependencies. oxo-flow supports:

Backend Use Case
conda Bioinformatics tools with complex dependencies
mamba Faster, C++ reimplementation of conda with parallel dependency solving
pixi Fast, modern alternative to conda
docker Full container isolation
singularity HPC-friendly containers
venv Python-only tools
system Default system shell — no isolation (used when no backend is declared)
modules HPC environment modules (Lmod/Environment Modules)

Environment Spec#

A declaration in a rule that specifies which environment to use.

Example: environment = { conda = "envs/bwa.yaml" }


Environment YAML#

A conda environment definition file listing packages and versions.

Example:

name: bwa_env
channels:
  - bioconda
  - conda-forge
dependencies:
  - bwa-mem2=2.2.1
  - samtools=1.20


Workflow Format#

TOML (Tom's Obvious Minimal Language)#

A configuration file format used for .oxoflow files. Key features:

  • Tables: [name] defines a section
  • Arrays of Tables: [[name]] defines multiple items (used for rules)
  • Key-Value: key = "value"

See TOML Specification for full syntax.


.oxoflow#

The file extension for oxo-flow workflow definitions. These files are TOML format and define the complete pipeline.


[defaults]#

A section that sets default values (threads, memory, environment) for all rules unless overridden.


[[pairs]]#

A section defining experiment-control sample pairs for comparative analyses like tumor-normal variant calling.

Fields:

  • pair_id: Unique identifier for the pair
  • experiment: Sample name (e.g., tumor)
  • control: Matched control sample (e.g., normal)

[[sample_groups]]#

A section organizing samples into named groups for cohort studies.

Fields:

  • name: Group name (e.g., "treatment", "control")
  • samples: Array of sample identifiers

transform#

An operator that unifies scatter (split) → map (process) → gather (combine) patterns in a single rule declaration.

Example: Split by chromosome, call variants per chromosome, then combine VCFs.


Placeholders#

{input}#

Expands to all input files, space-separated.

Example: {input} → sample_R1.fastq.gz sample_R2.fastq.gz


{input[N]}#

The Nth input file (0-indexed).

Example: {input[0]} → sample_R1.fastq.gz


{input.name}#

Named input file — input given as an inline map (input = { reads = "..." }).

Example: {input.reads} → the value of the reads entry in the rule's input map


{output}#

Expands to all output files, space-separated.


{output[N]}#

The Nth output file (0-indexed).

Example: {output[0]} → aligned/S001.bam


{output.name}#

Named output file — output given as an inline map (output = { bam = "..." }).

Example: {output.bam} → the value of the bam entry in the rule's output map


{threads}#

The thread count for the current rule.


{memory}#

The memory allocation for the current rule.


{effective_threads}#

Declared threads clamped to the machine's CPUs — the tool-facing concurrency. Use for flags like --threads when the declared value is an HPC-scale label; unset threads render as 1.


{effective_memory_mb}#

Declared memory clamped to the machine's total memory, as an integer number of MB — the tool-facing heap budget (e.g. for JVM -Xmx flags). Unset memory renders as the machine's full total.


{log}#

Path of the rule's log field; per-instance wildcards and {config.x} values are expanded, and the parent directory is created automatically.


{config.*}#

A configuration variable from the [config] section (plain value, declared default, or CLI override).

Example: {config.reference} → /data/ref/hg38.fa


{meta.<column>}#

Sample-group metadata column in [[sample_groups]] shell commands — same value as {wildcard.<column>} for group metadata. See the canonical placeholder table.


{wildcard.*}#

Per-instance binding of a wildcard key — sample-group metadata entries and expansion keys, under the same grammar as when conditions. Unbound keys render as their literal brace token.

Example: {wildcard.umi1_pattern} → the umi1_pattern metadata entry of the sample group


Output#

Report#

A structured output document (HTML, JSON, PDF) summarizing workflow execution, QC metrics, and results. Clinical reports include variant classifications and audit trails.


Provenance#

A record of how each output was produced: which inputs, which tool, which version, which parameters. Essential for reproducibility.


Benchmark#

Timing and resource usage data for each rule (wall time, memory, CPU).


Cluster/HPC#

Backend#

A cluster scheduler system. oxo-flow supports:

Backend Scheduler
slurm SLURM (SLURM Workload Manager)
pbs PBS/Torque
sge Sun Grid Engine / Oracle Grid Engine
lsf IBM Spectrum LSF
kubernetes / k8s Kubernetes
aws-batch AWS Batch
gcp-batch Google Cloud Batch

Partition/Queue#

A group of cluster nodes with specific characteristics. Jobs are submitted to a partition/queue.

Example: -q compute submits to the "compute" partition.


Wall-time#

The maximum time a job can run before being terminated by the scheduler.

Example: time_limit = "24h" → 24-hour wall-time limit.


Executor backend#

A pluggable execution adapter (ExecutorBackend) that maps the engine's static plan onto a scheduler API — submit, poll, cancel, logs. The first implementation targets SLURM/PBS/SGE/LSF; the plan itself stays executor-agnostic, so the dry-run preview is valid for every backend.

Static plan#

The fully determined execution plan computed before any execution: topological order, parallel groups, per-rule resource declarations, and the invalidation set. Executors consume the plan without re-deriving it.

Checkpoint re-entry#

A bounded dynamic-DAG mechanism (Workflow Format): a checkpoint = true rule writes a manifest at runtime declaring new samples; the engine merges them, re-expands the rule templates, and executes the new instances in the same run. Every round is still a static plan; the checkpoint records the rounds so resumes replay (and revoke) them deterministically.

Remote manifest entry#

The content identity of an s3:// / gs:// / az:// input recorded in a rule's input manifest: (scheme, key, size, etag) — S3 ETag (raw hex, possibly a composite multipart hash), GCS md5Hash, or Azure Content-MD5 (base64 for both GCS and Azure). Same-size remote rewrites invalidate the rule exactly as local content changes do (Cloud Storage).


Diagnostic Codes#

Every static check in oxo-flow tags its findings with a stable code so tools (and you) can branch on them programmatically:

  • Exx — an error: the workflow is rejected (validate fails) or flagged as not runnable. Example: E017 — an unknown key in a rule, the engine's way of saying "this TOML uses a dialect the schema does not have".
  • Wxx — a warning: the workflow runs, but the check found a likely mistake worth fixing (see oxo-flow lint).

Codes are three characters: the letter, then a zero-padded number. The authoritative, machine-readable source is the --json output of each command (validate --json, dry-run --json, lint --json); the tables below list every code the engine currently emits.

Validation errors (validate / dry-run)#

Code Meaning
E001 Workflow name is empty
E002 A rule definition failed engine validation — e.g. gpus declared without a docker image, or output_pattern combined with output/transform (mutually exclusive)
E003 A wildcard appears in a rule's output but not in its input — the engine cannot key instances
E004 Invalid memory / resources.memory specification. Parse rejects unparseable values before the diagnostic layer runs, so this code is defense-in-depth for programmatically constructed configs
E005 A {config.key} reference points at a key absent from [config]
E006 DAG construction error (e.g. a cycle)
E007 depends_on names a rule that does not exist
E008 extends names a rule that does not exist
E009 An input path contains .. and may escape the working directory
E010 A rule references an undefined env_group. The same code also flags an absolute input file that does not exist (a missing relative input gets W020 instead — see the lint table)
E011 A rule command matches a dangerous shell pattern (destructive command class). Recursive deletions (rm -rf/rm -r) are only blocked when the target resolves OUTSIDE the run workdir — pipelines may delete their own outputs (see Security model)
E013 A checkpoint = true rule lacks checkpoint_manifest. Emitted at warning severity — checkpoint = true predates the manifest field, so legacy workflows keep validating (it appears in --json warnings, not errors)
E014 A checkpoint rule is parameterized by {sample}/{group}/{pair_id} (not allowed)
E015 Re-entry declares a pair_id with conflicting content
E016 An environment field contains a shell-unsafe character — the spec would not render as safe argv; the same code also labels a re-entry pair_id colliding with an existing instance name (see Checkpoint re-entry)
E017 Unknown key in a rule — usually a foreign workflow dialect (e.g. foreach, inputs = {...}). A hard parse error; the message names the offending keys and points at the workflow format reference
E018 A when condition uses an unrecognized bare identifier (e.g. params_mode == 1) — the evaluator would treat it as true and the rule would silently run. Reference config keys as config.key, or use a quoted literal, a {...} placeholder, or a built-in function
E019 A rule's declared pixi manifest does not exist (resolved relative to the workflow directory). validate and plan/dry-run preflight it the same way they existence-check static inputs; run fails before executing anything
E020 A [wildcard_constraints] entry is not a valid regex (e.g. the impossible range [z-a]+). Parse rejects such entries for TOML-loaded configs, so this diagnostic is defense-in-depth for programmatically constructed configs — it runs the SAME shared compilation helper the plan-time fan-out and runtime projection use, making the validate-time error byte-identical to the run-time one
E021 A rule's input/output pattern uses Snakemake-style inline wildcard constraint syntax (e.g. {sample:[A-Z]+}). The engine's wildcard grammar is {name} only — a :-carrying brace group is never parsed as a wildcard, so it stays literal brace text and the tool fails on a path that looks valid. Constraints belong in [wildcard_constraints]
E022 A rule has no runnable command: shell is missing, empty, or whitespace-only and no non-empty script is declared. E002 sees shell = "" as a present field, and the engine then skips the rule at run time ("no shell or script defined") — this code makes that state a hard validation error. Rules driven by transform are exempt (their command lives in transform.map/transform.combine). The classic trigger is an imported Nextflow process whose script: block was not captured

Lint warnings (lint)#

Code Meaning
W001/W002 Workflow missing description / author
W003 Rule missing a description
W004 (info) Rule has a shell command but no log file specified — stdout/stderr are still captured into job records
W005 Rule uses >8 threads with no memory
W006 (info) Rule name uses hyphens — suggests snake_case (underscores)
W007 (info) Leaf rule (no dependents) that could be marked target = true
W008 (info) Rule declares no environment — runs in the bare system shell
W009 Rule uses >32 threads but has no memory — high-thread jobs typically need significant memory
W010 Rule has checkpoint = true but no output files
W012 Rule has retries but no retry_delay (retries execute immediately)
W013 (info) Rule has an on_failure hook but no retries — the hook runs on first failure
W014 depends_on references an unknown rule
W015 Rule extends an unknown base rule — validate reports the same condition as E008
W016 Conda/pixi spec is not a lockfile — builds may not be reproducible
W017 Input path is absolute
W018 Input path references a home directory
W019 Rule executes a command but declares no outputs
W020/W021/W022 A missing relative input file (W020), or a pre_exec/on_success/on_failure hook containing a dangerous pattern. W021 also labels a script that references another rule's output without an ordering edge (depends_on)
W023 Shell command uses flagged idioms ($(), backticks, rm -rf, chmod 777, eval, curl-pipe-shell) — aggregated one diagnostic per command; common in bioinformatics scripts, verify intentional
W024 A wildcard has no declared source — it will stay literal at run time. Also surfaced by validate (as a warning, exit 0), so the first gate in the validate → dry-run → run path is not silent about it
W025 Legacy threads =/memory = keys under the rule body (use resources.)
W027 A when condition references a wildcard nothing can bind — evaluates false
W028 Docker image is not fully qualified (resolves against docker.io)
W029 len(config.x) compared against a non-numeric value
W030 Malformed regex_extract in a when condition
W031 Producer is when-gated but its consumer expands the output unconditionally
W032 A config key looks like a secret but is not declared sensitive
W033 Two rules write the same output path (differently-named wildcards over one template count too) — second writer silently overwrites the first
W034 An {input[N]}/{output[N]} index is out of range, or a named {input.key}/{output.key} is not declared — renders as literal brace text and the tool fails on a path that looks valid
W035 A command reads a shell glob (e.g. cat stats/*.count.txt) while the rule declares no inputs — globs are expanded by the shell at run time and create no DAG dependency, so the rule may run before the files it reads exist
W036 A rule declares input_function, but the engine performs no dynamic input resolution — the field is parsed and round-tripped yet never called, so the input list stays exactly as declared (the declared function name does not fill it at run time)
W037 A resource_hint sets runtime or io_bound — both are parsed but consumed by nothing; the scheduler only uses input_size and memory_scale today
W038 A rule sets format_hint — parsed but consumed by nothing; the engine performs no format-specific I/O today. pipe is NOT flagged (since #997 it selects staging_mode = "stream_fifo")
W039 A dotted {namespace.key} placeholder uses an unrecognized namespace (e.g. {metadata.batch} instead of {meta.batch}) — the token is never substituted and the tool receives literal brace text
W040 A rule has no runnable command — shell is missing or empty/whitespace-only and no non-empty script is declared; the engine skips the rule at run time. The lint mirror of E022; transform rules are exempt
W041 A rule references the sample domain ({sample}, {group}, {meta.<column>}, {wildcard.sample}, {wildcard.group}) with no binding source — the rule runs once and the token stays unbound (the execution-time gate rejects it). Bind the domain in input/output (a uniquely-keying {meta.*} OUTPUT reference or a one-row constant table also binds), declare the source (sample_pattern/pairs_pattern declarations count without data), or use singleton = true to silence the lint diagnosis for an intentional one-instance {meta.*} read (the runtime gate still requires a resolvable value). {sample}/{group} stay flagged even for a singleton

Schema-check codes (S001–S007) come from the raw-file schema pass, reachable through oxo-flow lint (not validate, which reports only parse-level errors): S001 invalid TOML syntax, S002 missing [workflow], S003 missing workflow name, S004 malformed [[rules]], S005 a rule entry that is not a table, and S006 an unknown top-level section (checked against the parser's own E017 whitelist — the documented sections are never flagged; since the parsed config cannot retain unknown keys, S006 is the one schema code users can actually trigger). S007 warns when [workflow].format_version is present but unrecognized, and S008 is the secret-scanning warning: a token-shaped pattern in the raw file (security); like the whole S-series, it reaches users only through the lint pass.

The full, current list with suggestions is best read from the commands: oxo-flow lint --json prints every code with its message and suggestion (see lint). AI-generated drafts that fail a gate have their errors fed back to the model for correction (AI CLI); deterministic repairs for purely mechanical classes run first, before any model round is spent.

Benchmark terms#

  • pass@1 / pass@k — the probability that a single generation (or at least one of k attempts) passes all gates. Used in AI evaluation and eval/frontier/.
  • fidelity — a deterministic check that the tools an intent names actually appear in the generated pipeline. Gates alone cannot see a "wrong but valid" pipeline; the frontier benchmark reports success = gates ∧ fidelity.
  • seed — one repeat of the same intent under identical settings; multiple seeds measure sampling variance instead of quoting a single lucky/unlucky run.

See Also#