Skip to content

Troubleshooting Guide#

Common issues and their solutions when using oxo-flow.

Beginner Common Issues#

If your workflow isn't running as expected, check these common items first:

  1. Input files exist? Ensure your starting data (e.g., FASTQ files) is in the correct directory.
  2. Tools installed? If not using a managed environment (Conda/Docker), ensure tools like bwa or samtools are installed on your system.
  3. Dry-run first? Run oxo-flow dry-run workflow.oxoflow to see what oxo-flow intends to do without actually running commands.
  4. Working directory? Make sure you are running commands from the project root directory (the one containing your .oxoflow file).

"It ran but nothing happened"#

Symptom: oxo-flow run shows success but no output files appear.

Solution:

  1. Check the output directory path in your .oxoflow file — paths are relative to the workflow file location
  2. Use oxo-flow debug workflow.oxoflow to see the actual commands being run
  3. Verify the shell command actually creates the output file: 
    # Test the command manually
mkdir -p data && echo 'Hello from oxo-flow!' > data/greeting.txt
ls data/greeting.txt  # Does the file exist?

"No workflow file found"#

Symptom: oxo-flow run without arguments fails with "no .oxoflow file found"

Solution:

  1. Ensure you're in a directory containing a .oxoflow file
  2. Or explicitly specify the workflow path: oxo-flow run path/to/workflow.oxoflow
  3. Use oxo-flow init my-pipeline to create a new project if none exists

"Validate fails with parse error"#

Symptom: oxo-flow validate workflow.oxoflow shows TOML syntax errors

Common TOML mistakes:

Mistake Wrong Correct
Missing quotes name = my-rule name = "my-rule"
Wrong array syntax [rules] [[rules]]
Unclosed string shell = "echo shell = "echo hello"
Invalid table header [[rule]] [[rules]]

Use the TOML primer for syntax basics.

Workflow Parsing Errors#

TOML syntax error#

Symptom: parse error in workflow.oxoflow: ...

Solution: Check your TOML syntax. Common mistakes:

  • Missing quotes around string values
  • Incorrect array-of-tables syntax (use [[rules]], not [rules])
  • Unmatched brackets or braces

Use oxo-flow validate workflow.oxoflow to get detailed error messages.

Duplicate rule names#

Symptom: duplicate rule name: 'step1'

Solution: Every rule must have a unique name field. If you're using [[include]] directives, use the namespace field to avoid conflicts:

[[include]]
path = "shared_rules.oxoflow"
namespace = "shared"

Execution Errors#

Rule fails with non-zero exit code#

Symptom: rule 'bwa_align' failed with exit code 1

Solution:

  1. Run oxo-flow debug workflow.oxoflow -r bwa_align to see the expanded command with all variables substituted.
  2. Check the log output for stderr messages.
  3. Try running the expanded command manually in your terminal.
  4. Verify that the required tool is installed and available in the rule's environment.

Command not found#

Symptom: sh: bwa: command not found

Solution: The tool is not in the system PATH. Either:

  • Specify an environment in the rule: 
    environment = { conda = "envs/alignment.yaml" }
  • Or ensure the tool is installed and accessible.

Timeout exceeded#

Symptom: command timed out (exit code 124)

Solution: Increase the timeout via --timeout flag or allocate more resources (threads/memory) to the rule.

Rarefaction depth exceeds all samples (QIIME2 core diversity)#

Symptom: core-metrics-phylogenetic (or any rule running QIIME2 rarefaction) fails with:

The rarefied table contains no samples or features. Verify your table is
valid and that you provided a shallow enough sampling depth.

Cause: every sample in the feature table has fewer merged reads than sampling_depth, so the rarefied table is empty. This commonly bites pilot datasets run with a workflow default (e.g. 1000) sized for real cohort data.

Solution:

  1. Inspect per-sample depths (feature-table summarize / the workflow's summary rule) to find the actual maximum
  2. Lower the depth: oxo-flow run workflow.oxoflow sampling_depth=500
  3. Re-run — config-tracking re-executes only the affected rule and its dependents; earlier rules are skipped from checkpoint

Wildcard Issues#

Unresolved wildcards#

Symptom: wildcard error in rule '...' (e.g. a {sample} placeholder that could not be resolved)

Solution: Ensure wildcard values are provided. Wildcards like {sample} must be resolved from:

  • Input file patterns matched against existing files
  • Explicit values in the config section
  • Scatter configuration

Wildcard constraint violation#

Symptom: wildcard 'chr' value 'invalid' does not match constraint '^chr[0-9XYM]+$'

Solution: The wildcard value doesn't match the regex constraint defined in your workflow. Check that your input filenames follow the expected naming convention.

Environment Issues#

Run aborts before any rule: environment backend unavailable#

Symptom: N environment backend(s) required by pending rules are unavailable; no rules were run: followed by one entry per root cause, e.g.

  - conda: conda is not installed or not in PATH — required by 3 pending rule(s), e.g. `bwa_align` (environment: envs/alignment.yaml); install conda and ensure it is on PATH

Cause: a fail-fast preflight runs after DAG construction and before any rule executes — every unique environment spec of the pending rules is validated once, and the run aborts when a required backend binary is missing. Nothing runs, so no rule can fail halfway on a missing tool. Rules sharing one broken spec, and distinct specs failing with the same backend-missing message, are collapsed into a single line so the operator fixes one root cause per entry.

Solution:

  1. Install the missing backend (or fix its PATH) — see the per-backend sections below for each tool's check command
  2. Re-run; the preflight re-validates only the specs still pending
  3. If the backend is genuinely absent on this machine, switch the rule's environment to one the machine supports

Conda environment creation fails#

Symptom: environment error (conda): ...

Solution:

  1. Check that conda/mamba is installed: conda --version
  2. Verify the environment YAML file exists and is valid
  3. Check for network connectivity (package downloads)
  4. Try creating the environment manually: conda env create -f envs/tool.yaml

Docker image not found#

Symptom: environment error (docker): ...

Solution:

  1. Check that Docker is installed and running: docker info
  2. Verify the image reference: docker pull quay.io/biocontainers/bwa:0.7.19--h577a1d6_1
  3. Check for authentication if using private registries

HPC modules not available#

Symptom: Module load errors when using modules in environment spec

Solution: Verify that the module system is available on your HPC node and that the specified module names and versions are correct:

module avail gcc
module avail cuda

DAG Issues#

Cycle detected#

Symptom: cycle detected in workflow DAG: A → B → C → A

Solution: Your rules have circular dependencies. The error message shows the full cycle path with → arrows connecting each rule in the loop.

  1. Use oxo-flow graph workflow.oxoflow to visualize the DAG
  2. Pick one edge in the cycle and decide how to break it:
  3. File-based edge: If A produces a file that B consumes, and B produces a file that A consumes, rename one output to break the match
  4. Explicit depends_on: Remove the depends_on entry that closes the cycle
  5. If the cycle is intentional (e.g., iterative refinement), split the rule into separate pre/post steps
  6. Re-validate: oxo-flow validate workflow.oxoflow

Missing input#

Symptom: missing input for rule 'step2': intermediate.txt

Solution: Ensure that some other rule produces intermediate.txt as an output, or that the file already exists before the workflow runs.

Missing required source input (fail-fast at run start)#

Symptom: workflow is missing required source input(s) — no rule produces them: rule 'qc': input 'raw/S1.fq' does not exist and no rule produces it

Cause: A required rule input resolves to files that are absent and no rule in the DAG produces them — typically deleted raw data, a mistyped path, or a sample_pattern that no longer matches. The run aborts before any rule executes instead of "succeeding" from stale checkpoint outputs.

Solution:

  1. Restore the source files (re-download, re-copy, or fix the path in the workflow config)
  2. Check the sample_pattern/samples declaration if the file layout changed
  3. If the input is genuinely optional, mark the rule optional = true
  4. Re-run oxo-flow validate workflow.oxoflow to confirm the fix

Note: when --workdir points somewhere other than the workflow directory, source files that live in the workflow repo (declared rule inputs with no producer, and interpreter scripts) are linked into the workdir automatically as symlinks at run start — you never hand-copy them, and the gate only fires for data that is genuinely absent. Companion index files that tools auto-derive from a linked file's path (ref.fa.fai/ref.fa.dict, bwa's .bwt/.amb/.ann/.pac/.sa, .tbi/.csi/.gzi/.crai/.bai, .mmi) are linked alongside it.

Rule not found#

Symptom: rule not found: 'algn' with a list of available rule names

Solution: The target name passed to -t or referenced in depends_on doesn't match any rule. oxo-flow supports prefix matching — try a shorter prefix:

# Instead of guessing the exact name
oxo-flow run pipeline.oxoflow -t align  # matches "align_reads", "align_bwa", etc.

Use oxo-flow graph workflow.oxoflow to see all rule names.

Rules not running in parallel#

Symptom: Workflow executes rules one-at-a-time despite -j 8

Causes and solutions:

  1. DAG is naturally sequential: Run oxo-flow graph workflow.oxoflow and check Width in the header. If width=1, every rule depends on the previous one — no parallelism is possible. Consider splitting large rules into independent sub-tasks.
  2. Resource constraints: If rules declare high thread/memory requirements (e.g., 32 threads each on a 64-thread machine), the resource pool may only allow 1-2 concurrent jobs. Either reduce declarations or increase --max-threads/--max-memory.
  3. Implicit file dependencies: Check that intermediate output files use unique names — if two rules write the same output path, one silently overwrites the other (the engine does not refuse to run; oxo-flow lint flags it as W033).

Orphan rules (rules that never connect)#

Symptom: A rule exists in the workflow but has no connections to other rules — neither consuming their outputs nor producing inputs for them.

Detection: Use oxo-flow graph workflow.oxoflow -f tree and look for rules with no upstream or downstream indicators. (oxo-flow clean --orphans targets a different thing — leftover partial-transfer chunk directories under .oxo-flow/chunks/, not disconnected rules.)

Solution: Check input/output paths for typos. An orphan is usually a misspelled file path that prevents the engine from matching it to other rules.

Output collisions (silent overwrite)#

Symptom: Two rules declare outputs that resolve to the same file — e.g. both write variants/sample.vcf (possibly via differently-named wildcards like {smp} and {sample} over the same directory). validate, graph, and run all pass — the run reports success, but the second rule to finish overwrote the first rule's output, and downstream rules consumed the wrong file.

Detection: The engine treats multi-producer outputs as legitimate (shared staging directories, multi-tool fan-ins), so it does not refuse to run. oxo-flow lint flags every colliding pair instead:

$ oxo-flow lint pipeline.oxoflow
  warning [W033]: output collision: rules 'caller_a' ('variants/{smp}.vcf') and 'caller_b' ('variants/{sample}.vcf') write the same path(s)
    hint: if both rules must stay, gate one with `when` (the intended-writer idiom) or route one to a distinct output directory

Solution: Give each rule distinct output directories — or, when the two rules are alternative writers that must never both run (e.g. WGS vs WES mode), gate them with mutually exclusive when conditions:

# Before (collision — lint reports W033)
[[rules]]
name = "caller_a"
output = ["variants/{sample}.vcf"]

[[rules]]
name = "caller_b"
output = ["variants/{sample}.vcf"]  # ❌ Same path!

# After (fixed — unique directories)
[[rules]]
name = "caller_a"
output = ["variants/caller_a/{sample}.vcf"]  # ✅ Unique path

[[rules]]
name = "caller_b"
output = ["variants/caller_b/{sample}.vcf"]  # ✅ Unique path

Alternative writers that must never both run can share a path safely when gated with mutually exclusive when conditions (see the wgs_coverage/wes_coverage idiom in the conditional execution gallery); lint still reports the pair so the intent stays visible.

Deadlock detected#

Symptom: Deadlock detected: 3 rules stuck. Stuck rules: align_S001, align_S002, align_S003. Check resource constraints (threads/memory) and dependencies.

Solution: Pending rules are stuck because none can become ready — typically an upstream rule failed and its dependents stay pending forever (resource waits cannot deadlock: over-capacity requests are clamped, and explicit budget violations fail fast before any rule runs). Check:

  1. oxo-flow status for failed upstream rules (the stuck rules' dependencies)
  2. Dependency declarations (depends_on) that may never be satisfiable
  3. Re-run with --keep-going to surface all upstream failures at once

Resource budget exceeded#

Symptom: rule 'bwa_align' requires 64 threads but --max-threads caps the run at 32

Solution: The pre-flight budget check caught a rule whose requirements exceed the explicit limits. Either:

# Increase the budget
oxo-flow run pipeline.oxoflow --max-threads 64

# Or reduce the rule's requirement in the .oxoflow file
[rules.resources]
threads = 32

Note: This check only fires when --max-threads/--max-memory are explicitly set on the CLI. Auto-detected system resources don't trigger budget failures (only warnings).

Checkpoint and Resume#

Resuming a failed workflow#

After fixing the cause of a failure, re-run the same workflow. oxo-flow will check checkpoints and skip already-completed rules:

oxo-flow run workflow.oxoflow

Clearing checkpoint state#

To force a full re-run, delete the checkpoint file:

rm -rf .oxo-flow/checkpoint.json
oxo-flow run workflow.oxoflow

Changed inputs rebuild automatically#

Re-running does not blindly reuse completed rules: the checkpoint records the file set each rule's inputs resolved to (paths + size + mtime, plus a content hash for files up to 64 MiB). When a glob or directory input gained or lost files, or a plain input file was edited — including same-size rewrites that preserve the mtime — the rule and its downstream re-execute, even though their outputs still exist, and the console prints input changes invalidated N rule(s). See Input changes and manifest invalidation.

To see the blast radius before spending compute, dry-run predicts exactly which rules would re-run (including the downstream cascade) and how much completed work stays protected.

Performance Tips#

Workflow runs slowly#

  1. Increase parallelism: Use -j to run more jobs concurrently: 

    oxo-flow run workflow.oxoflow -j 8

  2. Check resource constraints: Use oxo-flow debug to verify that resource requirements are reasonable.

  3. Use caching: a rule with cache_key reuses its outputs from the content cache when the key, inputs, outputs, and rendered command hash identically to a previous run — bump the key when a dependency's behavior changes without touching the declared inputs. (pipe is parsed but not wired up yet — do not rely on it.)

Memory issues with large workflows#

For workflows with many samples (>1,000):

  1. Process samples in batches using scatter/gather patterns
  2. Increase system memory limits
  3. Use cluster backends for distributed execution

Getting Help#

  • Run oxo-flow --help for CLI usage
  • Run oxo-flow <command> --help for subcommand details
  • Run oxo-flow debug workflow.oxoflow to inspect resolved commands
  • Check LIMITATIONS.md for known limitations
  • Open an issue for bugs or feature requests

Reporting Real-World Issues#

We particularly value feedback from real-world deployments. If you encounter issues in your actual bioinformatics workflows (as opposed to test examples), please use the [Real-World Testing] prefix in your issue title:

[Real-World Testing] SLURM GPU job scheduling fails on cluster with multiple partitions
[Real-World Testing] Conda environment detection issue with custom channels

Include details about your cluster type, oxo-flow version, and a description of what happened versus what you expected. See our Contributing Guide for more guidance on providing effective feedback.