Skip to content

Troubleshooting Guide#

Common issues and their solutions when using oxo-flow.

Beginner Common Issues#

If your workflow isn't running as expected, check these common items first:

  1. Input files exist? Ensure your starting data (e.g., FASTQ files) is in the correct directory.
  2. Tools installed? If not using a managed environment (Conda/Docker), ensure tools like bwa or samtools are installed on your system.
  3. Dry-run first? Run oxo-flow dry-run workflow.oxoflow to see what oxo-flow intends to do without actually running commands.
  4. Working directory? Make sure you are running commands from the project root directory (the one containing your .oxoflow file).

"It ran but nothing happened"#

Symptom: oxo-flow run shows success but no output files appear.

Solution:

  1. Check the output directory path in your .oxoflow file — paths are relative to the workflow file location
  2. Use oxo-flow debug workflow.oxoflow to see the actual commands being run
  3. Verify the shell command actually creates the output file: 
    # Test the command manually
mkdir -p data && echo 'Hello from oxo-flow!' > data/greeting.txt
ls data/greeting.txt  # Does the file exist?

"No workflow file found"#

Symptom: oxo-flow run without arguments fails with "no .oxoflow file found"

Solution:

  1. Ensure you're in a directory containing a .oxoflow file
  2. Or explicitly specify the workflow path: oxo-flow run path/to/workflow.oxoflow
  3. Use oxo-flow init my-pipeline to create a new project if none exists

"Validate fails with parse error"#

Symptom: oxo-flow validate workflow.oxoflow shows TOML syntax errors

Common TOML mistakes:

Mistake Wrong Correct
Missing quotes name = my-rule name = "my-rule"
Wrong array syntax [rules] [[rules]]
Unclosed string shell = "echo shell = "echo hello"
Invalid table header [[rule]] [[rules]]

Use the TOML primer for syntax basics.

Workflow Parsing Errors#

TOML syntax error#

Symptom: parse error in workflow.oxoflow: ...

Solution: Check your TOML syntax. Common mistakes:

  • Missing quotes around string values
  • Incorrect array-of-tables syntax (use [[rules]], not [rules])
  • Unmatched brackets or braces

Use oxo-flow validate workflow.oxoflow to get detailed error messages.

Duplicate rule names#

Symptom: duplicate rule name: 'step1'

Solution: Every rule must have a unique name field. If you're using [[include]] directives, use the namespace field to avoid conflicts:

[[include]]
path = "shared_rules.oxoflow"
namespace = "shared"

Execution Errors#

Rule fails with non-zero exit code#

Symptom: rule 'bwa_align' failed with exit code 1

Solution:

  1. Run oxo-flow debug workflow.oxoflow -r bwa_align to see the expanded command with all variables substituted.
  2. Check the log output for stderr messages.
  3. Try running the expanded command manually in your terminal.
  4. Verify that the required tool is installed and available in the rule's environment.

Command not found#

Symptom: sh: bwa: command not found

Solution: The tool is not in the system PATH. Either:

  • Specify an environment in the rule: 
    environment = { conda = "envs/alignment.yaml" }
  • Or ensure the tool is installed and accessible.

Timeout exceeded#

Symptom: command timed out (exit code 124)

Solution: Increase the timeout via --timeout flag or allocate more resources (threads/memory) to the rule.

Wildcard Issues#

Unresolved wildcards#

Symptom: wildcard error in rule '...' (e.g. a {sample} placeholder that could not be resolved)

Solution: Ensure wildcard values are provided. Wildcards like {sample} must be resolved from:

  • Input file patterns matched against existing files
  • Explicit values in the config section
  • Scatter configuration

Wildcard constraint violation#

Symptom: wildcard 'chr' value 'invalid' does not match constraint '^chr[0-9XYM]+$'

Solution: The wildcard value doesn't match the regex constraint defined in your workflow. Check that your input filenames follow the expected naming convention.

Environment Issues#

Conda environment creation fails#

Symptom: environment error (conda): ...

Solution:

  1. Check that conda/mamba is installed: conda --version
  2. Verify the environment YAML file exists and is valid
  3. Check for network connectivity (package downloads)
  4. Try creating the environment manually: conda env create -f envs/tool.yaml

Docker image not found#

Symptom: environment error (docker): ...

Solution:

  1. Check that Docker is installed and running: docker info
  2. Verify the image reference: docker pull quay.io/biocontainers/bwa:0.7.19--h577a1d6_1
  3. Check for authentication if using private registries

HPC modules not available#

Symptom: Module load errors when using modules in environment spec

Solution: Verify that the module system is available on your HPC node and that the specified module names and versions are correct:

module avail gcc
module avail cuda

DAG Issues#

Cycle detected#

Symptom: cycle detected in workflow DAG: A → B → C → A

Solution: Your rules have circular dependencies. The error message shows the full cycle path with → arrows connecting each rule in the loop.

  1. Use oxo-flow graph workflow.oxoflow to visualize the DAG
  2. Pick one edge in the cycle and decide how to break it:
  3. File-based edge: If A produces a file that B consumes, and B produces a file that A consumes, rename one output to break the match
  4. Explicit depends_on: Remove the depends_on entry that closes the cycle
  5. If the cycle is intentional (e.g., iterative refinement), split the rule into separate pre/post steps
  6. Re-validate: oxo-flow validate workflow.oxoflow

Missing input#

Symptom: missing input for rule 'step2': intermediate.txt

Solution: Ensure that some other rule produces intermediate.txt as an output, or that the file already exists before the workflow runs.

Rule not found#

Symptom: rule not found: 'algn' with a list of available rule names

Solution: The target name passed to -t or referenced in depends_on doesn't match any rule. oxo-flow supports prefix matching — try a shorter prefix:

# Instead of guessing the exact name
oxo-flow run pipeline.oxoflow -t align  # matches "align_reads", "align_bwa", etc.

Use oxo-flow graph workflow.oxoflow to see all rule names.

Rules not running in parallel#

Symptom: Workflow executes rules one-at-a-time despite -j 8

Causes and solutions:

  1. DAG is naturally sequential: Run oxo-flow graph workflow.oxoflow and check Width in the header. If width=1, every rule depends on the previous one — no parallelism is possible. Consider splitting large rules into independent sub-tasks.
  2. Resource constraints: If rules declare high thread/memory requirements (e.g., 32 threads each on a 64-thread machine), the resource pool may only allow 1-2 concurrent jobs. Either reduce declarations or increase --max-threads/--max-memory.
  3. Implicit file dependencies: Check that intermediate output files use unique names — if two rules produce the same output file, the engine reports an Output pattern collision error and refuses to run.

Orphan rules (rules that never connect)#

Symptom: A rule exists in the workflow but has no connections to other rules — neither consuming their outputs nor producing inputs for them.

Detection: Use oxo-flow graph workflow.oxoflow -f tree and look for rules with no upstream or downstream indicators, or run oxo-flow clean --orphans to find them.

Solution: Check input/output paths for typos. An orphan is usually a misspelled file path that prevents the engine from matching it to other rules.

Output collisions#

Symptom: Output pattern collision: rules 'caller_a' and 'caller_b' both produce '{sample}.vcf'

Solution: Two rules produce files matching the same pattern. This is dangerous — the second rule to finish will overwrite the first rule's output. Give each rule distinct output directories:

# Before (collision)
[[rules]]
name = "caller_a"
output = ["variants/{sample}.vcf"]

[[rules]]
name = "caller_b"
output = ["variants/{sample}.vcf"]  # ❌ Same pattern!

# After (fixed)
[[rules]]
name = "caller_a"
output = ["variants/caller_a/{sample}.vcf"]  # ✅ Unique path

[[rules]]
name = "caller_b"
output = ["variants/caller_b/{sample}.vcf"]  # ✅ Unique path

Deadlock detected#

Symptom: Deadlock detected: 3 rules stuck. Stuck rules: align_S001, align_S002, align_S003. Check resource constraints (threads/memory) and dependencies.

Solution: Pending rules are stuck because none can become ready — typically an upstream rule failed and its dependents stay pending forever (resource waits cannot deadlock: over-capacity requests are clamped, and explicit budget violations fail fast before any rule runs). Check:

  1. oxo-flow status for failed upstream rules (the stuck rules' dependencies)
  2. Dependency declarations (depends_on) that may never be satisfiable
  3. Re-run with --keep-going to surface all upstream failures at once

Resource budget exceeded#

Symptom: rule 'bwa_align' requires 64 threads but --max-threads caps the run at 32

Solution: The pre-flight budget check caught a rule whose requirements exceed the explicit limits. Either:

# Increase the budget
oxo-flow run pipeline.oxoflow --max-threads 64

# Or reduce the rule's requirement in the .oxoflow file
[rules.resources]
threads = 32

Note: This check only fires when --max-threads/--max-memory are explicitly set on the CLI. Auto-detected system resources don't trigger budget failures (only warnings).

Checkpoint and Resume#

Resuming a failed workflow#

After fixing the cause of a failure, re-run the same workflow. oxo-flow will check checkpoints and skip already-completed rules:

oxo-flow run workflow.oxoflow

Clearing checkpoint state#

To force a full re-run, delete the checkpoint file:

rm -rf .oxo-flow/checkpoint.json
oxo-flow run workflow.oxoflow

Changed inputs rebuild automatically#

Re-running does not blindly reuse completed rules: the checkpoint records the file set each rule's inputs resolved to (paths + size + mtime, plus a content hash for files up to 64 MiB). When a glob or directory input gained or lost files, or a plain input file was edited — including same-size rewrites that preserve the mtime — the rule and its downstream re-execute, even though their outputs still exist, and the console prints input changes invalidated N rule(s). See Input changes and manifest invalidation.

To see the blast radius before spending compute, dry-run predicts exactly which rules would re-run (including the downstream cascade) and how much completed work stays protected.

Performance Tips#

Workflow runs slowly#

  1. Increase parallelism: Use -j to run more jobs concurrently: 

    oxo-flow run workflow.oxoflow -j 8

  2. Check resource constraints: Use oxo-flow debug to verify that resource requirements are reasonable.

  3. Use caching: a rule with cache_key reuses its outputs from the content cache when the key, inputs, outputs, and rendered command hash identically to a previous run — bump the key when a dependency's behavior changes without touching the declared inputs. (pipe is parsed but not wired up yet — do not rely on it.)

Memory issues with large workflows#

For workflows with many samples (>1,000):

  1. Process samples in batches using scatter/gather patterns
  2. Increase system memory limits
  3. Use cluster backends for distributed execution

Getting Help#

  • Run oxo-flow --help for CLI usage
  • Run oxo-flow <command> --help for subcommand details
  • Run oxo-flow debug workflow.oxoflow to inspect resolved commands
  • Check LIMITATIONS.md for known limitations
  • Open an issue for bugs or feature requests

Reporting Real-World Issues#

We particularly value feedback from real-world deployments. If you encounter issues in your actual bioinformatics workflows (as opposed to test examples), please use the [Real-World Testing] prefix in your issue title:

[Real-World Testing] SLURM GPU job scheduling fails on cluster with multiple partitions
[Real-World Testing] Conda environment detection issue with custom channels

Include details about your cluster type, oxo-flow version, and a description of what happened versus what you expected. See our Contributing Guide for more guidance on providing effective feedback.