Skip to content

Troubleshooting Guide#

Common issues and their solutions when using oxo-flow.

Beginner Common Issues#

If your workflow isn't running as expected, check these common items first:

  1. Input files exist? Ensure your starting data (e.g., FASTQ files) is in the correct directory.
  2. Tools installed? If not using a managed environment (Conda/Docker), ensure tools like bwa or samtools are installed on your system.
  3. Dry-run first? Run oxo-flow dry-run workflow.oxoflow to see what oxo-flow intends to do without actually running commands.
  4. Working directory? Make sure you are running commands from the project root directory (the one containing your .oxoflow file).

"It ran but nothing happened"#

Symptom: oxo-flow run shows success but no output files appear.

Solution:

  1. Check the output directory path in your .oxoflow file — paths are relative to the workflow file location
  2. Use oxo-flow debug workflow.oxoflow to see the actual commands being run
  3. Verify the shell command actually creates the output file: 
    # Test the command manually
mkdir -p data && echo 'Hello from oxo-flow!' > data/greeting.txt
ls data/greeting.txt  # Does the file exist?

"No workflow file found"#

Symptom: oxo-flow run without arguments fails with "no .oxoflow file found"

Solution:

  1. Ensure you're in a directory containing a .oxoflow file
  2. Or explicitly specify the workflow path: oxo-flow run path/to/workflow.oxoflow
  3. Use oxo-flow init my-pipeline to create a new project if none exists

"Validate fails with parse error"#

Symptom: oxo-flow validate workflow.oxoflow shows TOML syntax errors

Common TOML mistakes:

Mistake Wrong Correct
Missing quotes name = my-rule name = "my-rule"
Wrong array syntax [rules] [[rules]]
Unclosed string shell = "echo shell = "echo hello"
Invalid table header [[rule]] [[rules]]

Use the TOML primer for syntax basics.

Workflow Parsing Errors#

TOML syntax error#

Symptom: parse error in workflow.oxoflow: ...

Solution: Check your TOML syntax. Common mistakes:

  • Missing quotes around string values
  • Incorrect array-of-tables syntax (use [[rules]], not [rules])
  • Unmatched brackets or braces

Use oxo-flow validate workflow.oxoflow to get detailed error messages.

Duplicate rule names#

Symptom: duplicate rule name: 'step1'

Solution: Every rule must have a unique name field. If you're using [[include]] directives, use the namespace field to avoid conflicts:

[[include]]
path = "shared_rules.oxoflow"
namespace = "shared"

Execution Errors#

Rule fails with non-zero exit code#

Symptom: rule 'bwa_align' failed with exit code 1

Solution:

  1. Run oxo-flow debug workflow.oxoflow -r bwa_align to see the expanded command with all variables substituted.
  2. Check the log output for stderr messages.
  3. Try running the expanded command manually in your terminal.
  4. Verify that the required tool is installed and available in the rule's environment.

Command not found#

Symptom: sh: bwa: command not found

Solution: The tool is not in the system PATH. Either:

  • Specify an environment in the rule: 
    environment = { conda = "envs/alignment.yaml" }
  • Or ensure the tool is installed and accessible.

Timeout exceeded#

Symptom: command timed out after 3600s for rule 'variant_calling'

Solution: Increase the timeout via --timeout flag or allocate more resources (threads/memory) to the rule.

Wildcard Issues#

Unresolved wildcards#

Symptom: wildcard error in rule '...': unresolved wildcards: sample

Solution: Ensure wildcard values are provided. Wildcards like {sample} must be resolved from:

  • Input file patterns matched against existing files
  • Explicit values in the config section
  • Scatter configuration

Wildcard constraint violation#

Symptom: wildcard 'chr' value 'invalid' does not match constraint '^chr[0-9XYM]+$'

Solution: The wildcard value doesn't match the regex constraint defined in your workflow. Check that your input filenames follow the expected naming convention.

Environment Issues#

Conda environment creation fails#

Symptom: environment error (conda): ...

Solution:

  1. Check that conda/mamba is installed: conda --version
  2. Verify the environment YAML file exists and is valid
  3. Check for network connectivity (package downloads)
  4. Try creating the environment manually: conda env create -f envs/tool.yaml

Docker image not found#

Symptom: environment error (docker): ...

Solution:

  1. Check that Docker is installed and running: docker info
  2. Verify the image reference: docker pull biocontainers/bwa:0.7.17
  3. Check for authentication if using private registries

HPC modules not available#

Symptom: Module load errors when using modules in environment spec

Solution: Verify that the module system is available on your HPC node and that the specified module names and versions are correct:

module avail gcc
module avail cuda

DAG Issues#

Cycle detected#

Symptom: cycle detected in workflow DAG: A -> B -> A

Solution: Your rules have circular dependencies. Use oxo-flow graph to visualize the DAG and identify the cycle. Break the cycle by:

  • Removing unnecessary input/output connections
  • Splitting a rule into separate steps
  • Using depends_on for explicit ordering instead of file-based dependencies

Missing input#

Symptom: missing input for rule 'step2': intermediate.txt

Solution: Ensure that some other rule produces intermediate.txt as an output, or that the file already exists before the workflow runs.

Checkpoint and Resume#

Resuming a failed workflow#

After fixing the cause of a failure, re-run the same workflow. oxo-flow will check checkpoints and skip already-completed rules:

oxo-flow run workflow.oxoflow

Clearing checkpoint state#

To force a full re-run, delete the checkpoint file:

rm -rf .oxo-flow/checkpoint.json
oxo-flow run workflow.oxoflow

Performance Tips#

Workflow runs slowly#

  1. Increase parallelism: Use -j to run more jobs concurrently: 

    oxo-flow run workflow.oxoflow -j 8

  2. Check resource constraints: Use oxo-flow debug to verify that resource requirements are reasonable.

  3. Use streaming: For I/O-bound rules, set pipe = true to enable FIFO-based streaming where supported.

  4. Enable caching: Set cache_key on rules to enable content-based output reuse.

Memory issues with large workflows#

For workflows with many samples (>1,000):

  1. Process samples in batches using scatter/gather patterns
  2. Increase system memory limits
  3. Use cluster backends for distributed execution

Getting Help#

  • Run oxo-flow --help for CLI usage
  • Run oxo-flow <command> --help for subcommand details
  • Run oxo-flow debug workflow.oxoflow to inspect resolved commands
  • Check LIMITATIONS.md for known limitations
  • Open an issue for bugs or feature requests

Reporting Real-World Issues#

We particularly value feedback from real-world deployments. If you encounter issues in your actual bioinformatics workflows (as opposed to test examples), please use the [Real-World Testing] prefix in your issue title:

[Real-World Testing] SLURM GPU job scheduling fails on cluster with multiple partitions
[Real-World Testing] Conda environment detection issue with custom channels

Include details about your cluster type, oxo-flow version, and a description of what happened versus what you expected. See our Contributing Guide for more guidance on providing effective feedback.