Skip to content

Environment System#

The environment system manages software environment resolution, activation, and deactivation for each rule in a workflow.

Overview#

Each rule can declare an environment specification. Before a rule's shell command runs, oxo-flow:

  1. Resolves the environment spec to a concrete backend
  2. Ensures the environment is ready (created, pulled, etc.)
  3. Activates the environment
  4. Runs the shell command
  5. Deactivates the environment

Architecture#

graph TD
    Rule["Rule (environment spec)"] --> Resolver["EnvironmentResolver"]
    Resolver --> Conda["CondaBackend"]
    Resolver --> Pixi["PixiBackend"]
    Resolver --> Docker["DockerBackend"]
    Resolver --> Singularity["SingularityBackend"]
    Resolver --> Venv["VenvBackend"]
    Resolver --> Cache["EnvironmentCache"]
    Cache --> File["Cache File (JSON)"]

EnvironmentResolver#

The central coordinator that:

  • Detects available backends on the system
  • Validates environment specifications
  • Dispatches to the appropriate backend
  • Tracks environment setup state via EnvironmentCache
let resolver = EnvironmentResolver::new();
let available = resolver.available_backends(); // ["conda", "docker", "venv"]
resolver.validate_spec(&rule.environment)?;

EnvironmentCache#

Tracks which environments have been successfully set up:

  • In-memory cache: Tracks ready environments during execution
  • Persistent cache: Optionally saves state to a JSON file for reuse across runs
// Create resolver with persistent cache
let resolver = EnvironmentResolver::with_cache_dir(Path::new(".oxo-flow/cache"));

When using --cache-dir, oxo-flow saves environment setup state after each run. Subsequent runs skip setup for already-ready environments, reducing startup time.

Environment Setup Process#

Before executing a rule with an environment specification:

  1. Check cache: If the environment is already marked as ready, skip setup
  2. Run setup command: Execute the backend's setup command (e.g., conda env create -f env.yaml)
  3. Mark ready: Cache the environment as successfully set up

Setup Commands by Backend#

Backend Setup Command
Conda conda env create -f <yaml_file>
Pixi pixi install (if pixi.toml exists)
Docker docker pull <image>
Singularity singularity pull <image>
Venv python -m venv <path> && pip install -r <requirements>

Skipping Setup#

Use --skip-env-setup when environments are pre-built:

# Environments already exist on the system
oxo-flow run pipeline.oxoflow --skip-env-setup

Backend Implementations#

Conda#

  • Detection: Checks for conda on $PATH
  • Resolution: Parses YAML environment file
  • Activation: Runs conda activate <env_name> before the command
  • Caching: Environments are created once and reused across rules that share the same YAML file

Pixi#

  • Detection: Checks for pixi on $PATH
  • Resolution: Parses pixi.toml project file
  • Activation: Uses pixi run to execute within the environment
  • Lockfile: Pixi's native lockfile ensures reproducible resolution

Docker#

  • Detection: Checks for docker on $PATH and daemon availability
  • Resolution: Parses image reference (registry/image:tag)
  • Execution: Wraps shell command in docker run --rm -v $(pwd):$(pwd) -w $(pwd) <image> <cmd>
  • Pull policy: Images are pulled on first use if not locally available

Singularity / Apptainer#

  • Detection: Checks for singularity or apptainer on $PATH
  • Resolution: Parses image reference (can be docker://, .sif file, or library URI)
  • Execution: Wraps shell command in singularity exec <image> <cmd>
  • Binding: Working directory is automatically bound into the container

Python venv#

  • Detection: Checks for python3 and pip on $PATH
  • Resolution: Parses requirements.txt file
  • Activation: Creates a venv (if needed) and activates it before the command
  • Caching: Venvs are stored in a cache directory keyed by the requirements hash

Environment Specification#

The EnvironmentSpec struct supports one backend per rule:

pub struct EnvironmentSpec {
    pub conda: Option<String>,
    pub pixi: Option<String>,
    pub docker: Option<String>,
    pub singularity: Option<String>,
    pub venv: Option<String>,
    pub modules: Vec<String>,
}

In TOML:

# Only one backend per rule (plus optional HPC modules)
environment = { conda = "envs/tools.yaml" }
environment = { docker = "biocontainers/bwa:0.7.17" }
environment = { venv = "envs/requirements.txt" }
environment = { modules = ["gcc/11.2", "cuda/11.7"] }

If multiple backends are specified, the first one found is used in this priority order: docker, singularity, conda, pixi, venv. HPC modules are loaded regardless of the primary backend if the executor supports them.

Default Environments#

Set a default in [defaults]:

[defaults]
environment = { conda = "envs/base.yaml" }

Rules without an explicit environment field inherit the default. Rules with an explicit environment override the default completely.

Validation#

# Check that all backends are available
oxo-flow env list

# Validate all environments in a workflow
oxo-flow env check pipeline.oxoflow

The env check command verifies:

  1. The backend type is available on the system
  2. The specification file exists (for conda YAML, pixi TOML, requirements.txt)
  3. The image reference is syntactically valid (for Docker/Singularity)

See Also#