Environment System#

The environment system manages software environment resolution, activation, and deactivation for each rule in a workflow.

Overview#

Each rule can declare an environment specification. Before a rule's shell command runs, oxo-flow:

Resolves the environment spec to a concrete backend
Ensures the environment is ready (created, pulled, etc.)
Activates the environment
Runs the shell command
Deactivates the environment

Architecture#

graph TD
    Rule["Rule (environment spec)"] --> Resolver["EnvironmentResolver"]
    Resolver --> Conda["CondaBackend"]
    Resolver --> Mamba["MambaBackend"]
    Resolver --> Pixi["PixiBackend"]
    Resolver --> Docker["DockerBackend"]
    Resolver --> Singularity["SingularityBackend"]
    Resolver --> Venv["VenvBackend"]
    Resolver --> Modules["ModulesBackend"]
    Resolver --> System["SystemBackend"]
    Resolver --> Cache["EnvironmentCache"]
    Cache --> File["Cache File (JSON)"]

EnvironmentResolver#

The central coordinator that:

Detects available backends on the system
Validates environment specifications
Dispatches to the appropriate backend
Tracks environment setup state via EnvironmentCache

let resolver = EnvironmentResolver::new();
let available = resolver.available_backends(); // e.g. ["system", "mamba", "conda", "pixi", "docker", "singularity", "venv", "modules"] — installed backends only
resolver.validate_spec(&rule.environment)?;

EnvironmentCache#

Tracks which environments have been successfully set up:

In-memory cache: Tracks ready environments during execution
Persistent cache: Optionally saves state to a JSON file for reuse across runs

// Create resolver with persistent cache
let resolver = EnvironmentResolver::with_cache_dir(Path::new(".oxo-flow/cache"));

When using --cache-dir, oxo-flow saves environment setup state after each run. Subsequent runs skip setup for already-ready environments, reducing startup time.

Environment Setup Process#

Before executing a rule with an environment specification:

Check cache: If the environment is already marked as ready, skip setup
Run setup command: Execute the backend's setup command (e.g., conda env create -f env.yaml)
Mark ready: Cache the environment as successfully set up

Setup Commands by Backend#

Backend	Setup Command
Conda	`conda env create -f <yaml_file>`
Mamba	`mamba env create -f <yaml_file>` (auto-detects mamba, micromamba, or conda)
Pixi	`pixi install` (if pixi.toml exists)
Docker	`docker pull <image>`
Singularity	`singularity pull <image>`
Venv	`python3 -m venv <path> && source <path>/bin/activate && pip install -r <requirements>`
Modules	None (no setup — modules are loaded at execution time)
System	None (no setup needed — commands run directly in the current shell)

Skipping Setup#

Use --skip-env-setup when environments are pre-built:

# Environments already exist on the system
oxo-flow run pipeline.oxoflow --skip-env-setup

Backend Implementations#

Conda#

Detection: Checks for conda on $PATH
Resolution: Parses YAML environment file
Activation: Runs conda run -n <env_name> bash -c '<command>'
Caching: Environments are created once and reused across rules that share the same YAML file

Mamba#

Detection: Checks for mamba, then micromamba, then falls back to conda on $PATH
Resolution: Parses YAML environment file (same format as conda)
Activation: Runs mamba run -n <env_name> bash -c '<command>'
Caching: Environments are created once and reused across rules that share the same YAML file
Usage: Set environment.mamba = "envs/qc.yaml" in the rule. Uses the same YAML format as conda but with the mamba/micromamba binary for faster solving.

Pixi#

Detection: Checks for pixi on $PATH
Resolution: Parses pixi.toml project file
Activation: Uses pixi run to execute within the environment
Lockfile: Pixi's native lockfile ensures reproducible resolution

Docker#

Detection: Checks for docker on $PATH and daemon availability
Resolution: Parses image reference (registry/image:tag)
Execution: Wraps shell command in docker run --rm -v $(pwd):$(pwd) -w $(pwd) <image> <cmd>
Pull policy: Images are pulled on first use if not locally available

Singularity / Apptainer#

Detection: Checks for singularity or apptainer on $PATH
Resolution: Parses image reference (can be docker://, .sif file, or library URI)
Execution: Wraps shell command in singularity exec <image> <cmd>
Binding: Working directory is automatically bound into the container

Python venv#

Detection: Checks for python3 on $PATH
Resolution: Parses requirements.txt file
Activation: Creates a venv (if needed) and activates it before the command
Caching: Venvs are stored in a cache directory keyed by the requirements hash

HPC Modules#

Detection: Checks for modulecmd or module on $PATH
Resolution: Parses the module list (comma-separated) from environment.modules
Activation: Initializes the module system (sources /etc/profile.d/modules.sh or common Lmod/Modules init scripts), then runs module load <modules> before the command. Fails with a clear error if the module command is unavailable.
Usage: Set environment.modules = ["gcc/11.2", "cuda/11.7"] in the rule. Modules are used when environment.modules is set and no other backend is declared (priority: mamba, conda, pixi, docker, singularity, venv, modules).

System#

Detection: Always available
Resolution: No environment spec required
Activation: No-op — the command runs directly in the current shell environment
Usage: This is the default backend for rules without an environment field

Environment Specification#

The EnvironmentSpec struct supports one backend per rule:

pub struct EnvironmentSpec {
    pub conda: Option<String>,
    pub mamba: Option<String>,
    pub pixi: Option<String>,
    pub docker: Option<String>,
    pub singularity: Option<String>,
    pub venv: Option<String>,
    pub modules: Vec<String>,
    pub conda_prefix: Option<String>,
    pub mamba_prefix: Option<String>,
    pub venv_requirements: Option<String>,
}

In TOML:

# Only one backend per rule — uncomment the one you need:
environment = { conda = "envs/tools.yaml" }
# environment = { docker = "biocontainers/bwa:0.7.17" }
# environment = { venv = "envs/requirements.txt" }
# environment = { modules = ["gcc/11.2", "cuda/11.7"] }

If multiple backends are specified, the first one found is used in this priority order: mamba, conda, pixi, docker, singularity, venv, modules. The system backend is used when a rule declares no environment spec.

Default Environments#

Set a default in [defaults]:

[defaults]
environment = { conda = "envs/base.yaml" }

Rules without an explicit environment field inherit the default. Rules with an explicit environment override the default completely.

Validation#

# Check that all backends are available
oxo-flow env list

# Validate all environments in a workflow
oxo-flow env check pipeline.oxoflow

The env check command verifies:

The backend type is available on the system
The specification file exists (for conda YAML, pixi TOML, requirements.txt)
The image reference is syntactically valid (for Docker/Singularity)