Environment System#
The environment system manages software environment resolution, activation, and deactivation for each rule in a workflow.
Overview#
Each rule can declare an environment specification. Before a rule's shell command runs, oxo-flow:
- Resolves the environment spec to a concrete backend
- Ensures the environment is ready (created, pulled, etc.)
- Activates the environment
- Runs the shell command
- Deactivates the environment
Architecture#
graph TD
Rule["Rule (environment spec)"] --> Resolver["EnvironmentResolver"]
Resolver --> Conda["CondaBackend"]
Resolver --> Mamba["MambaBackend"]
Resolver --> Pixi["PixiBackend"]
Resolver --> Docker["DockerBackend"]
Resolver --> Singularity["SingularityBackend"]
Resolver --> Venv["VenvBackend"]
Resolver --> Modules["ModulesBackend"]
Resolver --> System["SystemBackend"]
Resolver --> Cache["EnvironmentCache"]
Cache --> File["Cache File (JSON)"]
EnvironmentResolver#
The central coordinator that:
- Detects available backends on the system
- Validates environment specifications
- Dispatches to the appropriate backend
- Tracks environment setup state via EnvironmentCache
let resolver = EnvironmentResolver::new();
let available = resolver.available_backends(); // e.g. ["system", "mamba", "conda", "pixi", "docker", "singularity", "venv", "modules"] — installed backends only
resolver.validate_spec(&rule.environment)?;
EnvironmentCache#
Tracks which environments have been successfully set up:
- In-memory cache: Tracks ready environments during execution
- Persistent cache: Optionally saves state to a JSON file for reuse across runs
// Create resolver with persistent cache
let resolver = EnvironmentResolver::with_cache_dir(Path::new(".oxo-flow/cache"));
When using --cache-dir, oxo-flow saves environment setup state after each run. Subsequent runs skip setup for already-ready environments, reducing startup time.
Environment Setup Process#
Before executing a rule with an environment specification:
- Check cache: If the environment is already marked as ready, skip setup
- Run setup command: Execute the backend's setup command (e.g.,
conda env create -f env.yaml) - Mark ready: Cache the environment as successfully set up
Setup Commands by Backend#
| Backend | Setup Command |
|---|---|
| Conda | conda env create -n <env> -f <yaml_file> (falls back to conda env update -n <env> -f <yaml_file> if the env already exists) |
| Mamba | <mamba\|micromamba\|conda> env create -n <env> -f <yaml_file> (falls back to env update; auto-detects mamba, micromamba, or conda) |
| Pixi | pixi install --manifest-path <pixi.toml> (manifest must exist — validated upfront) |
| Docker | docker image inspect <image> \|\| docker pull <image> — pull only when absent; bare names that 404 on Docker Hub get one quay.io/biocontainers retry re-tagged to the original name |
| Singularity | Pull when needed: a URI (docker://…) is pulled to a derived .sif name; a local .sif path is used as-is |
| Venv | python3 -m venv <path> && . <path>/bin/activate && pip install -r <requirements> — the pip install leg is skipped, with a warning naming the expected path, when the requirements file does not exist |
| Modules | None (no setup — modules are loaded at execution time) |
| System | None (no setup needed — commands run directly in the current shell) |
Setup runs with the operator's environment (accepted risk, issue #1174)#
The setup commands above — and their teardown, retry, and verification
forms — run under the full ambient environment of the oxo-flow
process. This is deliberately different from rule execution: rule shells,
pre_exec hooks, and rule-/workflow-level hooks run under a cleared
environment plus a fixed allowlist (issue #1108, RB-15), so they cannot
read ambient secrets from their own process environment. Environment setup
keeps proxy, mirror, and certificate configuration (HTTP_PROXY,
HTTPS_PROXY, NO_PROXY, PIP_INDEX_URL, PIP_TRUSTED_HOST,
SSL_CERT_FILE, CONDA_*, corporate mirrors), which conda, pip, and pixi
need in order to provision at all.
This is an explicit accepted risk, not an isolation guarantee. Setup inputs
are workflow-authored executable content: a pinned sdist's build backend
(setup.py, pyproject.toml) runs during pip install, and conda
installs run package post-link scripts. That code runs as the operator with
ordinary filesystem access, so it can read ~/.aws/credentials,
~/.docker/config.json, or $HOME/.oxo-flow/ai_config.json whether or not
its environment was cleared — clearing the environment would be cosmetic.
The real boundary is the workflow-trust decision: do not run workflows
you do not trust on a machine whose operator credentials you would not
hand to the workflow's author. The same caveat is restated in
the workflow format reference
and the run reference.
Skipping Setup#
Use --skip-env-setup when environments are pre-built:
With the flag set, the engine does not create anything — a rule whose env
is missing fails inside conda. For file-backed conda specs it therefore
checks conda env list up front and names the expected <name>-<hash8>
env before running: if an env under the plain <name> exists, it was
likely built from the same spec before the content-hash suffix — symlink
or rename it, build the suffixed env, or drop --skip-env-setup.
Backend Implementations#
Conda#
- Detection: Checks for
condaon$PATH - Resolution: Parses YAML environment file
- Naming: A file-backed spec resolves to the env name
<name>-<hash8>, where<name>is the YAML'sname:field (falling back to the file stem) and<hash8>is the first 4 bytes of the spec's SHA-256 as 8 hex chars — two workflows shipping different YAMLs under the same name then build into distinct envs instead of silently sharing one. Same content → same name, so identical specs deduplicate. Pre-create envs with exactly that name (conda env create -n <name>-<hash8> -f envs/spec.yaml); the engine prints the expected name when it detects the env is missing under--skip-env-setup - Activation: Runs
conda run --no-capture-output -n <env_name> bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; <command>'—--no-capture-output(conda ≥ 4.13) keeps stdout/stderr live, and thePATHprefix makes the rule see the env's own tools first - Caching: Environments are created once and reused across rules that share the same YAML file
- Empty dependency sets: a file-backed YAML whose
dependencies:list is empty (or absent) is rejected at validation —env checkand the run's backend preflight name the offending file before any solve starts.conda env createotherwise exits 0 after a full solve and leaves a prefix with nobin/, which then fails verification and reads like an interrupted creation. Add at least one package or remove the environment declaration. Mamba specs use the same format and the same check
Mamba#
- Detection: Checks for
mamba, thenmicromamba, then falls back tocondaon$PATH - Resolution: Parses YAML environment file (same format as conda)
- Activation: Runs
<mamba|micromamba|conda> run -n <env_name> bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; <command>'— thePATHprefix makes the rule see the env's own tools first on hybrid boxes (same rationale as conda). When the detected binary isconda(fallback),--no-capture-outputis added like the conda backend; native mamba/micromamba reject that flag, so it is only added for conda - Caching: Environments are created once and reused across rules that share the same YAML file
- Usage: Set
environment.mamba = "envs/qc.yaml"in the rule. Uses the same YAML format as conda but with the mamba/micromamba binary for faster solving.
Pixi#
- Detection: Checks for
pixion$PATH - Resolution: Parses the
pixi.tomlmanifest the rule names - Activation: Runs
pixi run --manifest-path <pixi.toml> bash -c '<command>'— the whole rule command travels inside the env.pixi runis a child-process environment (like docker), not a PATH-mutating one (like modules/venv): with a bare prefix wrap, shell operators (&&,|) and the executor's prependedmkdir -pline escape the env and run on the host PATH (live-caught on a SLURM cluster). The spec is a manifest path, not an environment name (-ewould only search the current directory for a discoverable manifest) - Lockfile: Pixi's native lockfile ensures reproducible resolution, and is part of the environment's identity: the pixi cache is keyed by the manifest spec plus content hashes of the manifest and its sibling
pixi.lock(pixi:<path>:<hash8>:<hash8>), so an in-place edit of either — e.g. committing a reproducibility pin to the lockfile — invalidates the cache and the next run rebuilds the environment. Unreadable or missing files degrade to empty tags. Rule fingerprints carry the same digests, so completed rules referencing the spec are invalidated too (see run — config changes and precise invalidation) - Site quirks (live-verified): hand-written manifests must declare
platforms(e.g.platforms = ["linux-64"]) — modern pixi refuses to solve a workspace without them; andpiximust be on$PATHinside batch jobs, which start with a clean environment
Docker#
- Detection: Checks for
dockeron$PATH(the daemon itself is only contacted when a rule runs) - Resolution: Parses image reference (registry/image:tag)
- Execution: Wraps the command in
docker run --rm --user $(id -u):$(id -g) -v <workdir>:<workdir> -w <workdir> <image> sh -c '<bash shim>' sh '<command>'; absolute host paths referenced by the rule but living outside the workdir are added as extra read-only binds (-v /data/ref:/data/ref:ro) - Pull policy: Images are pulled on first use if not locally available
Singularity / Apptainer#
- Detection: Prefers
apptaineroversingularity, whichever is found first on$PATH - Resolution: Parses image reference (can be
docker://,.siffile, or library URI) - Execution: Wraps the command in
<apptainer|singularity> exec --bind <workdir>:<workdir> <image> sh -c '<bash shim>' sh '<command>' - Binding: The working directory is bound into the container with an absolute path (
--bindrejects relative sources), plus read-only binds for host paths the rule references
The container <bash shim>
Both container backends hand the rule's command to the image's shell as
sh -c 'if command -v bash >/dev/null 2>&1; then exec bash -c "$1"; else
exec sh -c "$1"; fi' sh '<command>'. The image's entrypoint shell runs
first, and it is often a minimal sh that cannot execute multi-line or
pipefail-using scripts — the shim re-execs bash whenever the image
ships it, and falls back to sh when it does not.
Python venv#
- Detection: Checks for
python3on$PATH(withensurepip/venvimportable) - Resolution:
environment.venvnames the venv directory to create/reuse (e.g..venv);environment.venv_requirementsnames the requirements file (default:requirements.txt, resolved against the workflow file's directory like every other file-backed environment spec). Declaring a requirements file path invenvitself fails — the backend creates a directory at that path - Activation: Creates the venv (if needed) and activates it before the command. If the requirements file does not exist, setup still creates/reuses the venv but skips the
pip installleg and logs a warning naming the expected path — declaringvenvon an existing environment never fails inside pip with "Could not open requirements file". An explicitvenv_requirementspointing at a missing file behaves identically - Caching: Venvs are stored in a cache directory keyed by the declared
venv spec plus a content hash of the requirements file
(
venv:<path>:<hash8>), so editingrequirements.txtin place invalidates the cache and the next run rebuilds the environment. A missing or unreadable requirements file (including the defaultrequirements.txtwhen it does not exist) degrades to the plainvenv:<path>key; the file appearing later changes the key, so the next run performs the skipped install.
HPC Modules#
- Detection: Checks for
modulecmdormoduleon$PATH - Resolution: Parses the module list (comma-separated) from
environment.modules - Activation: Initializes the module system (sources
/etc/profile.d/modules.shor common Lmod/Modules init scripts), then runsmodule load <modules>before the command. Fails with a clear error if themodulecommand is unavailable. - Usage: Set
environment.modules = ["gcc/11.2", "cuda/11.7"]in the rule. Modules are used whenenvironment.modulesis set and no other backend is declared (priority: mamba, conda, pixi, docker, singularity, venv, modules). - Site quirks (live-verified): the wrap mutates
$PATHinside the rule's shell, so a same-named binary from a user-level tool manager (e.g.~/.pixi/bin,~/.local/bin, a conda base) shadows the module-provided one — probe withcommand -v <tool>when diagnosing. The init fallback chain covers Environment Modules under both/usr/share/modulesand/usr/share/Modules.
System#
- Detection: Always available
- Resolution: No environment spec required
- Activation: No-op — the command runs directly in the current shell environment
- Usage: This is the default backend for rules without an
environmentfield
Resolver Order#
A rule resolves at most one backend, checked in this order:
mamba → conda → pixi → docker → singularity → venv → modules.
Declaring modules alongside any other backend is a hard error rather than a
silent drop — a container has no module system, so the module loads could
only ever be lost.
Environment Specification#
The EnvironmentSpec struct supports one backend per rule:
pub struct EnvironmentSpec {
pub conda: Option<String>,
pub mamba: Option<String>,
pub pixi: Option<String>,
pub docker: Option<String>,
pub singularity: Option<String>,
pub venv: Option<String>,
pub modules: Vec<String>,
pub conda_prefix: Option<String>,
pub mamba_prefix: Option<String>,
pub venv_requirements: Option<String>,
/// GPU passthrough for docker — `--gpus <value>` (e.g. `"all"`);
/// rejected at validate time unless a docker image is set.
pub gpus: Option<String>,
}
In TOML:
# Only one backend per rule — uncomment the one you need:
environment = { conda = "envs/tools.yaml" }
# environment = { docker = "biocontainers/bwa:0.7.17" }
# environment = { venv = ".venv/", venv_requirements = "envs/dev-requirements.txt" }
# environment = { modules = ["gcc/11.2", "cuda/11.7"] }
If multiple backends are specified, the first one found is used in this priority order: mamba, conda, pixi, docker, singularity, venv, modules. The system backend is used when a rule declares no environment spec.
Default Environments#
Set a default in [defaults]:
Rules without an explicit environment field inherit the default. Rules with an explicit environment override the default completely.
Validation#
# Check that all backends are available
oxo-flow env list
# Validate all environments in a workflow
oxo-flow env check pipeline.oxoflow
The env check command calls the resolver's validate_spec for each rule's
environment (relative file specs — conda/mamba/pixi/
venv_requirements — resolve against the workflow directory first),
which verifies backend availability on the current system
(e.g. the conda/pixi/docker/singularity binary exists and is runnable;
a pixi spec must point at a pixi.toml manifest). It does not
re-check spec files or image reference syntax — validate and lint cover
declaration-level checks.
Shared environment store (opt-in)#
By default every workflow is self-contained: pixi and venv environments
materialize inside the workflow tree (.pixi next to the manifest, the venv
dir) like any other artifact. Setting OXO_FLOW_ENV_STORE to a non-empty
path activates the shared store (location IS activation): the env then
materializes once per content key under the store root and the in-tree
location becomes a symlink, so pixi run --manifest-path and
source .venv/bin/activate keep working unchanged. The variable is
per-process — on a multi-user server each user who opts in gets their own
store unless an operator deliberately points everyone at a shared path.
Maintenance is oxo-flow env gc (checkpoint-reference + age gated,
--dry-run first); a broken link is detected on the next run, the dangling
in-tree symlink is detached, and the env is rebuilt automatically. A
content-key change (edited requirements.txt or pixi.toml) gets a new
store entry: the next run detaches the stale in-tree
symlink, builds the new environment in-tree, and re-homes it under the new
key. The previous store dir is left untouched for other workflows and for
env gc. The store deliberately covers pixi and venv
only: conda/mamba name-based environments already live in a shared central
prefix, conda prefix environments are not relocatable (absolute paths baked
into the env), containers pull images into their own registries, and HPC
modules are provided by the cluster — none of them benefit from a
content-keyed store. Details: env command.
Shared conda-family lifecycle (create → use → decommission)#
conda and mamba are one lifecycle implementation, not two copies: every
command the RULE lifecycle runs for them is generated by the single
conda_family_* builder family in crates/oxo-flow-core/src/environment.rs
(conda_family_target, conda_family_setup, conda_family_verify,
conda_family_wrap, conda_family_teardown, conda_family_cache_key).
Two deliberate exceptions exist: the standalone oxo-flow env create bootstrap
(commands/infra.rs) creates a user-named environment outside the rule
lifecycle (its name is the documented user contract), and the generated
container Dockerfiles emit their own conda env create line
(container.rs) while sharing the name derivation.
CondaBackend and MambaBackend contribute only the binary token (plus the
documented conda-only --no-capture-output on the wrap);
EnvironmentResolver is the single dispatch point the executor calls for
every backend.
The lifecycle the engine drives for every backend is:
- create —
EnvironmentResolver::setup_command: conda/mamba share thecreate || updatebuilder, pixi ispixi install --manifest-path; the executor runs the command throughsh -cin the workdir. - verify — conda/mamba only: the shared
run … test -d "$CONDA_PREFIX/bin"probe. A failure tears the environment down through the shared teardown builder and retries the same create command once. - use —
EnvironmentResolver::wrap_commandwraps the rule: the sharedconda run … bash -c 'export PATH="$CONDA_PREFIX/bin:$PATH"; …'orpixi run --manifest-path … bash -c …. - decommission — conda/mamba remove the environment with the shared
env remove -n <name> -yteardown; pixi (and venv) are store-managed, so their end of life isoxo-flow env gc(checkpoint-reference + age gated; see Shared environment store).
The store deliberately does not manage conda/mamba, so env gc is not their
decommission path — their lifecycle ends in the shared teardown builder.
batch -e "conda: <spec>" reuses the same wrap builder, so its items run in
the SAME environment an oxo-flow run rule would create — including the
<yaml name|stem>-<hash8> name for a file-backed spec.
Evidence: cross-backend lifecycle test#
crates/oxo-flow-cli/tests/conda_family_lifecycle.rs spawns the real CLI
binary against fake conda/mamba/pixi executables on a temp PATH — no
network and no real toolchain. Each fake records its argv and materializes
the prefix/marker the resolver checks, closing the loop
generate → execute → verify → use through production code. Tests:
conda_lifecycle_create_use_teardown_and_reuse_share_one_pathmamba_lifecycle_create_use_teardown_and_reuse_share_one_pathconda_and_mamba_execute_one_shared_command_shape— the create and verify argv of both backends are identical modulo the binary token, and the wrap argv is identical modulo the conda-only--no-capture-output.pixi_lifecycle_create_use_store_rehome_and_env_gc— create → use → store re-home →env gc(kept while a checkpoint references the entry, deleted once unreferenced and older than--min-age-days) → documented broken-link rebuild.batch_conda_environment_wraps_with_the_engine_created_env— regression test for thebatch -e "conda: …"divergence fixed in issue #1155.
The in-process builder-equality test
environment::tests::conda_and_mamba_share_one_lifecycle_command_builder
(cargo test -p oxo-flow-core) pins the same property at the command-string
level and that the resolver emits the backend's commands unchanged.
What the evidence proves: command construction is one shared path (modulo the
documented per-binary flag), and create → use → teardown/gc executes end to
end through the production resolver, executor, env store, and env gc. What
it does not prove: real conda/pixi solver or network behavior,
real-toolchain CLI compatibility beyond the emulated contract,
container/module backends, cluster execution paths, or env gc for
conda/mamba (they are not store-managed).
See Also#
- Environment Management tutorial — getting started
- Use Environments how-to — practical recipes
envcommand — CLI referenceruncommand —--skip-env-setupand--cache-diroptions- Shared env store (opt-in) —
OXO_FLOW_ENV_STOREsemantics and GC - China Network Mirrors — measured reachability of the conda/bioconda, PyPI, Rust, and Docker mirrors users on mainland-China networks configure for environment provisioning (includes a re-runnable probe script).