oxo-flow publish#

Bundle a workflow with its environment files into a verifiable, self-contained archive for sharing, archival, or remote execution.

Usage#

oxo-flow publish [OPTIONS] <WORKFLOW>

Description#

Reads the .oxoflow workflow file, recursively follows [[include]] references to discover all environment spec files, collects scripts/ and bin/ directories, and produces a single .tar.zst archive containing:

The workflow file
Every included sub-workflow .oxoflow file (at its including-file-relative path, so consumer-side include resolution resolves unchanged), together with each sub-workflow's own environment spec files, [workflow] data files, and container references — local path includes only; repo/URL includes are re-fetched by the consumer and are not vendored
All referenced environment files (conda, mamba, pixi, venv)
The [workflow] data files the workflow cannot parse without — metadata_file, pairs_file, and sample_groups_file (kept at their declared relative path inside the bundle)
scripts/ and bin/ directories (Nextflow-style auto-PATH convention)
manifest.json with per-file SHA-256 checksums and container image references

The archive is self-contained and checksum-verified — consumers can verify every file's integrity before execution.

With --with-lockfiles, also generates deterministic conda lockfiles for each environment YAML, ensuring exact reproducibility across time.

Options#

Option	Short	Description
`--output`	`-o`	Output path for the bundle archive (default: `<name>-bundle.<ext>`)
`--with-lockfiles`		Generate `conda-lock` lockfiles for reproducible environments
`--format`		Archive format: `tar.zst` (default, better compression) or `tar.gz` (universal compatibility)
`--verbose`	`-v`	Enable verbose (debug-level) logging
`--quiet`		Suppress non-essential output (errors only)
`--no-color`		Disable colored output

The global --json flag is not supported by this command — passing it fails fast instead of being silently ignored. Machine-readable output is available from: run, dry-run, validate, lint, test, status, batch, info, schema, license, ai, and provenance verify.

Examples#

# Publish a workflow (zstd compression, default)
oxo-flow publish my_pipeline.oxoflow
# → my_pipeline-bundle.tar.zst

# Publish with gzip compression (universal compatibility)
oxo-flow publish my_pipeline.oxoflow --format tar.gz
# → my_pipeline-bundle.tar.gz

# Publish with custom output path
oxo-flow publish my_pipeline.oxoflow -o /path/to/bundle.tar.zst

# Publish with deterministic lockfiles
oxo-flow publish my_pipeline.oxoflow --with-lockfiles

# Run a published bundle (extract → verify → confirm → execute)
oxo-flow run --bundle my_pipeline-bundle.tar.zst -j 16

# Run with --yes to skip confirmation (CI/scripts)
oxo-flow run --bundle my_pipeline-bundle.tar.zst -j 16 --yes

# Pull a remote bundle and run it
oxo-flow pull gh:user/repo@v0.23.2
oxo-flow run --bundle repo-bundle.tar.zst --yes

Manifest Format#

Bundle members keep the workflow's own relative paths (an environment declared as envs/fastp.yaml lands at envs/fastp.yaml inside the bundle, not at the archive root), so the bundled workflow's references resolve unchanged.

The manifest.json inside each bundle:

{
  "format": "oxoflow-bundle-v1",
  "workflow": "my_pipeline.oxoflow",
  "oxo_flow_version": "0.23.2",
  "created_at_epoch": 1234567890,
  "entrypoint": "my_pipeline.oxoflow",
  "files": [
    {
      "path": "my_pipeline.oxoflow",
      "sha256": "sha256:abcdef...",
      "size": 1024
    },
    {
      "path": "envs/fastp.yaml",
      "sha256": "sha256:123456...",
      "size": 256
    }
  ],
  "containers": [
    {
      "type": "docker",
      "image": "biocontainers/bwa:0.7.17"
    }
  ],
  "resources": {
    "rules": [
      {
        "rule": "bwa_align",
        "threads": 16
      }
    ],
    "recommendations": {
      "min_threads": 16
    }
  },
  "signatures": []
}

signatures is reserved for future bundle signing and is always empty today. It is present so that adding signatures later is an additive change rather than a manifest format bump.

Reproducibility Caveats#

A bundle captures the workflow, its environment specifications, and checksums for every file. That makes a bundle verifiable — you can prove you received exactly what was published. It does not make execution identical everywhere, and it is worth being explicit about the limits:

Environment specs are resolved on the consumer's machine. publish bundles environment.yaml / pixi.toml spec files, not solved environments. A solve run months later, or against different channels, can pick different package versions. Use --with-lockfiles to pin the resolution.
Lockfiles still are not binaries. Even an exact package set can behave differently across glibc versions, CPU features, or filesystem layouts. Tools built against a newer glibc will not run on an older host.
Container images are referenced, not vendored. The manifest records image type and tag. The image is pulled at run time, so a mutable tag can resolve to different content later. Prefer digest-pinned references where it matters.
Containers do not normalise resources. An image that runs fine on the publisher's machine can be OOM-killed on a smaller host, and thread counts vary with the available CPUs.
Sample DATA is not bundled. The engine-owned data files the workflow declares (metadata_file, pairs_file, sample_groups_file) travel in the bundle; raw analysis inputs (FASTQs, BAMs, …) do not. For --bundle runs the workflow is executed from a temp extraction dir, so wildcard discovery (sample_pattern, pairs_pattern) scans the extraction first — but when a pattern matches nothing there, discovery retries against the run workdir (--workdir), so raw inputs can live beside the bundle and be pointed at with --workdir. Data placed inside the bundle still wins: the extraction scan is the primary anchor and the workdir is only a fallback.

None of these are specific to oxo-flow — they apply to Snakemake and Nextflow bundles equally. The goal is honest reproducibility, not a guarantee we cannot make.