oxo-flow provenance#

Verify output file integrity using stored checksums.

Usage#

oxo-flow provenance verify [OPTIONS] <CHECKPOINT_PATH>

Description#

Re-computes SHA-256 checksums of output files and compares them against previously recorded values stored in a checkpoint (generated by oxo-flow run --provenance). Reports any mismatches, missing files, or unverified outputs.

The header also surfaces the workflow provenance recorded at run start — the git HEAD SHA of the repository the workflow lives in and the workflow file path — so every verification names the exact workflow version that produced the results (see Workflow Versioning).

This is essential for reproducible, auditable analysis.

Options#

Option	Description
`-v, --verbose`	Enable verbose (debug-level) logging
`--quiet`	Suppress non-essential output (errors only)
`--no-color`	Disable colored output
`--json`	Output machine-readable JSON to stdout

Examples#

# Generate a provenance-tracked run
oxo-flow run workflow.oxoflow --provenance

# Verify output integrity
oxo-flow provenance verify .oxo-flow/checkpoint.json

Output#

Provenance Verify .oxo-flow/checkpoint.json

  • workflow git HEAD: 3f2a1c9d8b7e6f5a4c3d2e1f0a9b8c7d6e5f4a3b
  • workflow path: /home/user/proj/pipeline.oxoflow

  ✓ output/sample1.vcf sha256:abc123...
  ✓ output/sample2.vcf sha256:def456...
  ✗ output/report.html (expected: sha256:xxx, actual: sha256:yyy)
  ✓ .oxo-flow/chunks/chr/chr1.g.vcf.gz sha256:ccc... (cleaned by design)

Summary: 2 matched, 1 mismatched, 0 missing, 1 cleaned (by design)

Notes#

Runs inside a git repository record the workflow's HEAD commit SHA in the checkpoint; verify prints it next to the workflow path so each result set is auditable to its workflow version. Runs outside a git repository omit the line — the field is simply absent.
Checksums are stored only when --provenance is passed to oxo-flow run.
File checksums use streaming I/O (64KB buffer) to handle large files (BAM, FASTQ >100GB) without excessive memory usage.
Mismatches and missing files exit with code 1 (useful for CI/CD pipelines).
Output paths are resolved against the checkpoint's recorded workdir (the directory the run executed in), not the checkpoint's own directory — so the checkpoint can live anywhere (e.g. .oxo-flow/ on the run host while outputs are read from the recorded run directory). For legacy checkpoints without a workdir field, the checkpoint's parent directory is used.
Transform chunk intermediates deleted by transform.cleanup = true at the end of a successful run are recorded under cleaned_checksums and reported as cleaned (by design) — they are excluded from the missing/mismatched counts and never fail verification. The cleaned count appears in the summary only when the checkpoint has any. Checkpoints written by releases ≤ 0.17.1 record chunks under checksums and still report them missing after cleanup — the distinction exists only in checkpoints written by newer binaries.

Verify without stored checksums#

When the checkpoint contains no checksums (the run was executed without --provenance), verify degrades to a status summary: it prints a note explaining that checksum tracking was not enabled and lists the completed rules from the checkpoint, then exits with code 0. This keeps provenance verify safe to run on any checkpoint while signaling clearly that integrity was not verified.

Provenance Verify .oxo-flow/checkpoint.json

  Note: No stored checksums found. Run workflow with --provenance to enable tracking.
  Found completed rules: 3
  ✓ align
  ✓ sort_bam
  ✓ trim_reads

  Hint: To verify integrity, provide a checksums file.

To get real integrity verification, re-run with --provenance.

JSON output#

With --json, the verification document is written to stdout (stdout carries nothing else):

{
  "command": "provenance",
  "verify": {
    "checkpoint": "/path/to/.oxo-flow/checkpoint.json",
    "matched": 2,
    "mismatched": 1,
    "missing": 0,
    "cleaned": 1,
    "entries": [
      {"file": "output/sample1.vcf", "status": "matched", "expected": "sha256:abc123...", "actual": "sha256:abc123..."},
      {"file": "output/report.html", "status": "mismatched", "expected": "sha256:xxx", "actual": "sha256:yyy"},
      {"file": ".oxo-flow/chunks/chr/chr1.g.vcf.gz", "status": "cleaned", "expected": "sha256:ccc..."}
    ]
  }
}

entries is byte-stable: stored files first in sorted path order, then cleaned files in sorted path order. When the checkpoint has no stored checksums, the document is still emitted with a note field describing the degradation.