Skip to content

oxo-flow Scientific Analysis#

oxo-flow is a Rust-native bioinformatics workflow engine: DAG execution, 8 environment backends, sample wildcards, checkpoint/resume, provenance. This skill is a compressed operating manual for agents — read it in full once per task; follow the links for details on demand.

Canonical entry points URL
Documentation https://traitome.github.io/oxo-flow/
This skill (site) https://traitome.github.io/oxo-flow/latest/SKILL/ (newest tagged release; /dev/SKILL/ tracks main)
This skill (raw) https://raw.githubusercontent.com/Traitome/oxo-flow/main/docs/guide/src/SKILL.md
Repository (dev context: AGENTS.md) https://github.com/Traitome/oxo-flow
Workflow JSON Schema https://raw.githubusercontent.com/Traitome/oxo-flow/main/docs/schema/oxoflow-v1.schema.json
Community template library https://oxo-flow-community.github.io/
Reference workflows https://github.com/Traitome/oxo-flow/tree/main/examples/gallery
VS Code extension https://traitome.github.io/oxo-flow/latest/reference/vscode-extension/ (.oxoflow editing: completion, diagnostics, one-click run/validate/graph; code --install-extension traitome.oxo-flow — published to Open VSX and the VS Code Marketplace; offline VSIX on every GitHub release)

GitHub unreachable? Prefix any GitHub URL with a mirror proxy (https://ghfast.top/<url> or https://gh-proxy.com/<url>), or web-search for a current proxy; ecosystem mirrors: see how-to/china-mirrors.

The Analysis Loop#

Seven stages, four approval gates (stages 0, 1, 2, and 6). Every stage ends with a Done when condition — do not proceed without it.

0. Data & Task Contract#

  • Restate the natural-language task as: scientific question, inputs, expected outputs, success criteria, compute budget.
  • Verify every input path on the filesystem (ls/stat); never invent or assume paths.
  • Done when: all inputs exist, formats identified, PI role approves.

1. Plan the Workflow Shape#

  • Prefer an existing base: the docs gallery (validated example workflows), the community library, examples/gallery. Write new only when no base matches.
  • Choose tools via the knowledge bases (see Link Index); containers must be fully qualified (quay.io/...) — there is no implicit registry fallback.
  • Decide sample dimensions (sample_pattern, [[pairs]], [[sample_groups]]), one of the 8 environment backends, and explicit depends_on between rules.
  • Done when: rule sketch + tool list + chosen base approved by PI.

2. Author the .oxoflow Project#

  • oxo-flow init scaffolds the project; write the workflow TOML (+ scripts); keep it in git — versioning is reproducibility.
  • Mandatory static gates, in order, before any compute:
  • oxo-flow validate --json <workflow.oxoflow> — structure, missing inputs, capacity
  • oxo-flow dry-run --json <workflow.oxoflow> — expansion preview, resource audit, -j suggestion
  • oxo-flow lint <workflow.oxoflow> — best practices
  • Fix and re-run until clean; oxo-flow graph to eyeball the DAG. Edge inference is best-effort but broad: exact strings, globs, and directory inputs all form edges when a producer's output matches; keep depends_on for ordering files can't express (side-effect-only steps).
  • Optional AI assistance: [ai] enabled = true in the workflow activates AI validate/lint/debug/recover.
  • Done when: all gates pass; engine-QC role sign-off.

3. Submit & Execute#

  • Local: oxo-flow run -j <N> --provenance (always --provenance).
  • Detached + scripted supervision: oxo-flow run … --detach then oxo-flow wait --workdir <dir> --timeout <secs> (exit 0 success, 1 failed/incomplete, 3 timeout, 4 no run) instead of a poll-for-files loop.
  • Cluster: oxo-flow run --profile <NAME> with a [cluster] block; manual escape hatch oxo-flow cluster submit|status|cancel|logs.
  • Web: POST /api/runs or the web UI.
  • Done when: run id + checkpoint path recorded.

4. Monitor & Correct#

  • oxo-flow status (or cluster status) to completion.
  • On failure: read structured diagnostics, diagnose, fix, re-run the static gates, then resume — oxo-flow resume <checkpoint> (or run --resume-failed) (checkpoint avoids recompute). Optional AI recovery: run --ai-recover --ai-max-retries 3. An applied fix is verified by one automatic rerun: the engine emits REPAIR_APPLIED then REPAIR_RERUN_SUCCEEDED/REPAIR_RERUN_FAILED/REPAIR_RERUN_SKIPPED contract lines and records repair_record in the checkpoint.
  • Retry budget: max 3 correction loops, then escalate to the human.
  • Done when: all rules complete, evidenced by checkpoint state.

5. QC Review (independent)#

  • Outputs exist, non-empty, plausible; oxo-flow provenance verify .oxo-flow/checkpoint.json passes; spot-check logs for silent errors.
  • Judge against the Stage 0 success criteria. Performed by an evaluator role — never the author of the workflow.
  • Fail → back to Stage 2 with written findings.

6. Reproducible Delivery (Receipt)#

  • oxo-flow report -f md -o REPORT.md --ci (execution truth: real exit codes, checksums, expanded commands) + -o report.json copy.
  • Receipt bundle: REPORT.md + checkpoint JSON + provenance-verify output + workflow git SHA + --versions-yml versions.yml (newer releases; skip if the flag is rejected) + input manifest.
  • Optional archival: oxo-flow export -f toml|docker|singularity.
  • Done when: PI signs off against the original task.

Data Safety & Workspace Discipline (BINDING)#

You are the actual operator of the user's data. Input data is read-only. The engine protects rule shells at run time (the shell-command guard blocks deletions outside the run workdir — lint code E011 flags the pattern early; [engine] protected_data_trees detects drift), but those gates do not cover your own terminal. These rules bind every command you run:

  1. Never delete or overwrite inputs without explicit human authorization. This includes rm, mv, truncate, dd of=, sed -i, git checkout --, git clean, and any > redirect into a path you did not create this task. Ask first: quote the exact command, name the target, say why — then wait for a human "yes". Destructive + no human reachable = stop.
  2. Snapshot before mutating. Copy what you are about to change into your scratch tree (cp -a) and record sha256sum. Write to NEW paths and rename on success; never edit in place. Append with >>; never > an existing file you did not create.
  3. Do all probing in one scratch tree inside the project — <project>/.oxo-flow/agent/<date>-<task>/ (gitignored), not $HOME, not a global /tmp scatter, not outside the working directory. Point tools there with OXO_FLOW_TMPDIR=<scratch>/tmp (or [engine] tmp_dir). Delete only your own scratch, and only when the task is done.
  4. Journal every mutating operation — one JSON line per command appended to .oxo-flow/agent/operations.log: {"ts":"<ISO8601>","cwd":"<dir>","cmd":"<argv>","exit":<code>,"note":"<why>"}. Read-only inspections need not be journaled.
  5. Protect input trees at plan time. For every workflow consuming vendor data, recommend [engine] protected_data_trees = ["raw/", "reference/"] — drift is then detected before the next run. Detection, not prevention: the deletion itself must still never happen.
  6. Inspect without mutating. Prefer oxo-flow status, oxo-flow report, oxo-flow provenance verify over ad-hoc shell. When unsure, dry-run.

Recipes (snapshots, journal helper, scratch layout, protected-tree config): how-to/agent-safety.

Guardrails#

  • Evidence first — never claim success without checkpoint/report evidence.
  • Real paths only — verify inputs exist before referencing them.
  • Static gates before compute — every workflow edit re-runs validate → dry-run → lint. An unbound placeholder ({sample}, {meta.<col>}, {config.<key>}) is a hard run-time error by default; lint W041 names the rule whose sample domain has no binding, and singleton = true declares a deliberate one-instance rule. Shell ${...} is never a placeholder.
  • Correct, don't restart — prefer checkpoint resume unless inputs were invalidated.
  • Explicit edges — edge inference covers exact strings, globs, and directory inputs; depends_on remains the escape hatch for ordering that file matching can't express.
  • Version everything — workflow in git; its SHA lands in checkpoint and report.
  • Budget & stop-loss — retry caps, human escalation, resource limits from dry-run.
  • Don't guess tools — search the knowledge bases; prefer live-tested gallery/community workflows.

Scientist Team Mode#

One skill covers 3–4 roles. Roles are stage owners in the loop above; the handoff artifact is each stage's output. Roles can be played sequentially by one agent or by parallel subagents; the independence rule is binding either way.

Role Stages Responsibility
PI (science lead) 0, 1 gate; 6 sign-off Task contract, scope veto, final acceptance
Computational biologist 1–4 Workflow design, tools, TOML, fixes
oxo-flow engine & QC expert 2 gates, 4, 5 Static gates, failure diagnosis, verification
Independent evaluator (optional 4th; a community expert doubles as tool-choice advisor in Stage 1) 5–6 Adversarial review vs. the scientific question

Minimal configuration (3 roles): PI doubles as evaluator — but must be a separate agent instance (independence rule: the author never verifies their own work).

Progressive disclosure — open only what the current stage needs.

Topic Links
Start tutorials/quickstart, tutorials/first-workflow, how-to/create-workflow
Execute how-to/run-on-cluster, commands/run, commands/cluster, commands/resume, commands/status
QC commands/validate, commands/dry-run, commands/lint, commands/debug, commands/graph, how-to/troubleshooting
Delivery how-to/generate-reports, commands/report, commands/provenance, commands/export, reference/versioning, reference/reporting-system
Reference reference/architecture, gallery/, tutorials/custom-scripts, tutorials/environment-management
Data safety (read early) how-to/agent-safety, reference/security, reference/workflow-format ([engine] protected_data_trees, tmp_dir)
Knowledge bases (raw) crates/oxo-flow-ai/src/knowledge/bioconda_tools.jsonl, skills_index.jsonl, nfcore_modules.jsonl, pipeline_graph.jsonl

Docs links resolve under https://traitome.github.io/oxo-flow/latest/<path> (mike serves each version under /latest/, /dev/, or /<tag>/); raw file links under https://raw.githubusercontent.com/Traitome/oxo-flow/main/<path>.