Skip to content

Web API#

oxo-flow includes a built-in REST API server for building, validating, running, and monitoring bioinformatics workflows. The server is built with axum and follows a domain-driven modular monolith architecture.


API Design Conventions#

  • Envelope: success responses are bare JSON objects/arrays; errors are { code, message, detail, suggestion } — detail and suggestion are always present and are null when the error carries no extra context
  • Errors: { code: "E001", message, detail, suggestion } — the code value uses the engine's diagnostic codes
  • Lists: GET /api/runs returns a cursor-paginated envelope { items, next_cursor, total } (limit ≤ 500, status/q filters); other list endpoints return bare arrays (≤ 100 items)
  • Versioning: /api/ prefix for all endpoints
  • Authentication: in team/hpc mode, protected endpoints accept an Authorization: Bearer <token> session token or an X-API-Key header. The generated OpenAPI spec at GET /api/openapi.json declares both bearerAuth and apiKey security schemes; public endpoints (health, login, license, openapi.json, etc.) require no auth.
  • Self-discoverable: OpenAPI 3.1 spec at GET /api/openapi.json. The spec is code-generated via utoipa from the #[utoipa::path] annotations on every route handler — there is no hand-maintained static file. crates/oxo-flow-web/tests/openapi_gate.rs is the drift gate: it asserts every route in the router appears in the generated spec, so a new route without an annotation fails CI.

Structured Error Format#

All errors return a unified JSON format:

{
  "code": "AUTH_REQUIRED",
  "message": "Authentication is required for this endpoint",
  "detail": "The request did not include a valid session token or Bearer token",
  "suggestion": "Please login at POST /api/auth/login to obtain a session token"
}

detail and suggestion are always serialized; when an error has no extra context they are null rather than absent, so clients can parse the four-field shape unconditionally (the OpenAPI ApiError schema marks both nullable).

Rate limiting follows the same contract: over-limit requests get 429 {"code":"RATE_LIMITED", "message":"Rate limit exceeded", "detail":"retry in Ns", ...} with a Retry-After header (sliding window, 100 requests / 60 s per client IP by default).

POST /api/auth/login adds a dedicated brute-force guard: 10 failed attempts per 5 minutes per (account, source IP) trip a lockout that answers the same structured 429 RATE_LIMITED (the countdown is in detail; the login handler's error tuple cannot attach a Retry-After header) before any credential check runs. Per-account keying means a sprayed username cannot lock out other accounts; a successful login resets the counter; OXO_FLOW_DISABLE_RATE_LIMIT=1 disables both limiters (tests/dev only).

Malformed input is answered in the same envelope, never axum's bare-text rejection: a query string that does not fit the endpoint's parameters gets 400 INVALID_QUERY, and a JSON body that does not deserialize against the endpoint's schema gets 400 INVALID_BODY — both carry axum's parse message in detail and a pointer to GET /api/openapi.json in suggestion.


Starting the Server#

# Mode 1: Personal (default) — SQLite, no auth, localhost
oxo-flow serve

# Mode 2: Team — auth enabled, network-facing
oxo-flow serve --mode team

# Mode 3: HPC — cluster-aware
oxo-flow serve --mode hpc

# Or via the standalone binary:
oxo-flow-web --mode personal -p 3000

System & Monitoring#

Health Check#

GET /api/health
Returns status, version, mode, uptime, component health (database, filesystem, scheduler, AI provider), resource usage, and license info.

In hpc mode, components.scheduler reports {"status":"ok"} when a scheduler backend is detected (the same cached probe /api/hpc uses); it is null in other modes or when no backend is installed.

components.ai_key_storage is "encrypted" when OXO_FLOW_MASTER_KEY is set, "plaintext" otherwise.

components.ai_provider is {"status": "connected", "latency_ms": null} while a provider is active and {"status": "disabled", "latency_ms": null} when AI is off — explicitly pinned with OXO_FLOW_AI_PROVIDER=disabled, disabled at runtime, or nothing resolved. The pin outranks every persisted tier: ~/.oxo-flow/ai_config.json, the platform config file, and DB-persisted settings can supply credentials for any other provider value but cannot resurrect a disabled registry. In that state the AI endpoints answer the structured AI_NOT_CONFIGURED error without any outbound request. Configuration updates follow the same rule with one distinction: the runtime-only POST /api/ai/config answers 409 AI_DISABLED_BY_ENV (it has no durable tier, so refusing is the only honest answer), while the persisted PUT /api/ai/config/server and PUT /api/ai/config/user still save (200, applied_to_runtime: false) — Settings stays usable and the saved config applies after an unpinned restart — but never reconfigure the pinned runtime. POST /api/ai/test answers success: false with provider: "disabled" without constructing a provider or opening a connection (see the AI endpoints section below).

components.engine reports the engine CLI the server spawns for runs, resolved once at startup by running <binary> --version:

{
  "status": "ok",
  "components": {
    "engine": {
      "version": "0.31.0",
      "path": "/usr/local/bin/oxo-flow",
      "version_mismatch": false
    }
  }
}

version_mismatch is true when the engine's major.minor differs from the server's — runs will execute under the engine's semantics, and the server log carries a drift warning with a hint to pin OXO_FLOW_BIN. The field is null when the probe failed (binary missing or unparseable --version output); the run log header remains the version record in that case.

System Info#

GET /api/system
Returns OS, architecture, PID, uptime, and version. Team/hpc modes require authentication (the endpoint left the anonymous whitelist in the v0.11 hardening).

The response also carries engine — the same engine-CLI info as /api/health's components.engine.

Runtime Metrics#

GET /api/metrics
Returns real-time resource metrics: CPU%, memory (used/total/swap), active workflows, total requests, CPU count. Team/hpc modes require authentication.

Audit Logs#

GET /api/audit?days=7&page=1&per_page=50
Returns paginated structured audit entries:
{
  "entries": [{ "timestamp", "user", "action", "resource", "result" }],
  "days": 7,
  "page": 1,
  "per_page": 50,
  "total": 128
}
page defaults to 1 and per_page defaults to 50 (max 200). Team/hpc modes: admin-only (the trail spans every user's actions; personal mode keeps the localhost trust model).

user is the actor's canonical users.id, stored verbatim with each row — the same identity sessions and ownership columns carry. That id is a UUID for DB-created/bootstrap accounts and the username itself for env-password-provisioned identities (user, viewer) and the default personal-mode pseudo-user, so a result set legitimately mixes both shapes. Historical rows are never rewritten; resolve a UUID to a human label through GET /api/users when one is needed (admin-only).

Server-Sent Events#

GET /api/events
Accept: text/event-stream
SSE stream for real-time workflow execution events: terminal events (run_completed, run_failed, run_cancelled) plus per-rule events (rule_started, rule_completed, rule_failed, rule_skipped — parsed live from the engine's execution log). Keepalive is axum's 15-second comment ping; if the client falls behind the broadcast buffer, the stream carries a synthetic {"type":"lagged","data":{"missed":N}} event — refetch run state when you see it.

Team/hpc modes require a one-time ?ticket= minted at POST /api/events/ticket under normal bearer auth (EventSource cannot set an Authorization header; the ticket is single-use, expires in 30 s, and replaces the session token that used to travel — and leak — in the URL). POST /api/events/ticket responds {"ticket": "...", "expires_in_secs": 30}. The stream is filtered to the subscriber's own runs — admins see everything. Events carry a user field (the owning user id, or null for system-wide events). The run_started event's data.username field predates the canonical user model and also carries the canonical user id (kept for wire compatibility — read the envelope user field instead).

run_completed carries a summary field: the CLI's invalidation summary extracted from the execution log — config changes, edited rule definitions, and input-set changes that invalidated checkpoint records this run (null when the run had no invalidation activity).

The stream ends when the server shuts down (SIGTERM/Ctrl+C): the response is closed as part of the graceful drain, so an EventSource sees its error/close at shutdown rather than a hung connection. Clients should reconnect with backoff after an unexpected close.


Authentication & Authorization#

Login#

POST /api/auth/login
Content-Type: application/json

{"username": "admin", "password": "admin"}
Returns session token, username, and role.

Credentials are environment-driven, not hardcoded. The example above only works when OXO_FLOW_DEV_MODE=1 (accepts password == username) or when matching env passwords are set: OXO_FLOW_ADMIN_PASSWORD, OXO_FLOW_USER_PASSWORD, and OXO_FLOW_VIEWER_PASSWORD define the admin/user/viewer accounts. Without one of these the login returns 401 — set the env vars when launching oxo-flow serve (see Web System Architecture).

Identity is pinned server-side: the shared user/viewer passwords never mint or adopt the admin identity (or any existing API-created account) — admin signs in only with OXO_FLOW_ADMIN_PASSWORD, and a username that belongs to a managed account requires that account's own password. OAuth identities are namespaced (<provider>:<name>) and can never collide with local usernames. Sessions store the canonical users.id resolved at login; role lookups match by id only and a session whose user row is gone (deleted user, stale pre-upgrade session) is rejected with 401.

Session lifetime is operator-tunable: every minted session (login and OAuth callback) expires after OXO_FLOW_SESSION_TTL_HOURS hours — default 24, minimum 1, cap 8760 (one year). Unset keeps the historical 24-hour TTL; an unparseable or out-of-range value warns and falls back to the default. Existing sessions keep their minted expiry — the variable is read at session creation only.

Check Session#

GET /api/auth/me
Authorization: Bearer <token>
Returns {"authenticated": true, "username": "admin", "role": "admin"} or {"authenticated": false}.

Sign Out#

POST /api/auth/logout
Authorization: Bearer <token>
Revokes the presented session server-side — the token stops authenticating immediately instead of living out its configured TTL. The client should drop its local copy (logged_out: true on success). Deleting a user (DELETE /api/users/{id}) cascades: the deleted user's sessions and API keys are removed with the account.

License Status#

GET /api/license
Returns license type, validity, commercial use flag, and contact info.

Upload License#

POST /api/license/upload
Admin-only outside personal mode; the payload is bounded (64 KiB). The raw blob is not stored — only its size and SHA-256 land in the audit trail (the endpoint does not yet activate licenses — no license format or verification machinery exists yet; GET /api/license reports the built-in academic license regardless of any upload).

Users (admin)#

GET    /api/users          # List users (admin only)
POST   /api/users          # Create user
DELETE /api/users/{id}     # Delete user

Pipeline Lifecycle (v0.8 /api/pipelines/*)#

The pre-v0.8 /api/workflows/* endpoints no longer exist in the source — the running server exposes only the /api/pipelines/* API below.

Parse#

POST /api/pipelines/parse
Content-Type: application/json

{"toml_content": "<workflow TOML>", "format_version": "1.0"}
Returns structured pipeline: pipeline_id, name, version, rules (with summaries), dag (nodes + edges), stats. Pure function, zero side effects.

Validate#

POST /api/pipelines/validate
Content-Type: application/json

{"pipeline_id": "...", "toml_content": "<TOML>", "base_dir": "inputs"}
Returns { valid, errors: [{ code, message, rule, suggestion }] }. base_dir (optional, for missing-input checks) must be relative (no ..) and resolves inside the acting user's workspace.

Prepare#

POST /api/pipelines/prepare
Content-Type: application/json

{"toml_content": "<TOML>", "resolve_wildcards": true, "apply_defaults": true}
Expands wildcards, resolves environments. Returns expanded_rules_count, wildcard_combinations, environment_setup_cmds.

Build DAG#

POST /api/pipelines/dag
Content-Type: application/json

{"pipeline_id": "...", "toml_content": "<TOML>"}
Returns { nodes, edges, parallel_groups, critical_path, metrics } as structured JSON.

Format#

POST /api/pipelines/format
Content-Type: application/json

{"toml_content": "<TOML>"}
Returns canonical TOML formatting.

Lint#

POST /api/pipelines/lint
Content-Type: application/json

{"toml_content": "<TOML>"}
Returns { valid, errors: [{ code, message, rule, suggestion }] } — validation errors plus lint-level findings (e.g. missing description).

Stats#

POST /api/pipelines/stats
Content-Type: application/json

{"toml_content": "<TOML>"}
Returns aggregate pipeline statistics.

Diff#

POST /api/pipelines/diff
Content-Type: application/json

{"toml_a": "<TOML A>", "toml_b": "<TOML B>"}
Returns structured diffs: { diffs: [{ path, category, description, severity }] }.

Export#

POST /api/pipelines/export
Content-Type: application/json

{"toml_content": "<TOML>", "format": "docker|singularity"}
Generates Dockerfile or Singularity definition.

List / Save / Get / Update / Delete#

GET    /api/pipelines              # List pipelines (most recent first, up to 100)
POST   /api/pipelines              # Save new pipeline
GET    /api/pipelines/{id}         # Get pipeline with TOML content
PUT    /api/pipelines/{id}         # Update pipeline
DELETE /api/pipelines/{id}         # Delete pipeline
POST   /api/pipelines/search       # Search by name, tags, content

The pipeline's version is always derived from the stored TOML's [workflow] version key — the TOML is the source of truth. An explicit version field on POST/PUT is ignored (never stored), so a save whose content carries version = "1.2.3" reports 1.2.3 on the next GET, and updating content without touching [workflow] version bumps the reported version to match. This mirrors the CLI, where the file on disk defines the version.


Execution & Runs#

Create Run#

Database Backends: run-record persistence and lifecycle routing use the unified StorageBackend, with SQLite and feature-gated PostgreSQL implementations. A CI smoke test against a live PostgreSQL container verifies the schema bootstrap and the core StorageBackend CRUD surface, and the storage contract was live-verified against postgres:16 (2026-11-05: full contract round-trip passed, web suite 554 passed / 0 failed with the postgres feature); concurrency, crash recovery, and multi-instance behavior remain unvalidated. The runs gate is backend-agnostic: if no storage backend is initialized at all, these endpoints return 503 {"code": "RUNS_REQUIRE_STORAGE_BACKEND", "message", "detail", "suggestion"}. Team/HPC execution also lacks an OS-level tenant boundary and must not be exposed to untrusted workflow authors.

POST /api/runs
Content-Type: application/json

{"toml_content": "<workflow TOML>", "max_jobs": 4, "dry_run": false, "keep_going": false, "pipeline_id": "<uuid>", "cluster_id": "<cluster-id>", "samples": ["S1", "S2"], "targets": ["rule_name"], "include_files": {"envs/clinical/pixi.toml": "[env] ..."}, "config_overrides": ["min_quality=30"], "allow_protected_drift": false}
toml_content is the workflow source (required — the run is created from it, not from a saved pipeline); max_jobs, dry_run, keep_going, pipeline_id, cluster_id, samples, targets, include_files, config_overrides, and allow_protected_drift are top-level fields (not nested under a config object). Run-mutating endpoints (create/cancel/pause/resume/retry/clean/resume-checkpoint) require the user role or above — the viewer role is read-only — and a pipeline_id must be readable by the acting user. This flat shape is the typed CreateRunRequest schema — the OpenAPI spec declares it as the endpoint's requestBody, and the handler parses exactly that type, so the spec, the server, and the TypeScript client stay in lockstep. Returns { run_id, status: "queued", estimated_resources, execution_plan }.

Before spawning the executor the run stages the acting user's uploaded files (POST /api/files → workspace/users/<user>/inputs/) into the run's working directory, so metadata_file and relative input paths resolve the same way they do on the CLI. Files already present in the workdir are never overwritten.

pipeline_id (optional) only associates the run with a saved pipeline — the TOML still comes from toml_content. It makes the run execute in the pipeline's persistent working directory (workspace/users/<user>/pipelines/<id>), so the checkpoint survives across re-runs. Re-running with a changed config rebuilds exactly the rules referencing the changed keys (plus their DAG downstream) — the rest keep their checkpoint records and are skipped. Runs without pipeline_id get a fresh per-run sandbox and execute everything. Malformed or unknown pipeline ids are rejected (400 INVALID_PIPELINE_ID / 404 PIPELINE_NOT_FOUND).

Typed-workflow admission limit: directly submitted typed workflows are checked by the same core BioType validator as CLI preflight. Typed declarations are detected structurally from parsed TOML, including both table forms such as [types.*] / [rules.types] and inline forms such as types = { ... }. The web admission layer resolves the full include graph by staging workflow.oxoflow and all include_files sources on disk, then re-parsing through the canonical from_file path (the same machinery the CLI preflight uses) and running validate_workflow_types on the merged config. This covers typed workflows with local include graphs, typed includes, and config_overrides (issue #1018). Unresolvable includes (repo = ... git includes, missing files) surface as 400 {"code":"RUN_ERROR"} from the canonical parser — not a blanket typed-admission rejection. Only the host workflow and locally staged sources reached through [[include]] participate in that check; unrelated staged assets are ignored.

Execution flags are forwarded to the CLI executor:

  • dry_run: true spawns the preview subcommand (oxo-flow dry-run) — nothing executes; the log shows the would-be plan.
  • max_jobs defaults to 4 at this endpoint (the CLI flag itself defaults to 1): an omitted max_jobs still spawns the CLI with -j 4, which is also the value the resource estimate assumes.
  • keep_going (boolean, optional, default false) maps to --keep-going (the CLI also accepts -k). The default is the historical abort-on-required-failure policy: when a required rule fails, the engine cancels the run's in-flight jobs. With true, already-running independent rules finish, ready independent rules keep dispatching, dependents of the failure stay blocked (never executed), and the run still exits non-zero. Run-only — ignored for dry_run: true (nothing executes). Cluster (cluster_id) runs carry the flag into the remote wrapper command the same way. Web retry does not re-apply it: the runs row does not persist the failure policy, and a retry re-executes under the default. Pass keep_going on a new create request instead.
  • samples (array of strings, optional) maps to --samples (comma-joined).
  • targets (array of strings, optional) maps to -t per entry.
  • include_files (object, optional) stages [[include]] source files alongside the workflow: keys are workflow-relative paths (e.g. envs/clinical/pixi.toml), values are the file contents. Keys are validated as relative workspace paths — absolute paths, .. components, and degenerate keys (empty, .-only, trailing slash) are rejected (400 INVALID_INCLUDE_PATH). Keys are canonicalized before validation and staging, so aliases such as ./workflow.oxoflow and duplicate canonical keys, including case-insensitive aliases, are rejected; any canonical key equal to workflow.oxoflow (case-insensitively) is refused because it would overwrite the main workflow. Files are written beside workflow.oxoflow in the run directory, where the CLI's resolve_includes resolves them unchanged — the same machinery local oxo-flow run uses. Note: pipeline_snapshot in the run row stores only the host TOML, so status/DAG endpoints re-parse it without includes; the run itself executes correctly because the spawned CLI re-parses from disk. There is no per-key content cap beyond the request-body limit (2 MiB by default, body_limit in the server config).
  • config_overrides (array of strings, optional) passes CLI config overrides through to the spawned run as --arg=KEY=VALUE entries. Each entry must be a non-empty KEY=VALUE string (400 INVALID_CONFIG_OVERRIDE otherwise); full override semantics (canonicalization, unknown keys, declared-key value checks) are enforced by the CLI — it warns on unknown keys (they change nothing) and hard-fails on an invalid value for a declared key (e.g. config 'min_quality' expects an integer). A typo in an override key is therefore only a warning, not a run failure. Cluster (cluster_id) runs carry the same overrides into the remote wrapper command. Overrides and include-file keys are validated before the run row is persisted, so a 400 never leaves a ghost queued run behind (a 400 is validated before anything is written).
  • allow_protected_drift (boolean, optional, default false) forwards --allow-protected-drift to the spawned run: the CLI's run gate refuses to execute when files under a protected_data_trees tree changed since the recorded baseline; this flag is the web escape hatch to proceed anyway (the CLI still narrates the drift into the run log). Cluster (cluster_id) runs carry the flag into the remote wrapper command the same way. Web retry keeps the gate — only the original create request may relax it, mirroring the CLI's own resume. The flag only affects actual runs; with dry_run: true it is ignored (there is no gate to bypass in preview). Preview the drift first with GET /api/runs/{id}/protected-drift (see below) before overriding.
  • provenance (boolean, optional, default false) forwards --provenance to the spawned run: the CLI records sha256 checksums of every rule output into the checkpoint so the run can later be audited with GET /api/runs/{id}/provenance-verify (see below) or the CLI's provenance verify. Run-only — ignored for dry_run: true (the CLI rejects the flag there). Cluster (cluster_id) runs carry the flag into the remote wrapper command the same way. Web retry does not re-apply it (checksums are recorded by the original create request only).
  • cluster_id (optional) names a configured SSH cluster connection: the workdir is staged to that host and executed there (see Cluster Connections).

List Runs#

GET /api/runs?limit=100&cursor=<created_at>&status=<status>&q=<search>

Cursor pagination, not page numbers: pass the created_at of the last row of the previous page as cursor (created_at < cursor). limit defaults to 100 and is capped at 500. The response is the envelope { items, next_cursor, total }, where next_cursor: null means the last page. Each item carries dry_run — see Run Detail.

Run Status#

GET /api/runs/{id}/status
Real-time status: { status, phase, nodes: [{ rule, status, started_at, duration_ms, exit_code }], timeline, resources }. A rule that did not run — when-gated off (when_verdicts == false) or abort-cancelled (rule_runs status cancelled, which records no set membership) — is surfaced as skipped, never as a forever-pending node.

Timing guarantees for nodes[]: a terminal node that actually executed (success, failed, or run-level cancelled) reports started_at (RFC 3339 UTC) and duration_ms from the checkpoint's rule_runs start/end pair — completed rules pair the recorded start with their benchmark wall time; failed and cancelled rules derive the duration from the pair. Both fields are null when the underlying data genuinely does not exist: pending, running, skipped, synthetic "outputs up-to-date" completions (the rule did not execute in this run), and checkpoints written by engines predating the timestamp fields. The one legacy exception is a success node from an old checkpoint that recorded a benchmark but no start: it keeps the real duration_ms with started_at: null rather than fabricating a start.

DAG Status#

GET /api/runs/{id}/dag-status
DAG JSON with per-node live status. Color-coded: green=completed, blue=running, red=failed, gray=skipped or cancelled. Each node also carries the same started_at/duration_ms timing contract as Run Status (started_at is an RFC 3339 UTC timestamp, not a duration substitute). A rule killed because the run was cancelled reports status: "cancelled" — never failed — while a rule with its own recorded exit code stays failed even if the run was later cancelled. metrics partitions the node set exactly: completed_nodes counts successful rules only, and skipped_nodes (when-gated off or abort-cancelled siblings of a failed run) plus cancelled_nodes (run-level cancellations) keep completed + failed + cancelled + running + pending + skipped == total_nodes true at every instant. The ETA average is computed over successful nodes' durations — failures, skips, and cancellations never inflate it.

Diagnostics#

GET /api/runs/{id}/diagnostics
Deterministic error analysis: { failed_nodes: [{ rule, error_pattern, likely_cause, suggestions, auto_fixable, fix_action, relevant_log_lines }], warnings, resource_bottlenecks }. Uses 30+ deterministic error patterns — zero AI in this endpoint. Diagnosis reads only the newest 256 KiB of execution.log (seek-based tail, torn first line dropped), never the whole file.

Run Detail#

GET /api/runs/{id}
The full run row: { id, user_id, pipeline_id, workflow_name, pipeline_snapshot, duration_secs, status, phase, dry_run, workdir, started_at, finished_at, created_at } — the identity other run endpoints key off. workflow_name and duration_secs are the same values list items carry (duration_secs is the elapsed time between started_at and finished_at, null until the run finishes). (pipeline_id/pipeline_snapshot may be null for legacy rows; no pid — the host process id stays engine-internal.) dry_run mirrors the creation request (true for preview runs spawned by dry_run: true) and appears in both the list and detail responses — legacy rows created before the column existed read as false.

Run Preview#

GET /api/runs/{id}/preview
The instance-level dry-run plan (checkpoint_preview + execution order), persisted by the executor as dry-run-preview.json when a dry-run completes. Returns 404 NO_PREVIEW for runs that never produced one — or when the workdir path is not a regular file (a rule-writable workdir may contain special files). Files above the 16 MiB server-written maximum return 413 PREVIEW_TOO_LARGE.

Protected-Tree Drift Preview#

GET /api/runs/{id}/protected-drift
Read-only drift audit of the run's [engine] protected_data_trees. The response mirrors the CLI dry-run's protected_tree_drift object:

{
  "drifted_files": ["raw/S1.fq: 1024 B @<baseline-time> → 0 B @<current-time> (TRUNCATED)"],
  "new_trees": 0,
  "warnings": []
}
  • drifted_files — one human-readable line per drifted file (removed / added / changed, with (TRUNCATED) and sha256: labels — the same ManifestChange::describe rendering as the CLI dry-run and the run-gate refusal).
  • new_trees — trees declared under [engine] protected_data_trees that exist but have no baseline yet. The next real oxo-flow run adopts them; this preview never does.
  • warnings — per-tree problems that did not block the audit (unreadable tree, ignored empty path values).

The preview is read-only: unlike oxo-flow run, which persists new baselines into the checkpoint, nothing is recorded here — repeated calls are stable, and the checkpoint file is never created or modified. A run that is clean but not yet baselined reports empty drifted_files with a non-zero new_trees. The audit reads the tree as it exists now: files can still change between this call and the next run, which is exactly the window allow_protected_drift on Create Run covers. Returns 404 for unknown or foreign runs; 503 when runs are not available on this backend.

Provenance Verify#

GET /api/runs/{id}/provenance-verify
Audit the output-file checksums a run recorded into its checkpoint (CLI→web parity, oxo-flow --json provenance verify <checkpoint>). The endpoint shells out to the CLI binary against the run's workdir/.oxo-flow/checkpoint.json and forwards the verdict:

{
  "checkpoint": "/abs/workdir/.oxo-flow/checkpoint.json",
  "matched": 1,
  "mismatched": 0,
  "missing": 1,
  "cleaned": 0,
  "ok": false,
  "entries": [
    {"file": "gone.txt", "status": "missing"},
    {"file": "out.txt", "status": "matched", "expected": "sha256:…", "actual": "sha256:…"}
  ]
}
  • entries — one per audited file, flat (the CLI nests them under verify in its --json document); status is matched, mismatched, missing (vanished output), or cleaned (temporary output swept by the end-of-run gate). Checksums use the CLI's sha256:<hex> form.
  • ok — true only when mismatched == 0 && missing == 0 (mirrors the CLI's exit 1 = drift policy; drift is a 200 finding here, not an HTTP error).
  • note — present only when the checkpoint carries no stored checksums (the run was created without provenance: true): the CLI's hint to re-run with --provenance, plus matched: 0 and empty entries (ok: true).

Checksum tracking is opt-in: pass "provenance": true on Create Run (forwarded as the CLI's --provenance) so the run records sha256 checksums of its outputs into the checkpoint. The endpoint is read-only — verification never mutates the checkpoint. Errors: 404 NOT_FOUND (unknown/foreign run), 404 NO_WORKDIR (run has no workdir), 404 NO_CHECKPOINT (no checkpoint under the workdir), 400 INVALID_CHECKPOINT (the CLI rejected the checkpoint as malformed), 500 PROVENANCE_VERIFY_ERROR (CLI output unparseable — usually a stale binary; set OXO_FLOW_BIN to the current one), 503 when runs are not available on this backend.

AI Status#

GET /api/runs/{id}/ai-status
Node-level execution statuses pulled from the run's checkpoint — the input the AI endpoints (explain/interpret) consume.

Run Report#

GET /api/runs/{id}/report
The deterministic report object the report page renders: run identity, node statuses with timings, output file tree. Zero AI — the same object answers report/ask and feeds report/visualize. Like the diagnostics endpoint, the report summarizes only the newest 256 KiB of execution.log; the full log stays on GET /api/runs/{id}/logs. The report is failure-aware: a failed run's narrative headline names the failed rules and exit codes, and carries run_status + failed_rules: [{rule, exit_code, stderr_tail, stdout_tail}] from the checkpoint's structured failure data.

Report Q&A#

POST /api/runs/{id}/report/ask
Content-Type: application/json

{"question": "which rules failed and why"}
Answers from the deterministic report (pattern matching over failed nodes, durations, file tree) — not an LLM call. Returns the answer string. On a failed run, EVERY question answers from the failure data (rule, exit code, bounded stderr excerpt — falling back to the stdout tail when stderr is empty) — never a completion claim.

Run Status exit codes and failure tails#

GET /api/runs/{id}/status, /dag-status, and /instances populate exit_code and the terminal timing fields from the checkpoint's rule_runs records: completed rules report 0, failed rules their recorded code — plus bounded stderr_tail and stdout_tail on failed nodes (some tools print their root cause on stdout). The AI monitor (/ai-status) receives the same tails and classifies failures through the shared DiagnosticsEngine pattern table before falling back to exit-code-only signatures; its alerts carry a stable alert_id for cross-poll dedup. The AI chat get_run_status tool carries the same fields.

Report Visualization#

POST /api/runs/{id}/report/visualize
Content-Type: application/json

{"type": "files" | "durations" | "volcano"}
Chart data built from real sources: files maps the run's file tree ({name, size_bytes}), everything else maps per-rule durations from the checkpoint. No canned rows.

Smart Retry#

POST /api/runs/{id}/retry
Content-Type: application/json

{"skip_succeeded": true}
The retry spawns a real run, so it pays the same runs-per-minute limiter and the quota pre-flight as run creation — a retry that exceeds the budget answers 429 RUN_RATE_LIMITED / QUOTA_EXCEEDED. Like every ApiResult error, the 429 carries no Retry-After header (the typed tuple cannot carry headers) — the countdown lives in the error message instead (retry in {wait_secs}s). skip_succeeded (default true) chooses which rules re-execute. The returned plan and the spawned CLI flags are derived from the same mapping (retry_spawn_flags), so the promise and the execution cannot drift:

  • skip_succeeded: true — the retry spawns with --resume-failed (no --rerun). Failed and cancelled rules re-execute, the checkpoint's cascade invalidation re-runs their downstream dependents, and completed rules outside that closure keep their checkpoint records and are skipped. will_rerun lists the re-executed set; will_skip lists the skipped completed rules.
  • skip_succeeded: false — the retry spawns with --rerun, which re-executes EVERY rule regardless of checkpoint state. will_rerun lists every rule in the workflow (DAG order) and will_skip is empty — the same set the execution runs.

from_rule (issue #1017): when set to a rule name (e.g. "b"), the retry re-runs from that rule downstream — the engine computes the transitive-dependent closure of the named rule in the DAG and invalidates their checkpoint entries before the freshness gate runs, so downstream rules re-execute with the new upstream outputs instead of silently passing the freshness gate with stale artifacts. The spawned CLI receives --from-rule <name> directly (the same closure computation the CLI's own --from-rule flag runs — one engine, two surfaces). Failed and cancelled rules in the named rule's upstream target set have no success checkpoint record, so they re-execute as well; the plan lists them in will_rerun alongside the downstream closure. --from-rule subsumes skip_succeeded for the re-executed set: completed rules outside that set keep their checkpoint state, so the plan lists them in will_skip regardless of skip_succeeded. A from_rule naming a rule not in the DAG surfaces as a 400 RuleNotFound from the spawned CLI's own DAG walk.

The retry really executes: the returned new_run_id is a real run in the database (same workdir, same owner), spawned with the flag pair above so the rules the plan lists re-execute despite their existing outputs and the checkpoint's cascade invalidation re-runs their downstream dependents. The retry replays the original run's config_overrides: the new row carries the stored overrides forward and the spawned CLI receives the same --arg=KEY=VALUE entries, so a retry re-executes with the configuration it claims to redo instead of silently falling back to defaults. Returns { new_run_id, will_rerun: [...], will_skip: [...] } — plus an optional note when the plan is empty on a previously failed run (it died before executing any rule, e.g. a configuration error at spawn; the retry re-fails identically unless the error is fixed). When from_rule is set, the plan's will_rerun list contains the named rule plus every transitive dependent (the downstream closure) and any failed/cancelled rule in its upstream target set (those re-execute too, since they have no success checkpoint record), and will_skip contains the remaining completed rules — the same set the engine will execute at run start.

Cancel#

POST /api/runs/{id}/cancel
Cancels a running/pending run.

Pause / Resume#

POST /api/runs/{id}/pause
POST /api/runs/{id}/resume
Pauses a running run ({"reason": "..."} optional; other body fields are ignored) and resumes it. On resume, a from_rule field is rejected with 400 UNSUPPORTED_FIELD — a paused run continues in place, so re-running from a specific rule has no meaning (see Smart Retry above: cancel the run and retry it instead). A terminal run in either endpoint returns 409 RUN_NOT_ACTIVE.

Logs#

GET /api/runs/{id}/logs
Returns the execution log text — bounded to the newest 8 MiB with an explicit [log truncated …] marker line when the log is larger (buffering a multi-GB log per request was a memory-amplification surface). Read the full file from the run workdir when the marker appears.

Results#

GET /api/runs/{id}/results
Returns output file tree with sizes and types.

Files — download / preview / zip#

GET /api/runs/{id}/files?path=<relative-path>
GET /api/runs/{id}/files?path=<relative-path>&preview=true
The read-only result-delivery layer:

  • file → bytes with ETag, Content-Disposition: attachment, and single-range support (Range: bytes=a-b → 206; malformed/multi-range requests degrade to the full body per RFC 9110)
  • directory → a streaming STORE-mode zip (no temporary archive)
  • preview=true → truncated JSON for text-ish formats (100 KB cap) or inline image bytes; other types return 415 NO_PREVIEW
  • paths are sandboxed to the run's workdir (traversal rejected); sensitive filenames (.env, keys, credentials) are never served

Instances#

GET /api/runs/{id}/instances
The sample×rule instance table: every expanded instance the checkpoint knows about (qc_auto-discovered_S1 → rule qc, group auto-discovered, sample S1) with status, duration, and exit code — answers "which sample under which rule failed".

Upload & list user inputs#

POST /api/files        # multipart: field "path" (optional subdir) + file parts
GET  /api/files        # list the acting user's uploaded inputs
Uploads land in workspace/users/<user>/inputs/ (chunked to disk, 8 GiB per-file cap).


Data Discovery#

Analyze Data#

POST /api/data/analyze
Content-Type: application/json

{"paths": ["/data/*.fastq.gz", "/data/*.bam"], "max_depth": 2}
Deterministic file scanning + format inference + pipeline recommendation. Returns { files: [{ path, size, format, format_confidence, paired_with?, sample_name? }], summary, suggested_workflow }. Format detection uses filename extension matching — not AI. paired_with echoes the mate's client-supplied path when the request contains a mutual pair (_R1/_R2 or _1./_2. in the file name) and both files share a detected, non-unknown format; it is null otherwise.

Reference Discovery#

POST /api/data/reference
Content-Type: application/json

{"genome": "hg38", "components": ["fasta", "gtf", "star_index"]}
Finds installed reference genome components and reports missing ones with download commands.

Reference Status#

GET /api/data/reference/status
Which common reference genomes have their expected components present at the standard /data/references/ locations — the deterministic counterpart to the POST discovery above. Probed components per genome: hg38 and mm10 check genome.fa, star (STAR index directory) and genes.gtf; mm39 checks only genome.fa and star (the [[references]] auto-build needs no annotation to construct an index). A genome counts as installed only when at least one probed component exists; builds with nothing on disk are reported under missing_genomes. Response shape (components is a legacy alias of found — both carry the absolute paths that exist on disk):

{
  "installed": [
    { "genome": "hg38", "components": ["/data/references/hg38/genome.fa", "..."],
      "found": ["..."], "missing_components": ["..."] }
  ],
  "missing_genomes": ["mm39"],
  "missing": ["/data/references/mm39/genome.fa"],
  "download_commands": ["# mm39: stage the expected files under /data/references/mm39 (genome.fa, STAR index) ..."]
}

download_commands carries staging guidance, not fetchable URLs: oxo-flow hosts no reference mirror. Point users at reference_dir + [[references]] auto-build in the workflow format for index construction.

Data Perception#

POST /api/data/perceive
Content-Type: application/json

{"paths": ["/data/"], "description": "RNA-seq paired-end human samples"}
Data-profiling report built from paths (the same scanner as /api/data/analyze) or from a natural-language description. Powers the "what data do I have" panel.

Samplesheet Parsing#

POST /api/data/samplesheet/parse
Content-Type: application/json

{"content": "sample,fastq_r1,fastq_r2,condition\nS1,a_R1.fastq.gz,a_R2.fastq.gz,ctrl"}
Parses pasted CSV/TSV samplesheet content into the standard column structure (sample, fastq_r1, fastq_r2, condition). Returns { format, samples: [...] } — the bridge from a spreadsheet to [[sample_groups]].

Verify Data#

POST /api/data/verify
Content-Type: application/json

{"tree": "inputs/delivery-2026-10", "md5_content": "d41d8cd98f00b204e9800998ecf8427e  raw/S1.fq\nd41d8cd98f00b204e9800998ecf8427e  raw/S2.fq\n"}
Audits a delivered data tree against a vendor md5 sidecar — the web slice of the CLI's oxo-flow verify-data command. tree is workspace-relative (uploads live under inputs/, run workdirs under runs/<id>/; absolute paths and .. are rejected). The sidecar arrives inline in the standard md5sum text format and is staged server-side; the CLI subprocess streams an md5 over every file.

Response is HTTP 200 even when drift is found — the counts ARE the finding:

{
  "tree": "inputs/delivery-2026-10",
  "matched": 12, "mismatched": 1, "missing": 0, "extra": 1,
  "entries": [
    {"file": "raw/S1.fq", "status": "matched", "expected": "d41d...", "actual": "d41d..."},
    {"file": "raw/S2.fq", "status": "mismatched", "expected": "aa...", "actual": "bb..."},
    {"file": "raw/unlisted.txt", "status": "extra", "actual": "cc..."}
  ]
}

status vocabulary: matched, mismatched (includes unreadable files, with a note), missing, extra (present but unlisted — always non-compliant). An extra entry may carry a note instead of an actual checksum in two cases: the file could not be read for checksumming, or a subdirectory of the tree could not be traversed at all (tree walk failed note — the audit is incomplete, the named path may be a directory, and the verdict has already failed with exit-equivalent extra > 0). An empty sidecar returns all-zero counts with "note": "The md5 sidecar lists no files." and no extras scan. A sidecar the CLI hard-rejects (malformed line, duplicate path, traversal) yields 400 INVALID_SIDECAR with the CLI's stderr as detail. HTTP errors are otherwise reserved for infrastructure problems: 400 INVALID_TREE (unknown path or traversal), 500 VERIFY_ERROR (spawn failure or unparseable CLI output). Errors use the structured {code, message, detail, suggestion} shape. The inline sidecar counts against the server's default ~2 MiB JSON body limit (~50k entries); larger sidecars should be split by directory.


Templates#

GET    /api/templates
POST   /api/templates
GET    /api/templates/{id}
DELETE /api/templates/{id}
Built-in and user-created pipeline templates. System templates are read-only. The list endpoint does not accept filter query parameters.


Plugins#

Validate Plugin#

POST /api/plugins/validate
Content-Type: application/json

{"manifest": {"name": "...", "version": "1.0", "plugin_type": "rule"}, "trusted_keys": {"key1": "hex..."}}
Validates a plugin manifest and optionally verifies its HMAC signature against trusted keys. Returns { valid, name, version, plugin_type, signature_valid, errors }.


AI (Phase 2 — calls deterministic APIs above)#

POST /api/ai/translate          # NL intent → validated .oxoflow (JSON response)
POST /api/ai/translate/stream   # Same, streamed over SSE (progress → done events)
POST /api/ai/explain            # Explain run failure + suggest fix (run owner or admin)
POST /api/ai/interpret          # Interpret results with caveats (run owner or admin)
POST /api/ai/optimize           # Optimize pipeline parameters (pipeline reader via id; owner-supplied TOML unaffected)
GET  /api/ai/config             # Get AI provider configuration (public; authenticated admins see provider/model)
POST /api/ai/config             # Update the shared provider (admin-only outside personal mode)
POST /api/ai/test               # Test the provider (admin-only outside personal mode)
GET  /api/ai/config/user        # The acting user's private provider config
PUT  /api/ai/config/user        # Set the acting user's private provider config
GET  /api/ai/config/server      # Server-level provider config (admin-only outside personal mode)
PUT  /api/ai/config/server      # Set the server-level provider config
GET  /api/ai/config/effective   # The config a run would use: user → server → env fallback chain
GET  /api/knowledge/tools       # Tool catalog the AI can call
GET  /api/knowledge/skills      # Skill catalog the AI can follow

Both knowledge endpoints accept ?q=<query>&limit=<1-50> (default 20). /api/knowledge/tools treats an empty q as a browse: it returns the first limit Bioconda entries. /api/knowledge/skills tokenizes the query and ranks matches (rare terms first); an empty or unmatched q returns an empty skills array — total is always the full catalog size, not the hit count.

The translate endpoints take { "intent": "<natural-language description>", "context": { "data_analysis_id": "<uuid>" } } — intent is REQUIRED (sending only a prompt field fails deserialization); context is optional.

POST /api/ai/translate (the non-streaming variant) budgets the WHOLE generation — every tool and validation round — at OXO_FLOW_AI_TIMEOUT_SECS (default 300 s). The route budget fires one second before the provider client's per-request timeout, and a timeout-class provider error is classified into the same answer, so the documented response holds whichever timer expires first. When the budget expires it answers 504 {code: "AI_TIMEOUT"} with the standard {code, message, detail, suggestion} body, and suggestion names the knob. Use POST /api/ai/translate/stream for generations expected to run longer: the SSE route is not capped by this total budget.

GET /api/ai/config stays public for anonymous feature discoverability: anonymous callers (and non-admin callers outside personal mode) receive only {provider: "", model: null, api_url: null, is_configured}. Presenting a valid credential opts the request into normal authentication, so an admin sees the configured provider, model, and URL (never the API key); an invalid credential is rejected with 401 rather than silently downgraded to the masked anonymous payload.

The config endpoints form a three-level resolution chain: a user's private provider overrides the server-wide one, which overrides environment variables. GET /api/ai/config/effective shows the resolved result and which level supplied each field. The effective.model field is the model the active backend was actually constructed with: an explicit config model wins, otherwise the environment override (ANTHROPIC_MODEL, OPENAI_MODEL, OLLAMA_MODEL, OXO_FLOW_AI_MODEL) or the provider default is reported — it is null only when no provider is configured. GET /api/ai/config returns the same resolved model. API keys are never echoed back — responses mark key_set instead. Rows are encrypted at rest only when OXO_FLOW_MASTER_KEY is configured; without it, the local database uses the documented plaintext compatibility mode.

There is deliberately no DELETE endpoint: resetting a tier means saving provider: "disabled" through the same PUT endpoints. A disabled user-tier row makes that user fall back to the shared runtime; a disabled server-tier row disables the shared runtime (immediately via the same reconfiguration path, and again on the next startup when the row is restored). Omitting provider in a PUT body defaults to "disabled", so a save that only flips a toggle must carry the current provider over (the bundled Settings page does exactly that).

Provider names are validated before either configuration endpoint writes to storage. An unsupported name returns 400 INVALID_AI_PROVIDER and leaves the existing configuration untouched. If storage succeeds but the runtime cannot activate the new provider, the endpoint returns 500 AI_RECONFIGURE_FAILED; the durable row remains available for the next startup, but the caller must correct the provider settings rather than assuming the running process switched.

Pinned disabled AI (OXO_FLOW_AI_PROVIDER=disabled)#

An explicit disabled is an operator-level switch: it outranks every configuration tier and every runtime update. While it is set:

  • POST /api/ai/config (runtime-only — applying to the live registry is its only effect) returns 409 AI_DISABLED_BY_ENV with the standard {code, message, detail, suggestion} shape.
  • PUT /api/ai/config/server and PUT /api/ai/config/user still persist the row and return 200 with applied_to_runtime: false: Settings stays usable and the saved configuration applies after an unpinned restart, but the pinned runtime is never reconfigured.
  • POST /api/ai/test returns 200 with success: false, provider: "disabled", and a message stating that no connection was attempted. No provider is resolved and no HTTP client is constructed.
  • GET /api/ai/config and GET /api/ai/config/effective report provider: "disabled" / is_configured: false. The per-user and server reads add disabled_by_env: true and report the effective configured as false; the stored row is still returned so the Settings page can show what is saved.
  • Generation, chat, explain, interpret, and optimize keep answering 503 AI_NOT_CONFIGURED.

Lifting the pin requires restarting the server without the variable (or with a provider name); configuration saves are rejected, not queued, while it is set.

See AI Translation Layer for details.


Chat (v0.8 AI Companion)#

POST /api/chat/send        # Streamed chat over SSE
POST /api/chat/send/json   # Same, single JSON response
GET  /api/chat/sessions    # The acting user's chat sessions

Intent routing#

Chat requests are classified before any generation work happens, by a deterministic keyword/pattern classifier (domains/chat/intent.rs) — no extra model call, so routing never doubles a request's token cost. The decision is returned as the mode field:

Mode Trigger Behavior
pipeline Pipeline vocabulary (workflow/pipeline/rules/TOML/DAG, omics assays, file formats, tool names), run-diagnosis wording, an attached data_paths/samplesheet, or an explicit run_id scope The existing PipelineGenAgent loop: validated .oxoflow TOML plus toml_content, rules, and dag
chat Greetings/small talk, explanation questions ("What is FastQC?"), meta-instructions ("Reply with the single word OK.") — i.e. no pipeline vocabulary One plain-text completion; message carries the reply; no pipeline generation

The bias is deliberately conservative: a message naming a workflow, an assay, a file format, or a tool routes to generation unless it is phrased as an explanation question. context.data_paths / context.samplesheet and a request-level run_id always force generation (data analysis and run diagnosis need the tool loop). context.intent is a pipeline label hint only — it never changes the route, so the SPA's page name cannot turn conversational turns into pipelines.

POST /api/chat/send/json — response shapes#

Chat intent (one provider completion, no workflow):

{ "mode": "chat", "message": "OK" }

Pipeline intent (the existing fields are unchanged; mode is added):

{
  "mode": "pipeline",
  "pipeline_id": "…",
  "toml_content": "[workflow]\n…",
  "intent": "RNA-seq analysis",
  "data_report": null,
  "validation": { "valid": true, "errors": [] },
  "rules": [ … ],
  "dag": { "nodes": [ … ], "edges": [ … ], "parallel_groups": [ … ] }
}

POST /api/chat/send — SSE event contract#

The stream opens with a ready handshake, then forwards the real agent loop (tool rounds, validation rounds) as typed events:

Event Payload Meaning
ready {keepalive_secs, idle_timeout_secs, mode} Handshake, sent before the first provider call. keepalive_secs is the comment-frame cadence; idle_timeout_secs (3 × keepalive_secs) is the client watchdog budget — silence longer than this means the connection is dead (proxies can drop it silently), and the client must abort and offer a retry rather than spin forever. mode is the intent route (chat or pipeline), so the client can pick its renderer up front.
status {message} Human-readable phase (planning, executing tools). Pipeline mode only.
progress {round, max_rounds, step, tool, elapsed_ms, prompt_tokens, completion_tokens} Structured loop progress, emitted at every phase boundary: step is plan, generate, tool (with tool naming the running tool), or validate. elapsed_ms is measured from loop start; token counts are cumulative. Clients render this to distinguish "slow" from "dead". Pipeline mode only.
tool_call {name, args} The model requested a grounded tool call. Pipeline mode only.
tool_result {name, summary} The tool finished; summary is a truncated preview (errors ride here too). Pipeline mode only.
text {chunk} One complete round response (token-level streaming is not implemented). In chat mode this carries the whole plain-text reply.
action {action_type, data} Structured action — pipeline_ready carries the validated TOML and its validation result. Pipeline mode only; chat turns never emit it.
error {code, message} Terminal failure (CHAT_ERROR, AI_NOT_CONFIGURED, EMPTY_MESSAGE). In pipeline mode the loop aborts and its transcript is archived with outcome: "Failed" and the reason.
done {session_id, mode, rounds, prompt_tokens, completion_tokens} / {session_id, mode: "chat", message} Terminal success marker. Pipeline mode: always sent after action, clients stop reading here. Chat mode: message repeats the full reply (so a lost text frame cannot leave an empty bubble), and the token counters are omitted. The early EMPTY_MESSAGE / AI_NOT_CONFIGURED error paths also terminate with {session_id, mode}.

Transport behavior:

  • Keep-alive comment frames (: ping) are emitted every 15 s. They are not events, but they are bytes on the wire — an inactivity watchdog must treat them as traffic so a slow but healthy generation (8–14 minute runs are normal) is never cut off.
  • Each provider request is bounded by OXO_FLOW_AI_TIMEOUT_SECS (default 300 s). A stalled endpoint surfaces as an error event within that budget, never as an indefinite hang.
  • A client that disconnects mid-generation flips the loop's cancel flag; the orchestrator stops between provider calls and before each tool rather than burning paid rounds nobody is reading. The aborted run is archived (below).
  • The browser-side watchdog is derived from ready.idle_timeout_secs; on expiry the client aborts the fetch, renders the error with a Retry action, and re-enables the send button.
  • POST /api/chat/send/json (the non-streaming variant) is subject to the same whole-generation route budget as POST /api/ai/translate (described in the AI section above): every tool and validation round budgeted at OXO_FLOW_AI_TIMEOUT_SECS (default 300 s), firing one second before the provider client's per-request timeout, with 504 {code: "AI_TIMEOUT"} when it expires. The SSE route above is not capped by this total budget; use it for generations expected to run longer.

Chat session archives#

Every agent-loop run — successful or aborted — is persisted under .oxo-flow/ai_sessions/ as <timestamp>-<command>-<id>.json; set OXO_FLOW_AI_SESSION_DIR to redirect the archive (tests and CI point it at a scratch directory so a run never writes into an operator's real archive). Successful generations archive outcome: "Success"; a run that fails archives outcome: "Failed" with an error reason (the post-mortem record). A run aborted by a JSON route's total budget (the 504 AI_TIMEOUT path) also archives outcome: "Failed" with the timeout reason — the route timer drops the orchestrator future before it can archive its own failure, so the route writes the synthetic record. Writes are temp-sibling + atomic rename, so a reader never observes a truncated file. Pruning only reclaims archives older than ten minutes, so a fresh failure record — or the file an operator is inspecting — is never rotated out by concurrent writers. Chat-intent replies do not run the agent loop and therefore produce no archive; they are appended to the chat session store like every other assistant message.

AI provider probe#

POST /api/ai/test is single-flight: concurrent callers share one outbound provider round-trip (queued behind an async mutex) and a result younger than 10 seconds is reused. The probe calls the provider directly and never writes an ai_sessions archive, so repeated settings checks cannot pollute the audit trail or trip the provider's rate limiter.


OAuth#

POST /api/auth/oauth/authorize   # Begin the OAuth flow → provider authorize URL
POST /api/auth/oauth/callback    # Provider callback → session token

The redirect URI is OXO_FLOW_OAUTH_REDIRECT_URI when set, with a fixed fallback of http://localhost:3000/api/auth/oauth/callback (see the environment table in AGENTS.md).

The local user record is idempotent on the provider identity: the canonical users row is keyed by the namespaced id oauth:{provider}:{provider_user_id} (e.g. the ORCID iD), so re-login reuses the same row. Display names are NOT unique across provider accounts — two ORCID accounts can share a given name — so when a display name is already taken the row is persisted under a fallback username derived from the unique provider id ({provider}:{provider_user_id}); the login still succeeds and the session always resolves to a real user row. When the user row cannot be ensured (database unavailable or insert fails), the callback answers 503 {code:"DB_ERROR"} instead of minting a session that would fail closed on every subsequent request.


Collaboration (Phase 3)#

POST /api/pipelines/{id}/fork    # Fork into workspace (owner = session user)
POST /api/pipelines/{id}/share   # Share (link/public only; URL follows the actual transport)
POST /api/pipelines/import       # Import from oxo+http(s):// share URL
GET  /api/share/{token}          # PUBLIC landing payload (no session required)
GET /api/share/{token} powers the share landing page: pipeline identity, DAG rule order, TOML, owner, expiry, and the most recent terminal run — the token itself is the authorization. Expired links return 410.

Share URLs minted by POST .../share use oxo+http:// by default (the server has no native TLS) and oxo+https:// only when OXO_FLOW_PUBLIC_URL declares a public HTTPS origin; the import endpoint accepts both schemes. An override that carries its own mount path (e.g. https://flows.example.com/oxo-flow behind a sub-path proxy) replaces the local --base-path prefix instead of being appended to it.

See Collaboration for details.

Version History#

GET  /api/pipelines/{id}/revisions        # Snapshot list (newest first, ≤ 50)
GET  /api/pipelines/{id}/revisions/{rev}  # One snapshot's full TOML
POST /api/pipelines/{id}/rollback         # {"revision_id": ...} — restore
Every save/update snapshots the previous content; rollback preserves the current version as a new revision (nothing is lost).

Run Administration#

POST /api/runs/{id}/clean              # CLI clean on the run's workdir
POST /api/runs/{id}/resume-checkpoint  # CLI resume from .oxo-flow/checkpoint.json
resume-checkpoint continues an unfinished run in place as a NEW run row ({"max_jobs": 2} optional). Both are ownership-checked like every other run endpoint.

Webhooks#

GET /api/webhook   # admin-only outside personal mode — { enabled, url, secret_set, events, signature_scheme }, secret never echoed
PUT /api/webhook   # admin-only outside personal mode — {"enabled": true, "url": "https://...", "secret": "...", "events": [...], "signature_scheme": "..."}
Runs POST a signed payload to the configured URL on terminal states. The signature scheme is configurable:

  • hmac-sha256 — RFC 2104 HMAC (X-OxoFlow-Signature: hmac-sha256=<hex>). This is the default for a new web-server row and for a workflow [webhook] section.
  • sha256-keyed — frozen sha256(secret‖body) compatibility. Existing web-server rows keep this scheme when an update omits signature_scheme; it is not the default for a new row.

An unknown or non-string signature_scheme returns 400 INVALID_SIGNATURE_SCHEME rather than being silently coerced. The PUT response and subsequent GET report the scheme actually stored. The settings UI exposes the choice and labels the legacy scheme as compatibility-only.

events is an exact subscription: a terminal run posts only the event it actually reached, so ["workflow_completed"] stays silent on a failed run and ["workflow_failed"] stays silent on a successful one. An empty list (a saved [], or a row predating per-event filtering) subscribes to both terminal events.

Validation on PUT: url must parse as an absolute http:// or https:// URL with a host (400 INVALID_URL otherwise — a bare https:// prefix no longer passes). Enabling delivery requires a usable signing secret: either supply secret in the request or leave a previously stored one in place (400 WEBHOOK_SECRET_REQUIRED otherwise) — a webhook URL is a bearer credential, and unsigned delivery lets anyone who learns the URL forge events. Disabling (enabled: false) never demands a secret. A stored secret that fails to decrypt (e.g. the master key was rotated) reads as absent: enable is refused until a fresh secret is supplied — silently signing with an empty key would produce forgeable deliveries.

Admin-only outside personal mode — the endpoint is shared infrastructure.

API Keys#

POST   /api/auth/keys       # {"name": "ci-bot"} → { id, name, key } (shown once)
GET    /api/auth/keys       # The acting user's keys (hashes only)
DELETE /api/auth/keys/{id}  # Revoke immediately
Machine credentials: send X-API-Key: oxo_... instead of a Bearer session. Keys resolve to the same ownership context (a key's requests see exactly what its owner sees), are stored as SHA-256 hashes, and revocation is immediate.

Quota#

Runs pre-flight the quota tracker with the workflow's declared threads and memory; over-limit requests get 429 QUOTA_EXCEEDED with the violation list.

GET  /api/quota           # current limits and usage
PUT  /api/quota           # update limits (admin-only; personal mode bypasses admin checks)

PUT /api/quota accepts { max_concurrent_runs, max_total_threads, max_total_memory_mb, max_runs_per_day } — all four fields are required (no defaults applied server-side). Usage is visible at GET /api/quota ({ enabled, limits: {…}, usage: { active_runs, used_threads, used_memory_mb, runs_today } }).

Cluster Connections & Remote Execution#

GET    /api/clusters                  # configured SSH connections
POST   /api/clusters                  # upsert (admin-only outside personal mode)
DELETE /api/clusters/{id}
POST   /api/clusters/{id}/probe       # SSH connectivity + scheduler detection (admin-only outside personal mode)
The stored ssh_key is a secret: it is never returned by any response (including the upsert echo) — clients see ssh_key_set: true | false instead. When OXO_FLOW_MASTER_KEY is set, the key is sealed at rest with the same AES-256-GCM scheme as AI provider keys; without it, the database stores plaintext compatibility-mode rows. ssh_key accepts a path or key material: material is written to a private 0600 temp file for the SSH invocation, so either form works. POST /api/runs accepts cluster_id: the run then stages its workdir to the remote host (tar over stdio — no rsync), launches it detached (own session via setsid, with nohup fallback) so the launch call returns while the workflow keeps running, and pulls the results back on completion so every downstream endpoint (logs, files, report) works unchanged. Cancel kills the wrapper's recorded process group, so the remote CLI and rule subprocesses stop rather than being orphaned. See Run on a cluster.


HPC#

GET /api/hpc
Returns scheduler status (SLURM, PBS/Torque, LSF, SGE), available queues, and node count. Nodes are deduplicated by name (SLURM lists a node once per partition it belongs to). This route is only mounted when the server runs in hpc mode, and the generated OpenAPI document only advertises it in that mode.


DAG Editing#

POST /api/pipeline/{id}/command   # Apply an edit command (add/remove rule, connect, ...)
POST /api/pipeline/{id}/undo      # Revert the last edit (JSON body: {"toml_content": ...})
POST /api/pipeline/{id}/redo      # Re-apply the last undone edit (JSON body: {"toml_content": ...})

undo/redo require a JSON body carrying the caller's current TOML; the stacked edit is applied only when it matches that state (otherwise 404). See DAG Edit API for the full command reference.


See Also#