Deployment Modes#
oxo-flow supports three deployment modes from a single binary. Choose the mode that fits your team size and infrastructure.
Quick Reference#
# Personal workstation (default)
oxo-flow serve
# Team server
oxo-flow serve --mode team
# HPC submit panel
oxo-flow serve --mode hpc
Mode 1: Personal (Default)#
Use when: You're the only user, working on your own machine.
| Setting | Value |
|---|---|
| Network | 127.0.0.1:8080 (localhost only) |
| Database | SQLite file (oxo-flow.db) — override with DATABASE_URL |
| Auth | None (single user) — management endpoints (Users, Audit) follow the same localhost trust model |
| Workspace | workspace/users/local_user/runs/<run_id> |
# Start
oxo-flow serve
# With custom port
oxo-flow serve -p 9090
# Open browser
open http://localhost:8080
Mode 2: Team#
Use when: Multiple users share a server or cloud instance.
| Setting | Value |
|---|---|
| Network | 127.0.0.1:8080 by default — pass --host 0.0.0.0 to bind all interfaces |
| Database | SQLite (default) or PostgreSQL — see Database Selection |
| Run execution | Backend-routed code exists for SQLite and PostgreSQL; schema bootstrap + core CRUD are CI-smoke-verified on live PostgreSQL, while concurrency/crash-recovery/multi-instance remain unvalidated. Team mode has no OS-level tenant execution boundary, so expose it only to fully trusted workflow authors |
| Auth | Password env vars (OXO_FLOW_ADMIN_PASSWORD / OXO_FLOW_USER_PASSWORD / OXO_FLOW_VIEWER_PASSWORD) + optional ORCID/GitHub OAuth |
| Workspace | workspace/users/<username>/runs/<run_id> |
# Start with SQLite (default) and bind to all interfaces
oxo-flow serve --mode team --host 0.0.0.0
# Set auth credentials
export OXO_FLOW_ADMIN_PASSWORD=...
export OXO_FLOW_USER_PASSWORD=...
export OXO_FLOW_VIEWER_PASSWORD=...
export ORCID_CLIENT_ID=...
export ORCID_CLIENT_SECRET=...
# Test the ORCID flow against sandbox.orcid.org instead of production (optional)
# export ORCID_SANDBOX=1
export GITHUB_CLIENT_ID=...
export GITHUB_CLIENT_SECRET=...
oxo-flow serve --mode team
Authentication: role-based password authentication comes from the
OXO_FLOW_*_PASSWORD environment variables (with a dev-mode fallback when
OXO_FLOW_DEV_MODE=1). ORCID and GitHub OAuth are available as separate
login endpoints when their client credentials are configured — there is no
automatic fallback chain between mechanisms.
User management: beyond the env-var credentials, accounts created on
the Users page (or POST /api/users) carry a bcrypt-hashed password in
the database and sign in through the same /api/auth/login endpoint.
State-changing requests write an audit entry visible on the Audit page
or through GET /api/audit, with the acting user and recorded outcome. The
entry's durability is explicit per action (WEB-15 / issue #1140):
- Security-relevant actions — authentication attempts, user/role
changes, machine-key changes, run creation and lifecycle control,
pipeline deletion and share grants, AI provider key changes, cluster
connection secrets, webhook configuration, and license upload — fail
closed: if the row cannot be written the request answers
500 AUDIT_WRITE_FAILEDinstead of acknowledging the action. - Informational mutations — AI generation, parsing/validation, DAG edits, chat, and pipeline create/update/fork — log a warning and still answer normally if the write fails.
The audit row is written after the handler, not in the same transaction, so
a REQUIRED-audit 500 means the action may already have been applied —
verify the affected resource before retrying, since a blind retry can
duplicate the effect. Successful sign-ins are a handler-owned REQUIRED row
(the middleware only sees the pre-auth request and would misattribute it);
the session is persisted before the audit write, so a login 500 means a
session token may already have been minted. For every mutation the
middleware handles, the structured event stream emits an audit event even
when the database insert fails. Do not treat a missing entry as proof that
no action occurred; check the structured event stream as well. The trail
spans all users and is admin-only outside personal mode.
Workspace isolation:
Mode 3: HPC#
Use when: The web UI is a submit panel; actual execution happens on a cluster.
| Setting | Value |
|---|---|
| Network | 127.0.0.1:8080 by default — pass --host 0.0.0.0 to bind all interfaces |
| Database | SQLite or PostgreSQL — see Database Selection |
| Auth | Same as Team mode |
| Executor | SLURM / PBS / LSF / SGE |
| Resources | Scheduler-managed |
Connecting the app to a cluster (the deployment model):
- App anywhere, cluster elsewhere. The web UI can run on your
workstation, a lab portal server, or any cloud VM — it does not need to
live on the cluster. Cluster execution happens where the CLI runs:
SSH into the login node (or run the CLI there directly) and use
oxo-flow cluster submit/oxo-flow run --profile clusterto dispatch work through SLURM/PBS/LSF/SGE. - App on the login node (
--mode hpc). When the web server itself runs on a cluster login node, hpc mode detects the scheduler at startup (sinfo/qstat/pbsnodes/bjobs) and exposesGET /api/hpcwith live queue/partition status in the UI. - App mounted at a sub-path. On shared portal servers the app can be
mounted under any prefix with
--base-path /oxo-flow— the router, the SPA, and all asset URLs follow the mount point automatically (the SPA index gets a<base>tag injected at serve time), so it composes with reverse proxies without a rebuild.
# App on the login node, mounted under /oxo-flow
oxo-flow serve --mode hpc --base-path /oxo-flow
# App on your workstation, cluster execution via the CLI on the login node
ssh login-node
oxo-flow cluster submit workflow.oxoflow -b slurm
Switching Modes#
Database Selection (DATABASE_URL)#
All serving entry points — oxo-flow serve, the standalone
oxo-flow-web binary, and the desktop app — share one database-selection
rule: DATABASE_URL starting with postgres:// (or postgresql://)
selects the PostgreSQL backend; anything else (including unset, which
defaults to sqlite://oxo-flow.db in the working directory) stays SQLite.
# PostgreSQL (storage-backed web metadata and backend-routed run records)
export DATABASE_URL=postgres://oxo:secret@db.lab.example.edu:5432/oxo_flow
oxo-flow serve --mode team --host 0.0.0.0
Two constraints apply:
- Build feature: PostgreSQL support is a compile-time feature. The
published Docker image is built WITHOUT it — rebuild with
cargo build --features postgres(or the equivalent in your own image). On a build without the feature, apostgres://DATABASE_URLaborts startup with a clear error instead of silently falling back. - Run execution: the
StorageBackendtrait has PostgreSQL code paths for run-record persistence. CI runs a live-PostgreSQL smoke test (aservices: postgrescontainer) that verifies the schema bootstrap and the coreStorageBackendCRUD surface — users, runs, run nodes, audit logs, webhook config, chat sessions — against a real server. That does NOT extend to production parity: concurrency, crash recovery, and multi-instance behavior remain unvalidated, so test them before relying on PostgreSQL for execution metadata. The runs gate is backend-agnostic in behavior — any initialized storage backend passes — and answers503 RUNS_REQUIRE_STORAGE_BACKENDonly when no backend was initialized at all. - Tenant safety: team/HPC authorization is not an OS or container execution boundary. A normal workflow can run shell commands under the server account and may access server environment secrets. Do not invite untrusted tenants until the workspace isolation described in Multi-Tenancy is supplemented with your own OS-level isolation (dedicated service user, container, or similar).
Startup Verification#
On startup, oxo-flow prints the version banner, the serve mode and address, and tracing log lines:
oxo-flow v0.32.0 — Rust-native bioinformatics pipeline engine
Serve: Starting oxo-flow web server in personal mode on 127.0.0.1:8080
INFO Logging initialized at logs
INFO AI registry initialized: provider=disabled, model=default, enabled=false
INFO Building router for mode: personal
INFO Starting oxo-flow web server in personal mode on 127.0.0.1:8080
Log lines carry timestamps in a terminal, and the mode, host, and port follow --mode/--host/-p.
The banner line is interactive-terminal decoration: when stderr is redirected
(oxo-flow serve > serve.log 2>&1, nohup, service managers), it is omitted —
the log lines alone identify the running version.
Platform Configuration#
Server-tier settings, AI defaults, and SSH cluster definitions can live in
a platform config file — the lowest-precedence layer (CLI flag > env
var > config file > built-in default). The file is looked up at
OXO_FLOW_CONFIG, then oxo-flow.web.toml in the working directory, then
~/.config/oxo-flow/web.toml.
[server]
mode = "team" # personal | team | hpc
host = "0.0.0.0"
port = 8080
base_path = "/oxoflow" # sub-path mounts behind a reverse proxy
[ai]
provider = "deepseek" # anthropic | openai | deepseek | ollama | disabled
api_url = "https://api.deepseek.com"
model = "deepseek-v4-pro"
api_key_env = "DEEPSEEK_API_KEY" # secrets stay in env vars, never inline
[[clusters]]
id = "lab-slurm" # stable key (import is idempotent)
name = "Lab SLURM cluster"
ssh_host = "login.lab.example.edu"
ssh_port = 22
ssh_user = "bioinf"
ssh_key = "~/.ssh/id_ed25519"
scheduler = "slurm" # auto | slurm | pbs | lsf | sge
remote_dir = "~/oxo-flow-jobs"
The same choices are editable at runtime on the Clusters & Servers
page: add an SSH endpoint and press Probe — the server performs a
real SSH round-trip (BatchMode, 8 s timeout), reports the remote hostname
and detects the scheduler (slurm/pbs/lsf/sge) with its version.
Unknown fields in the config file are rejected loudly, not silently
ignored.
Multi-Tenancy#
Team/HPC modes scope every resource to the acting user:
- Ownership: runs and pipelines are owned by the session's user; foreign resources 404 (existence never leaks). Admins see and control everything; viewers/guests get read access to workspace-visible pipelines only.
- Anonymous surface:
/api/system,/api/metrics,/api/ai/test, and/api/hpcrequire authentication;GET /api/ai/configstays public;/api/eventsrequires a one-time?ticket=fromPOST /api/events/ticket(EventSource cannot set headers) and streams only the subscriber's runs;/api/share/{token}is public by design (the token is the credential). - Env-password logins auto-provision a real users row (previously any
username shared the
defaultpseudo-user's identity). - API keys (
X-API-Key: oxo_…) are first-class machine credentials with the owner's exact permissions; stored hashed, instantly revocable. - Shared infrastructure: the server AI provider, webhook endpoint, and SSH cluster connections are admin-managed outside personal mode.
Server Data Layout (OXO_FLOW_WORKSPACE)#
The web server keeps all user-scoped data under one workspace root,
OXO_FLOW_WORKSPACE (default: a relative workspace/ directory under the
server process's cwd):
users/<username>/inputs/— files uploaded viaPOST /api/files; mirrored into each new run's workdir at Create Run (POST /api/runs, which stagesinclude_filesstraight into the run dir without touchinginputs/)users/<username>/runs/<run_id>/— each run's own immutable workdir (the spawned CLI runs here; the workflow snapshot, checkpoint, logs, outputs, and envs live under it). Every create request — including for a saved pipeline — gets a fresh directory, so a later run can never overwrite an earlier run's artifacts (issue #1130 / audit RB-02); onlyretry/resume-checkpointcontinue their predecessor's directory.users/<username>/pipelines/<pipeline_id>/— legacy pre-#1130 run workdir. New creates no longer use it, but retrying or resuming a run created before the upgrade still writes into that run's recorded workdir here. Leave the tree in place while any pre-upgrade run may be retried; archive it only once those runs are no longer needed
Usernames and run ids are traversal-validated before they are joined onto
the root. In container or systemd deployments, point OXO_FLOW_WORKSPACE
at a persistent volume — a server started from a different cwd otherwise
silently loses sight of previous runs. The workspace root is server-tier
configuration only; it has no per-user override.
Per-User AI Credentials#
Every user-facing AI call (chat, translate, explain, interpret, optimize)
resolves the acting user's own saved provider (PUT /api/ai/config/user)
before falling back to the shared runtime:
- resolution order per call: the user's row (unless
disabled) → the server-level provider (env / admin config) → default - a non-admin's saved key never reconfigures the shared runtime — it is cached per user and invalidated on every config write
- a per-user row without an api key carries an empty key: their calls fail loudly instead of silently borrowing the server's key (no shared-secret leakage in either direction)
- server-level writes (
POST /api/ai/config,PUT /api/ai/config/server) stay admin-only outside personal mode and invalidate all per-user fallbacks
Testing & Development#
OXO_FLOW_DISABLE_RATE_LIMIT=1 disables the per-IP rate limiter — the
browser e2e suite (and any local automation) legitimately exceeds the
100 req/min budget and must set it. Never set it on a team/hpc
production server: the limiter is the brute-force protection.
Run Control Status (all modes)#
Run control uses process signaling, but lifecycle truth remains prototype-grade:
- Cancel attempts SIGTERM, then SIGKILL after a grace period, against the run's process group. Cancellation errors and races with completion are not yet reconciled through monotonic compare-and-set transitions.
- Pause/resume sends SIGSTOP/SIGCONT, but persistence failures and missing process groups can leave the database state inconsistent with the process.
- Crash restart attempts to probe and re-attach unfinished runs. The backend does not yet provide restart-safe multi-instance ownership or proof that a terminal state cannot be overwritten in every race. There is no cross-instance lease: two servers pointed at the same database can both adopt the same run after a restart.
Do not present these controls as crash-consistent or compliance-grade; treat run records written under crash/restart races as provisional until verified against the workspace files.
Production Deployment#
Two pieces complete a production serve: a supervised service and — when users reach the server over a network — a reverse proxy in front of it.
systemd service#
# /etc/systemd/system/oxo-flow.service
[Unit]
Description=oxo-flow web server
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=oxo
Group=oxo
WorkingDirectory=/opt/oxo-flow
EnvironmentFile=-/etc/oxo-flow/server.env
ExecStart=/usr/local/bin/oxo-flow serve
Restart=on-failure
RestartSec=2
TimeoutStopSec=15
[Install]
WantedBy=multi-user.target
Keep secrets in the EnvironmentFile (chmod 600), never inline in the
unit:
# /etc/oxo-flow/server.env
OXO_FLOW_MODE=team
OXO_FLOW_HOST=127.0.0.1
OXO_FLOW_ADMIN_PASSWORD=…
DEEPSEEK_API_KEY=…
A stop (systemctl stop oxo-flow) delivers SIGTERM; the server drains
connections and exits. Runs in flight are not force-killed. Cancel them from
the UI before a planned stop and verify the terminal state; restart recovery
is not yet qualified for unattended production use (see
Run Control Status).
nginx reverse proxy#
Behind a reverse proxy every request arrives at oxo-flow with the
proxy's own IP as the transport peer — the rate limiter would key all
clients into one shared bucket (a single abuser locks everyone out).
Set OXO_FLOW_TRUSTED_PROXY=1 so the limiter keys on the forwarded
X-Forwarded-For / X-Real-IP headers instead. Only enable this when
the proxy in front overwrites those headers on every request (both
recipes below do, via X-Real-IP $remote_addr); if clients can reach
the server without passing through it, a caller could spoof the header
and bypass the limiter.
server {
listen 80;
server_name flows.lab.example.edu;
location /oxo-flow/ {
proxy_pass http://127.0.0.1:8080; # no trailing slash: keep the mount prefix
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# Server-Sent Events (/api/events) require unbuffered streaming.
proxy_buffering off;
proxy_read_timeout 300s;
}
}
Start the app with the matching mount point and the trusted-proxy flag (the headers the recipe sets above are what the limiter then reads):
For TLS-terminating proxies (the common public deployment), also declare
the public origin so minted share URLs tell the truth about the transport
— by default they carry oxo+http:// with the local bind
host/port, which is unreachable from outside the proxy:
# /etc/oxo-flow/server.env (append) — public HTTPS entry point
OXO_FLOW_PUBLIC_URL=https://flows.lab.example.edu/oxo-flow
Share URLs then become oxo+https://flows.lab.example.edu/oxo-flow/share/<token>
(see Collaboration); the import API accepts both
schemes, so previously minted links keep working.
Performance#
| Mode | Startup | Memory (idle) | Memory (100 pipelines) |
|---|---|---|---|
| Personal | <0.1s | ~30MB | ~150MB |
| Team | <0.3s | ~40MB | ~200MB |
| HPC | <0.3s | ~40MB | ~200MB |
These are indicative figures, not benchmark results — measure on your own hardware before capacity planning.
Deployment Smoke Tests#
scripts/deploy-smoke.sh is the repeatable acceptance suite for the
deployment shapes above — every scenario runs in an isolated temp dir,
asserts against the live server, and cleans up after itself:
| Scenario | What it verifies |
|---|---|
| 1. Personal mode | API + health + SPA on a source build |
| 2. Standalone web binary | the release-style binary serves independently |
| 3. Sub-path mount | API under --base-path, root excluded, SPA <base> injection, assets |
| 4. Platform config file | port/base_path defaults + [[clusters]] seeding |
| 5. Team mode | 401 without a session; env-credential login → session → authenticated access |
| 6. HPC mode | scheduler endpoint responds with structured status |
| 7. Desktop bundle | the .app serves the SPA self-contained (OXO_APP=…) |
# Against the local debug build
scripts/deploy-smoke.sh
# Against a release build / packaged app
OXO_BIN=target/release/oxo-flow \
OXO_APP=target/release/bundle/osx/oxo-flow.app \
scripts/deploy-smoke.sh
Run it on every machine you deploy to — it exits non-zero on any failed assertion, so it doubles as a CI gate for packaging.