Skip to content

Deployment Modes#

oxo-flow supports three deployment modes from a single binary. Choose the mode that fits your team size and infrastructure.

Quick Reference#

# Personal workstation (default)
oxo-flow serve

# Team server
oxo-flow serve --mode team

# HPC submit panel
oxo-flow serve --mode hpc

Mode 1: Personal (Default)#

Use when: You're the only user, working on your own machine.

Setting Value
Network 127.0.0.1:8080 (localhost only)
Database SQLite file (oxo-flow.db) — override with DATABASE_URL
Auth None (single user) — management endpoints (Users, Audit) follow the same localhost trust model
Workspace workspace/users/local_user/runs/<run_id>
# Start
oxo-flow serve

# With custom port
oxo-flow serve -p 9090

# Open browser
open http://localhost:8080

Mode 2: Team#

Use when: Multiple users share a server or cloud instance.

Setting Value
Network 127.0.0.1:8080 by default — pass --host 0.0.0.0 to bind all interfaces
Database SQLite (default) or PostgreSQL — see Database Selection
Run execution Backend-routed code exists for SQLite and PostgreSQL; schema bootstrap + core CRUD are CI-smoke-verified on live PostgreSQL, while concurrency/crash-recovery/multi-instance remain unvalidated. Team mode has no OS-level tenant execution boundary, so expose it only to fully trusted workflow authors
Auth Password env vars (OXO_FLOW_ADMIN_PASSWORD / OXO_FLOW_USER_PASSWORD / OXO_FLOW_VIEWER_PASSWORD) + optional ORCID/GitHub OAuth
Workspace workspace/users/<username>/runs/<run_id>
# Start with SQLite (default) and bind to all interfaces
oxo-flow serve --mode team --host 0.0.0.0

# Set auth credentials
export OXO_FLOW_ADMIN_PASSWORD=...
export OXO_FLOW_USER_PASSWORD=...
export OXO_FLOW_VIEWER_PASSWORD=...
export ORCID_CLIENT_ID=...
export ORCID_CLIENT_SECRET=...
# Test the ORCID flow against sandbox.orcid.org instead of production (optional)
# export ORCID_SANDBOX=1
export GITHUB_CLIENT_ID=...
export GITHUB_CLIENT_SECRET=...

oxo-flow serve --mode team

Authentication: role-based password authentication comes from the OXO_FLOW_*_PASSWORD environment variables (with a dev-mode fallback when OXO_FLOW_DEV_MODE=1). ORCID and GitHub OAuth are available as separate login endpoints when their client credentials are configured — there is no automatic fallback chain between mechanisms.

User management: beyond the env-var credentials, accounts created on the Users page (or POST /api/users) carry a bcrypt-hashed password in the database and sign in through the same /api/auth/login endpoint. State-changing requests write an audit entry visible on the Audit page or through GET /api/audit, with the acting user and recorded outcome. The entry's durability is explicit per action (WEB-15 / issue #1140):

  • Security-relevant actions — authentication attempts, user/role changes, machine-key changes, run creation and lifecycle control, pipeline deletion and share grants, AI provider key changes, cluster connection secrets, webhook configuration, and license upload — fail closed: if the row cannot be written the request answers 500 AUDIT_WRITE_FAILED instead of acknowledging the action.
  • Informational mutations — AI generation, parsing/validation, DAG edits, chat, and pipeline create/update/fork — log a warning and still answer normally if the write fails.

The audit row is written after the handler, not in the same transaction, so a REQUIRED-audit 500 means the action may already have been applied — verify the affected resource before retrying, since a blind retry can duplicate the effect. Successful sign-ins are a handler-owned REQUIRED row (the middleware only sees the pre-auth request and would misattribute it); the session is persisted before the audit write, so a login 500 means a session token may already have been minted. For every mutation the middleware handles, the structured event stream emits an audit event even when the database insert fails. Do not treat a missing entry as proof that no action occurred; check the structured event stream as well. The trail spans all users and is admin-only outside personal mode.

Workspace isolation:

workspace/
└── users/
    └── <username>/
        └── runs/<run_id>/

Mode 3: HPC#

Use when: The web UI is a submit panel; actual execution happens on a cluster.

Setting Value
Network 127.0.0.1:8080 by default — pass --host 0.0.0.0 to bind all interfaces
Database SQLite or PostgreSQL — see Database Selection
Auth Same as Team mode
Executor SLURM / PBS / LSF / SGE
Resources Scheduler-managed
oxo-flow serve --mode hpc

Connecting the app to a cluster (the deployment model):

  • App anywhere, cluster elsewhere. The web UI can run on your workstation, a lab portal server, or any cloud VM — it does not need to live on the cluster. Cluster execution happens where the CLI runs: SSH into the login node (or run the CLI there directly) and use oxo-flow cluster submit / oxo-flow run --profile cluster to dispatch work through SLURM/PBS/LSF/SGE.
  • App on the login node (--mode hpc). When the web server itself runs on a cluster login node, hpc mode detects the scheduler at startup (sinfo/qstat/pbsnodes/bjobs) and exposes GET /api/hpc with live queue/partition status in the UI.
  • App mounted at a sub-path. On shared portal servers the app can be mounted under any prefix with --base-path /oxo-flow — the router, the SPA, and all asset URLs follow the mount point automatically (the SPA index gets a <base> tag injected at serve time), so it composes with reverse proxies without a rebuild.
# App on the login node, mounted under /oxo-flow
oxo-flow serve --mode hpc --base-path /oxo-flow

# App on your workstation, cluster execution via the CLI on the login node
ssh login-node
oxo-flow cluster submit workflow.oxoflow -b slurm

Switching Modes#

# Environment variable (alternative to --mode flag)
export OXO_FLOW_MODE=team
oxo-flow serve

Database Selection (DATABASE_URL)#

All serving entry points — oxo-flow serve, the standalone oxo-flow-web binary, and the desktop app — share one database-selection rule: DATABASE_URL starting with postgres:// (or postgresql://) selects the PostgreSQL backend; anything else (including unset, which defaults to sqlite://oxo-flow.db in the working directory) stays SQLite.

# PostgreSQL (storage-backed web metadata and backend-routed run records)
export DATABASE_URL=postgres://oxo:secret@db.lab.example.edu:5432/oxo_flow
oxo-flow serve --mode team --host 0.0.0.0

Two constraints apply:

  • Build feature: PostgreSQL support is a compile-time feature. The published Docker image is built WITHOUT it — rebuild with cargo build --features postgres (or the equivalent in your own image). On a build without the feature, a postgres:// DATABASE_URL aborts startup with a clear error instead of silently falling back.
  • Run execution: the StorageBackend trait has PostgreSQL code paths for run-record persistence. CI runs a live-PostgreSQL smoke test (a services: postgres container) that verifies the schema bootstrap and the core StorageBackend CRUD surface — users, runs, run nodes, audit logs, webhook config, chat sessions — against a real server. That does NOT extend to production parity: concurrency, crash recovery, and multi-instance behavior remain unvalidated, so test them before relying on PostgreSQL for execution metadata. The runs gate is backend-agnostic in behavior — any initialized storage backend passes — and answers 503 RUNS_REQUIRE_STORAGE_BACKEND only when no backend was initialized at all.
  • Tenant safety: team/HPC authorization is not an OS or container execution boundary. A normal workflow can run shell commands under the server account and may access server environment secrets. Do not invite untrusted tenants until the workspace isolation described in Multi-Tenancy is supplemented with your own OS-level isolation (dedicated service user, container, or similar).

Startup Verification#

On startup, oxo-flow prints the version banner, the serve mode and address, and tracing log lines:

oxo-flow v0.32.0 — Rust-native bioinformatics pipeline engine
Serve: Starting oxo-flow web server in personal mode on 127.0.0.1:8080
 INFO Logging initialized at logs
 INFO AI registry initialized: provider=disabled, model=default, enabled=false
 INFO Building router for mode: personal
 INFO Starting oxo-flow web server in personal mode on 127.0.0.1:8080

Log lines carry timestamps in a terminal, and the mode, host, and port follow --mode/--host/-p.

The banner line is interactive-terminal decoration: when stderr is redirected (oxo-flow serve > serve.log 2>&1, nohup, service managers), it is omitted — the log lines alone identify the running version.

Platform Configuration#

Server-tier settings, AI defaults, and SSH cluster definitions can live in a platform config file — the lowest-precedence layer (CLI flag > env var > config file > built-in default). The file is looked up at OXO_FLOW_CONFIG, then oxo-flow.web.toml in the working directory, then ~/.config/oxo-flow/web.toml.

[server]
mode = "team"            # personal | team | hpc
host = "0.0.0.0"
port = 8080
base_path = "/oxoflow"   # sub-path mounts behind a reverse proxy

[ai]
provider = "deepseek"    # anthropic | openai | deepseek | ollama | disabled
api_url = "https://api.deepseek.com"
model = "deepseek-v4-pro"
api_key_env = "DEEPSEEK_API_KEY"   # secrets stay in env vars, never inline

[[clusters]]
id = "lab-slurm"                  # stable key (import is idempotent)
name = "Lab SLURM cluster"
ssh_host = "login.lab.example.edu"
ssh_port = 22
ssh_user = "bioinf"
ssh_key = "~/.ssh/id_ed25519"
scheduler = "slurm"               # auto | slurm | pbs | lsf | sge
remote_dir = "~/oxo-flow-jobs"

The same choices are editable at runtime on the Clusters & Servers page: add an SSH endpoint and press Probe — the server performs a real SSH round-trip (BatchMode, 8 s timeout), reports the remote hostname and detects the scheduler (slurm/pbs/lsf/sge) with its version. Unknown fields in the config file are rejected loudly, not silently ignored.

Multi-Tenancy#

Team/HPC modes scope every resource to the acting user:

  • Ownership: runs and pipelines are owned by the session's user; foreign resources 404 (existence never leaks). Admins see and control everything; viewers/guests get read access to workspace-visible pipelines only.
  • Anonymous surface: /api/system, /api/metrics, /api/ai/test, and /api/hpc require authentication; GET /api/ai/config stays public; /api/events requires a one-time ?ticket= from POST /api/events/ticket (EventSource cannot set headers) and streams only the subscriber's runs; /api/share/{token} is public by design (the token is the credential).
  • Env-password logins auto-provision a real users row (previously any username shared the default pseudo-user's identity).
  • API keys (X-API-Key: oxo_…) are first-class machine credentials with the owner's exact permissions; stored hashed, instantly revocable.
  • Shared infrastructure: the server AI provider, webhook endpoint, and SSH cluster connections are admin-managed outside personal mode.

Server Data Layout (OXO_FLOW_WORKSPACE)#

The web server keeps all user-scoped data under one workspace root, OXO_FLOW_WORKSPACE (default: a relative workspace/ directory under the server process's cwd):

  • users/<username>/inputs/ — files uploaded via POST /api/files; mirrored into each new run's workdir at Create Run (POST /api/runs, which stages include_files straight into the run dir without touching inputs/)
  • users/<username>/runs/<run_id>/ — each run's own immutable workdir (the spawned CLI runs here; the workflow snapshot, checkpoint, logs, outputs, and envs live under it). Every create request — including for a saved pipeline — gets a fresh directory, so a later run can never overwrite an earlier run's artifacts (issue #1130 / audit RB-02); only retry/resume-checkpoint continue their predecessor's directory.
  • users/<username>/pipelines/<pipeline_id>/ — legacy pre-#1130 run workdir. New creates no longer use it, but retrying or resuming a run created before the upgrade still writes into that run's recorded workdir here. Leave the tree in place while any pre-upgrade run may be retried; archive it only once those runs are no longer needed

Usernames and run ids are traversal-validated before they are joined onto the root. In container or systemd deployments, point OXO_FLOW_WORKSPACE at a persistent volume — a server started from a different cwd otherwise silently loses sight of previous runs. The workspace root is server-tier configuration only; it has no per-user override.

Per-User AI Credentials#

Every user-facing AI call (chat, translate, explain, interpret, optimize) resolves the acting user's own saved provider (PUT /api/ai/config/user) before falling back to the shared runtime:

  • resolution order per call: the user's row (unless disabled) → the server-level provider (env / admin config) → default
  • a non-admin's saved key never reconfigures the shared runtime — it is cached per user and invalidated on every config write
  • a per-user row without an api key carries an empty key: their calls fail loudly instead of silently borrowing the server's key (no shared-secret leakage in either direction)
  • server-level writes (POST /api/ai/config, PUT /api/ai/config/server) stay admin-only outside personal mode and invalidate all per-user fallbacks

Testing & Development#

OXO_FLOW_DISABLE_RATE_LIMIT=1 disables the per-IP rate limiter — the browser e2e suite (and any local automation) legitimately exceeds the 100 req/min budget and must set it. Never set it on a team/hpc production server: the limiter is the brute-force protection.

Run Control Status (all modes)#

Run control uses process signaling, but lifecycle truth remains prototype-grade:

  • Cancel attempts SIGTERM, then SIGKILL after a grace period, against the run's process group. Cancellation errors and races with completion are not yet reconciled through monotonic compare-and-set transitions.
  • Pause/resume sends SIGSTOP/SIGCONT, but persistence failures and missing process groups can leave the database state inconsistent with the process.
  • Crash restart attempts to probe and re-attach unfinished runs. The backend does not yet provide restart-safe multi-instance ownership or proof that a terminal state cannot be overwritten in every race. There is no cross-instance lease: two servers pointed at the same database can both adopt the same run after a restart.

Do not present these controls as crash-consistent or compliance-grade; treat run records written under crash/restart races as provisional until verified against the workspace files.

Production Deployment#

Two pieces complete a production serve: a supervised service and — when users reach the server over a network — a reverse proxy in front of it.

systemd service#

# /etc/systemd/system/oxo-flow.service
[Unit]
Description=oxo-flow web server
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=oxo
Group=oxo
WorkingDirectory=/opt/oxo-flow
EnvironmentFile=-/etc/oxo-flow/server.env
ExecStart=/usr/local/bin/oxo-flow serve
Restart=on-failure
RestartSec=2
TimeoutStopSec=15

[Install]
WantedBy=multi-user.target

Keep secrets in the EnvironmentFile (chmod 600), never inline in the unit:

# /etc/oxo-flow/server.env
OXO_FLOW_MODE=team
OXO_FLOW_HOST=127.0.0.1
OXO_FLOW_ADMIN_PASSWORD=…
DEEPSEEK_API_KEY=…

A stop (systemctl stop oxo-flow) delivers SIGTERM; the server drains connections and exits. Runs in flight are not force-killed. Cancel them from the UI before a planned stop and verify the terminal state; restart recovery is not yet qualified for unattended production use (see Run Control Status).

nginx reverse proxy#

Behind a reverse proxy every request arrives at oxo-flow with the proxy's own IP as the transport peer — the rate limiter would key all clients into one shared bucket (a single abuser locks everyone out). Set OXO_FLOW_TRUSTED_PROXY=1 so the limiter keys on the forwarded X-Forwarded-For / X-Real-IP headers instead. Only enable this when the proxy in front overwrites those headers on every request (both recipes below do, via X-Real-IP $remote_addr); if clients can reach the server without passing through it, a caller could spoof the header and bypass the limiter.

server {
    listen 80;
    server_name flows.lab.example.edu;

    location /oxo-flow/ {
        proxy_pass http://127.0.0.1:8080;   # no trailing slash: keep the mount prefix
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # Server-Sent Events (/api/events) require unbuffered streaming.
        proxy_buffering off;
        proxy_read_timeout 300s;
    }
}

Start the app with the matching mount point and the trusted-proxy flag (the headers the recipe sets above are what the limiter then reads):

oxo-flow serve --mode team --base-path /oxo-flow
# /etc/oxo-flow/server.env (append)
OXO_FLOW_TRUSTED_PROXY=1

For TLS-terminating proxies (the common public deployment), also declare the public origin so minted share URLs tell the truth about the transport — by default they carry oxo+http:// with the local bind host/port, which is unreachable from outside the proxy:

# /etc/oxo-flow/server.env (append) — public HTTPS entry point
OXO_FLOW_PUBLIC_URL=https://flows.lab.example.edu/oxo-flow

Share URLs then become oxo+https://flows.lab.example.edu/oxo-flow/share/<token> (see Collaboration); the import API accepts both schemes, so previously minted links keep working.

Performance#

Mode Startup Memory (idle) Memory (100 pipelines)
Personal <0.1s ~30MB ~150MB
Team <0.3s ~40MB ~200MB
HPC <0.3s ~40MB ~200MB

These are indicative figures, not benchmark results — measure on your own hardware before capacity planning.

Deployment Smoke Tests#

scripts/deploy-smoke.sh is the repeatable acceptance suite for the deployment shapes above — every scenario runs in an isolated temp dir, asserts against the live server, and cleans up after itself:

Scenario What it verifies
1. Personal mode API + health + SPA on a source build
2. Standalone web binary the release-style binary serves independently
3. Sub-path mount API under --base-path, root excluded, SPA <base> injection, assets
4. Platform config file port/base_path defaults + [[clusters]] seeding
5. Team mode 401 without a session; env-credential login → session → authenticated access
6. HPC mode scheduler endpoint responds with structured status
7. Desktop bundle the .app serves the SPA self-contained (OXO_APP=…)
# Against the local debug build
scripts/deploy-smoke.sh

# Against a release build / packaged app
OXO_BIN=target/release/oxo-flow \
OXO_APP=target/release/bundle/osx/oxo-flow.app \
scripts/deploy-smoke.sh

Run it on every machine you deploy to — it exits non-zero on any failed assertion, so it doubles as a CI gate for packaging.