Web System Architecture#
oxo-flow-web is the web control plane for oxo-flow — not a thin wrapper around the CLI.
Design Principles#
1. AI-Native API Design#
All endpoints are designed to be consumable by both AI agents and browsers:
- Unified structured responses
- Long-running operations streamed via SSE
- Self-discoverable via
GET /api/openapi.json - Errors carry
code+message, plus optionaldetailandsuggestionfields
2. Intent-First Pipeline Authoring#
Users describe what they want ("run standard RNA-seq differential expression on this cohort"), and the system generates a validated pipeline. The TOML format becomes the compilation target, not the authoring surface.
3. DAG as the Primary Interaction Surface#
An interactive visual graph where users can drag nodes, connect inputs to outputs, and inspect environment/resource configuration. The graph is the workspace.
4. Results as Queryable Data#
Pipeline outputs are not just files — they are structured data that can be queried, compared, and visualized via the API.
5. Shareable, Forkable, Citable Pipelines#
A pipeline is a URL, not a file. Share it, fork it. Each run has a permanent provenance URL.
6. HPC/Cloud as a Detail#
The execution backend is a dropdown — local, HPC queue, Docker, cloud batch. Everything else (workspace setup, file staging, environment resolution) happens transparently.
Architecture Overview#
The web crate follows a domain-driven modular monolith pattern. Each domain has:
types.rs— request/response structsservice.rs— pure logic (zero HTTP dependency)handlers.rs— HTTP → service adapters
+-----------------------------------------------+
| Web Client (SPA) |
| Interactive DAG Editor / Dashboard |
+------------------+----------------------------+
| HTTP/SSE
+------------------v----------------------------+
| oxo-flow-web (Axum Server) |
| |
| +---------+ +---------+ +----------------+ |
| | Auth | |License | | Observability | |
| | OAuth2 | |Headers | | Health/Metrics | |
| +---------+ +---------+ +----------------+ |
| +---------+ +---------+ +----------------+ |
| |Workflow | |Execution| | AI/Translate | |
| |Pipeline | |Diagnose | | Explain/Interp | |
| +---------+ +---------+ +----------------+ |
| +---------+ +---------+ +----------------+ |
| |Collabor-| | Data | | HPC | |
| |ation | | Discovery| | Scheduler | |
| +---------+ +---------+ +----------------+ |
| +-------------------------------------------+|
| | Middleware Stack | |
| | (outermost first) Compression -> Trace ->| |
| | CORS -> RateLimit -> SecurityHeaders -> | |
| | LicenseHeader | |
| +-------------------------------------------+|
| +-------------------------------------------+|
| | StorageBackend trait (SQLite + PostgreSQL)| |
| +-------------------------------------------+|
+------------------+----------------------------+
| API calls
+------------------v----------------------------+
| oxo-flow-core (Engine) |
| DAG . Executor . Environment . Wildcards |
+-----------------------------------------------+
Middleware & Runtime Behavior#
Applied outermost-first in server.rs::build_router:
- Compression — tower-http
CompressionLayer(brotli + gzip). The default predicate skipstext/event-stream, so the SSE endpoints (/api/events,/api/ai/translate/stream) stream uncompressed. - Tracing —
TraceLayeremits an INFOhttp.requestspan plus a completion log (status, latency) per request;/api/healthruns at DEBUG because the dashboard polls it every few seconds. - CORS → Rate limit → Security headers → License header — the rate
limiter sits inside the
Extensionlayer so it can see the shared limiter state.
Static serving and caching:
/assets/*holds Vite content-hashed files, served withCache-Control: public, max-age=31536000, immutable;index.htmlstaysno-cacheso new deploys are picked up immediately.- The SPA
index.html(with<base>/base-path injection) is computed once per process and cached in memory, as is the serialized OpenAPI document behindGET /api/openapi.json.
Async-runtime hygiene:
- Run-log reads (logs, diagnostics, preview, finalize) and the log tailer
use
tokio::fs;metrics.jsonlappends are O(1) with a truncate-rewrite only past the 2000-line cap. - bcrypt password hash/verify runs via
tokio::task::block_in_placeso it does not block the async worker. GET /api/metricsreads a shared persistentsysinfo::Systemwith targeted CPU/memory refreshes (real CPU deltas; memory fields are true MB, matching their_mbnames).
Rate limiting and SSE:
- The rate limiter is a sliding window (default 100 requests / 60 s) keyed
by client IP: the transport peer address by default (spoofable
X-Forwarded-For/X-Real-IPheaders are ignored unlessOXO_FLOW_TRUSTED_PROXYmarks the deployment as behind a controlled reverse proxy, in which case the forwarded headers are honoured). Over-limit requests get429with the standard structured error body ({"code":"RATE_LIMITED", ...}) and aRetry-Afterheader; keys whose timestamps have all expired are evicted opportunistically (every 1024 checks, no background task). /api/eventsrelies solely on axum's 15-secondKeepAlivecomment ping. If a subscriber falls behind the 100-slot broadcast buffer, the server emits a synthetic{"type":"lagged","time":"…","data":{"missed":N}}event so the client can refetch instead of silently missing run state transitions.
Share URL truth:
- The server has no native TLS, so minted share URLs use
oxo+http://with the actual bind host/port (plus the--base-pathprefix) by default — a hardcodedoxo+https://advertised TLS the process never spoke. IPv6 host literals are bracketed so the URL stays parseable. OXO_FLOW_PUBLIC_URL(e.g.https://flows.example.com) lets a reverse-proxy deployment declare its public origin; scheme and authority of minted URLs then come from that override. A value that also carries the mount path (https://flows.example.com/oxo-flow, the sub-path deploy recipe) replaces the local--base-pathprefix instead of being appended to it; a scheme-less value still mintsoxo+http://but logs a warning; an empty value warns and is treated as unset; a value without any host (including port-only:8080) is ignored with a warning and a local URL is minted instead; IPv6 literals on the override path are bracketed. Import accepts both schemes, so olderoxo+https://links keep working.
Server process environment:
OXO_FLOW_BIN— explicit path to theoxo-flowCLI binary the server spawns for run execution. It is the first of the binary search steps (env override →CARGO_BIN_EXE_oxo-flow→ sibling of the running executable → parent directory →PATH), so a stalePATHentry can no longer silently win over a deliberate pin; the resolved path is logged at startup (from which the source is inferable) and a major/minor mismatch between the server and the spawned CLI is warned.OXO_FLOW_WORKSPACE— root directory for all user-scoped server data (users/<u>/inputs/,users/<u>/runs/<run_id>/; a legacyusers/<u>/pipelines/<pipeline_id>/tree from pre-#1130 releases is no longer written). Defaults to a relativeworkspace/under the process cwd; container deployments should point it at a persistent volume. Path components are traversal-validated before use.
Run workdir lifecycle#
Every run created through POST /api/runs owns users/<u>/runs/<run_id>/
for its whole life. The workflow snapshot, staged include_files, uploaded
inputs, .oxo-flow/checkpoint.json, logs, and outputs are written there and
nowhere else; the directory name is the run id, so no other run can address
it. retry and resume-checkpoint are explicit continuations of a
predecessor's lineage and deliberately write into that predecessor's tree;
they are not new run history. A saved pipeline's pipeline_id
is an association (permissions, listing, admission ordering), never a
workdir. This is what makes a run's recorded checksums and run history
immutable (issue #1130 / audit RB-02). Admission is a single guarded
INSERT ... SELECT ... WHERE NOT EXISTS ... ON CONFLICT DO NOTHING keyed on
pipeline_id for saved pipelines and on workdir for ad-hoc runs, backed by
partial unique indexes (idx_runs_live_pipeline, idx_runs_live_workdir) on
both SQLite and PostgreSQL. The indexes are the guarantee PostgreSQL needs:
under READ COMMITTED two concurrent transactions can both pass NOT EXISTS,
and only ON CONFLICT decides the loser (zero rows affected → the documented
409 RUN_NOT_TERMINAL). The saved-pipeline slot is global to the pipeline,
not per user, so a workspace-shared pipeline admits one live run across all
its readers. retry and resume-checkpoint are the only paths that
deliberately continue a predecessor's workdir.
Nothing reclaims those trees automatically: a long-lived server accumulates one
users/<u>/runs/<run_id>/ directory per run forever. The manual sweeper is
oxo-flow runs gc, which deletes only terminal-run
trees past an age gate, never a live run, never a tree a later retry/resume
still points at, and never a tree carrying a declared protected output — the
shared workdir string across run rows is exactly the lineage signal it keys
on.
UI Test Coverage#
The SPA's published surfaces are exercised by Playwright specs in
frontend/e2e/ against the real web server. Individual specs stub the
specific endpoints they need to force — AI provider/chat, pipeline
validation, auth, and file download — while the core-flow, mobile, WCAG and
Metro specs drive the real server. make frontend-test runs the suite
locally; the CI e2e job runs the same npx playwright test with no AI
provider environment (so the AI-disabled Dashboard/Settings states are the
ones gated) after asserting the served bundle is the freshly built artifact
(issue #555).
| Surface | Spec | Asserted contract |
|---|---|---|
| Browser E2E: author → validate → run → monitor | core-flows.spec.ts |
Editor renders TOML + DAG, the validation badge reflects a real POST /api/pipelines/validate, Save creates a pipeline, and the Stage 4/5 case creates a real run, then verifies provenance/checksums and the output preview in the monitor/report. |
| Mobile viewport | mobile-viewport.spec.ts |
At 390×844, routed content does not overflow: .main-content.scrollWidth <= clientWidth + 1 on the dashboard, pipelines, editor, runs, monitor and settings routes (the document-level check alone is vacuous because body/.main-content clip overflow-x). A negative-control test injects a 2000px child and proves the detector reports it. The navigation drawer reaches the editor and an enabled Run button opens the run dialog with its close control inside the viewport; the React Flow canvas does not widen the routed content. |
| WCAG / accessibility | accessibility.spec.ts |
axe-core scans the six primary routes in both color themes plus three mobile-viewport scans (drawer closed in both themes and open) with the WCAG 2.0/2.1 A + AA tag set; any violation fails the spec. No axe rule is disabled — the findings this gate surfaced were fixed in the UI. CI runs with no AI provider, so the AI-disabled dashboard/settings states are gated; both those and the configured variants pass. The scanned routes and the out-of-scope canvas surface are documented in the spec header. |
| Metro frame-time budget | metro-performance.spec.ts |
A 6-rule workflow is switched Rule → Process → Module (a Sugiyama re-layout per switch) and the interaction-to-next-paint latency is measured with performance.now plus two requestAnimationFrames after the click. Station counts prove the switch regrouped (Rule = 6, Process = 1); worst sample < 750 ms, median < 300 ms. |
| Keyboard graph navigation | metro-canvas.spec.ts |
Metro station keyboard access is owned by issue #1148 / ledger UX-06; that spec ships with the keyboard-access change. It is not duplicated here. |
| Desktop (wry/tao) lifecycle | scripts/desktop-smoke.sh (CI desktop job, make desktop-smoke) |
Build + launch against an isolated HOME, the embedded server creates its lock + SQLite database, a second instance is refused with the documented message, and SIGTERM exits 0. See docs/guide/src/how-to/desktop-app.md → Verification. |
Deliberately not covered by this suite: visual pixel rendering, real-device mobile gestures, screen-reader output beyond what axe-core's rule set evaluates, and Windows desktop packaging (the release pipeline does not build a Windows bundle).
Domain-Driven Module Structure#
| Domain | Path | Responsibility |
|---|---|---|
| workflow | domains/workflow/ |
Pipeline parse, validate, prepare, DAG, format, lint, stats, diff, export, search, data discovery, plugin validation |
| execution | domains/execution/ |
Run create/status/cancel/retry, diagnostics engine (31 error patterns), sandbox workspace; runs delegate to the oxo-flow CLI subprocess, which owns checkpointing and invalidation |
| ai | domains/ai/ |
AI translate, explain, interpret, optimize; generation runs the shared agent + orchestrator harness (see Architecture → AI Subsystem) over the provider fallback chain |
| auth | domains/auth/ |
Login, session management, ORCID/GitHub OAuth2, RBAC |
| chat | domains/chat/ |
Real-time AI copilot chat and streaming; generation uses the same harness as translate, plus read-only run-diagnosis tools |
| clusters | domains/clusters/ |
Remote cluster submission over SSH, job polling, result pull-back |
| collaboration | domains/collaboration/ |
Fork, diff, share, import pipelines |
| dag | domains/dag/ |
DAG-specific types and operations |
| observability | domains/observability/ |
Health check, system info, runtime metrics, structured logging (3-layer), audit, SSE |
| infra/db | infra/db/ |
StorageBackend trait with SQLite and PostgreSQL implementations; migrated metadata surfaces and the startup PostgreSQL capability matrix |
| infra/license | infra/license.rs |
License notice text, banner, X-OxoFlow-License header middleware (the web UI footer label lives in the SPA, frontend/src/components/Layout.tsx) |
| infra/sse | src/sse.rs |
Real-time SSE broadcast channel for execution events |
| hpc | core cluster.rs::generate_submit_script (crates/oxo-flow-core/src/cluster.rs) |
Slurm submit-script generation; scheduler detection stays in the web crate |
Durable runtime state#
Quota state — per-user counters, per-run resource reservations, and the
limits row — lives in the storage backend (quota_usage,
quota_reservations, quota_config), not in process memory. Admission is
one atomic compare-and-swap plus reservation insert, so concurrent requests
serialize on the database, restarts preserve usage, and multiple server
instances sharing a database enforce the same counters and limits (RB-09 /
issue #1133). Locally executed runs on Linux additionally enforce the
workflow's declared memory with RLIMIT_DATA on the spawned CLI process
tree; other platforms and remote (cluster) runs remain advisory — see
Quota → Declared-resource enforcement.
API Namespace#
/api
├── /health # Health check (with license, mode, component status)
├── /system # System info
├── /metrics # Runtime metrics (CPU, memory, active runs)
├── /openapi.json # OpenAPI 3.1 spec
├── /events # SSE event stream (real-time execution updates)
├── /audit # Audit logs (structured, with result field)
├── /hpc # HPC scheduler status (SLURM, PBS, etc.) — hpc mode only
│
├── /auth
│ ├── /login # Login (username/password)
│ ├── /me # Current session info
│ └── /oauth/... # OAuth2 authorize / callback
│
├── /license
│ ├── / # License status (type, validity, contact)
│ └── /upload # Upload commercial license file
│
├── /users # User list/create/delete (admin)
│
├── /pipelines (new v0.8 API — replaces /workflows)
│ ├── /parse # Parse TOML → structured pipeline
│ ├── /validate # Validate pipeline DAG
│ ├── /prepare # Prepare (expand wildcards, resolve envs)
│ ├── /dag # Build DAG as JSON
│ ├── /format # Canonical TOML formatting
│ ├── /lint # Lint pipeline (valid + errors/lint findings)
│ ├── /stats # Aggregate pipeline statistics
│ ├── /diff # Diff two pipelines (by TOML content)
│ ├── /export # Export Docker/Singularity packaging
│ ├── /search # Search pipelines by name, tags, content
│ ├── / # GET: list pipelines; POST: save pipeline
│ ├── /{id} # GET/PUT/DELETE pipeline by ID
│ ├── /{id}/fork # Fork into user workspace (v0.8 collab)
│ └── /{id}/share # Share pipeline (v0.8 collab)
│
├── /pipelines/import # Import from oxo+http(s):// share URL (v0.8 collab)
│
├── /runs
│ ├── / # POST: create run; GET: list runs
│ ├── /{id} # Run detail with log tail
│ ├── /{id}/status # Real-time status (nodes, timeline, resources)
│ ├── /{id}/dag-status # DAG JSON + per-node live status
│ ├── /{id}/diagnostics # Diagnostic engine results (31 error patterns)
│ ├── /{id}/logs # Execution logs
│ ├── /{id}/results # Output files with sizes
│ ├── /{id}/retry # Smart retry (failed + downstream only)
│ ├── /{id}/cancel # Cancel running workflow
│ ├── /{id}/pause # Pause running workflow
│ └── /{id}/resume # Resume paused workflow
│
├── /data
│ ├── /analyze # Scan files → detect format, suggest pipeline
│ ├── /reference # Reference genome discovery
│ ├── /perceive # Data perception (AI Companion)
│ ├── /reference/status # Reference genome status
│ └── /samplesheet/parse # Parse a samplesheet
│
├── /templates
│ ├── / # GET: list templates; POST: create
│ └── /{id} # GET/DELETE template
│
├── /plugins
│ └── /validate # Validate plugin manifest + signature
│
├── /ai
│ ├── /translate # Natural language → validated .oxoflow (JSON)
│ ├── /translate/stream # Same, streamed over SSE
│ ├── /explain # Explain run failure + suggest fix
│ ├── /interpret # Interpret results with caveats
│ ├── /optimize # Optimize pipeline parameters
│ └── /config, /test # AI provider configuration
│
├── /chat # AI Companion: /send, /send/json, /sessions
│
└── /pipeline/{id} # DAG edit API: /command, /undo, /redo
GET /api/openapi.json on any running server.)
Structured Error Response#
All errors follow this format — detail and suggestion are optional and
only present when a handler supplies them (e.g. 429 bodies carry
detail: "retry in <N>s"):
{
"code": "AUTH_REQUIRED",
"message": "Authentication required in team/hpc mode",
"suggestion": "Login at POST /api/auth/login to obtain a session token"
}
Error Codes#
| Code | HTTP Status | Description |
|---|---|---|
MISSING_FIELD |
400 | Required request field missing (e.g. toml_content) |
DAG_EDIT_ERROR |
400 | DAG edit command rejected (unknown operation, missing fields) |
AUTH_REQUIRED |
401 | Authentication required |
AUTH_FAILED |
401 | Invalid login credentials |
TOKEN_EXPIRED |
401 | Session token missing or expired |
ACCESS_DENIED |
403 | Permission denied |
NOT_FOUND |
404 | Resource not found |
NO_UNDO / NO_REDO |
404 | DAG edit history exists but the requested direction is exhausted |
EDIT_HISTORY_UNAVAILABLE |
409 | No DAG edit history for this identity + pipeline (never edited, or lost to a server restart / LRU eviction); undo/redo fails closed |
EDIT_HISTORY_STALE |
409 | Supplied toml_content is not the tip of the recorded DAG edit history; nothing is applied |
OAUTH_INVALID_STATE |
400 | OAuth CSRF state missing, unknown, malformed, replayed, or past its 10-minute TTL |
OAUTH_STATE_MISMATCH |
400 | OAuth CSRF state was issued for a different provider, redirect URI, or browser (missing/mismatched binding cookie — issue #1197). A browser-binding mismatch does not disclose which component failed; provider/redirect mismatches name the expected value by design |
OAUTH_REDIRECT_NOT_ALLOWED |
400 | OAuth redirect target origin is not on the server-side allowlist |
RATE_LIMITED |
429 | Request rate exceeded |
AUDIT_WRITE_FAILED |
500 | A REQUIRED audit row for a security-relevant action could not be persisted; the request fails closed (WEB-15 / issue #1140) |
DB_ERROR |
500 | Internal database error |
Domain-specific codes (all 400) include PARSE_ERROR, VALIDATE_ERROR, PREPARE_ERROR, DAG_ERROR, LINT_ERROR, STATS_ERROR, DIFF_ERROR, EXPORT_ERROR, RUN_ERROR, RETRY_ERROR, DATA_ERROR, REF_ERROR, PLUGIN_ERROR, AI_TRANSLATE_ERROR, CHAT_ERROR, INVALID_URL.
List Responses#
List endpoints do not share a single uniform envelope:
GET /api/runsuses cursor pagination —{items, next_cursor, total}, wherenext_cursor: nullmeans the last page.- Most other list endpoints return bare JSON arrays capped at 100 items:
GET /api/pipelinesreturns at most 100 items (ordered by last update), andGET /api/templatesreturns all templates.
AI Agent Integration Guide#
Discovery#
End-to-End Workflow#
1. GET /api/health # Check server availability
2. POST /api/auth/login # Authenticate (team/hpc mode)
3. POST /api/ai/translate # [AI] "Run standard RNA-seq differential expression"
4. POST /api/runs # Execute the pipeline (POST the TOML content)
5. GET /api/events # SSE real-time progress (optional)
6. GET /api/runs/{id}/results # Get structured results
API Streaming#
AI pipeline generation supports SSE streaming via the dedicated streaming endpoint:
POST /api/ai/translate/stream
Accept: text/event-stream
event: progress
data: {"step": "intent", "message": "Intent received", "intent": "..."}
event: progress
data: {"step": "match", "message": "Matching templates...", "templates_count": 5}
event: progress
data: {"step": "generate", "message": "Generating pipeline via AI..."}
event: done
data: {"pipeline_id": "...", ...}
SPA Interaction Patterns#
The frontend (frontend/, React 19 + TypeScript) follows a few shared UX conventions so behavior stays consistent across pages:
- Collapsible sections: Settings sections, the Rule Inspector field groups (inputs/outputs, resources, advanced), and the run monitor's alert cards + Recent Events table fold behind clickable headers (
role="button",aria-expanded, Enter/Space toggles, rotating chevron). Sections default to open — the monitor's alert cards and events table are the exception, defaulting collapsed so a burst of alerts doesn't bury the live view. Collapsing only hides children; form state lives in the parent component, so no draft input is lost. - Styled confirmation dialogs: destructive actions (pipeline delete) open the shared
Modalcomponent instead ofwindow.confirm— the native dialog cannot be styled and blocks the event loop in the embedded desktop shell. The modal provides Escape/backdrop close, a focus trap, and focus restore. - Honest AI-disabled state: when no AI provider is configured, the Dashboard entry card renders a "configure AI" call-to-action linking to
/settings(with a note about server-side configuration), and the chat composer's editor hand-off CTA only renders outside the editor context. - Unified app navigation: the SPA has exactly one primary nav — a sectioned sidebar (
WORKSPACE/INTELLIGENCE/INFRASTRUCTURE/ADMINISTRATION) defined once inLayout.tsxwith per-item role gating; the header is deliberately nav-free. At ≥1280px the sidebar is collapsible (240px ↔ 72px icon rail, persisted per device in localStorage, Ctrl/Cmd+B toggles); 1024–1279px forces the icon rail; below 1024px the sidebar becomes a drawer opened from the header toggle — arole="dialog"with focus trap, focus moved in on open, Escape/scrim/route-change dismissal. Icon-rail tooltips show on hover and keyboard focus. Version and license render exactly once, in the page footer;/loginis a standalone centered card outside the app shell (same pattern as/share). - Polling-stable UI state: expansion/collapse choices on the run monitor are keyed by stable alert identifiers (
alert_id, falling back torule-prediction), so they survive the 5-second polling re-render. - Truthful environment labels on the monitor canvas: the run monitor's DAG canvas draws each rule with the environment the run actually wraps it in —
GET /api/runs/{id}/dag-statusresolves it from the run's own pipeline snapshot the way the run path does (the sameapply_defaultsfoldrun/dry-runmake:env_group→ the rule's inline environment →[defaults] environment, issue #1210), and the canvas renders that kind (conda,mamba,pixi,docker,singularity,venv,modules,system) instead of painting every nodesystem(issue #1149, audit UX-07). A node that no longer maps to a snapshot rule (an orphaned instance left by an edited workflow) is labelledunknown— the canvas never guessessystem. The label is per rule, so the same rule's instances share it; the Metro view's process grouping therefore splits stations by real environment. - End-to-end workflow lifecycle & Command Palette: the Command Center (
Dashboard.tsx) surfaces a 5-stage lifecycle architecture grid (1. Data & Samples→2. Author & AI→3. Validate & Lint→4. Execute & Guard→5. Provenance & Report) that links directly to the corresponding editor modals (/editor?modal=data,/editor?modal=engine) and monitor tabs. A global keyboard Command Palette (⌘K/Ctrl+K) inLayout.tsxprovides quick search across pages, lifecycle modals, and theme/locale toggles without adding header navigation links. The editor integrates sample-sheet cohort parsing (POST /api/data/samplesheet/parse), workspace file uploads (GET/POST /api/files), vendor MD5 delivery verification (POST /api/data/verify),[engine]disk-pressure & protected-tree configuration, canonical formatting (POST /api/pipelines/format), and best-practice linting (POST /api/pipelines/lint); the run launcher and monitor expose--provenance(GET /api/runs/{id}/provenance-verify),--allow-protected-drift(GET /api/runs/{id}/protected-drift),--arg KEY=VALUEoverrides, and output file preview/download (GET /api/runs/{id}/files).