Skip to content

Evaluate the AI Features#

oxo-flow's AI surfaces (tool lookup, template --ai workflow generation, and ai explain) are evaluated with a self-contained benchmark in the eval/ directory of the repository; report --ai interpretation has its own seeded-failure benchmark under eval/interpretation/. --ai-recover repair application remains prototype/blocked (ROADMAP Stage 12.3). The judge is oxo-flow's own fresh knowledge base plus its own validators (validate, lint) — a closed loop an external MCQ benchmark cannot provide.

Terminology: pass@1 / pass@k, fidelity, and seed are defined in the Glossary; diagnostic codes in gate verdicts are listed under Diagnostic Codes.

Three layers#

Layer What is evaluated Example metric
Tool Knowledge grounding: does the AI name the right tool, the right version, and refuse fake tools? name_match, version_match, no_hallucination
Rule Single-step generation: right tool, pinned version that exists in the knowledge base, key parameters, normalized I/O declarations, explicit resources tool_present, version_pinned, io_declared, resources_in_range
Workflow End-to-end generation: structural validity, step/tool coverage, DAG edges, final outputs across repeated trials validate_pass, lint_pass, step_coverage, edge_coverage

The loop#

# 1. Capture AI outputs for the approved gold rows
python3 eval/scripts/capture_tool.py --out outputs/tool_answers.csv --trials 5
python3 eval/scripts/capture_workflow.py workflow --out outputs/workflows --oxo-flow target/debug/oxo-flow --trials 5

# 2. Judge them
python3 eval/scripts/runner.py tool --captures outputs/tool_answers.csv
python3 eval/scripts/runner.py workflow --captures outputs/workflows --oxo-flow target/debug/oxo-flow

Results land in eval/results.csv; each row is one judged trial with per-metric scores and an overall mean. The runner also writes eval/results.items.csv (per-item aggregates with pass@1 and pass@k) and eval/results.summary.json (dataset summary, 95% bootstrap confidence intervals, breakdowns, capture availability, and capture manifest).

Capture manifests record git SHA, gold/knowledge hashes, provider/model identity, sampling settings, timestamps, and per-trial metadata so later analysis can be audited and reproduced. Failed or missing captures are excluded from model metrics and reported as infrastructure availability, not silently scored as zero.

Optional independent AI review#

Add --ai-judge to request an advisory expert-model review after deterministic judging:

AI_JUDGE_PROVIDER=openai \
AI_JUDGE_BASE_URL=https://example.endpoint/v1 \
AI_JUDGE_MODEL=chosen-judge-model \
AI_JUDGE_API_KEY=... \
python3 eval/scripts/runner.py workflow --captures outputs/workflows \
  --oxo-flow target/debug/oxo-flow --out eval/results-workflow.csv --ai-judge

Never pass the API key as a CLI argument or commit it to the repository. The advisory reviewer returns four 0--10 dimensions—scientific correctness, completeness, engine conformance, and usability—with rationales, confidence, and major risks. Results are written separately as eval/results-workflow.ai_judge.csv and eval/results-workflow.ai_judge.summary.json; they never change the deterministic score of record. Review/API failures are retained as errors so a failed judge cannot fabricate a score.

Capture uses the same provider resolution order as oxo-flow itself: OXO_FLOW_AI_PROVIDER, generic OXO_FLOW_AI_* overrides, provider-specific env vars (Anthropic/OpenAI/DeepSeek/Ollama), then ~/.oxo-flow/ai_config.json. This keeps benchmark captures aligned with the real CLI/web AI surfaces instead of assuming only one wire protocol. The advisory judge has a separate AI_JUDGE_* configuration so it can be an independent model and can be replaced without changing the benchmark.

Gold set and human review#

The gold answers live in eval/gold/*.csv. tool.csv is generated deterministically from the embedded knowledge base (python3 eval/scripts/gen_tool_csv.py); workflow.csv can be regenerated with python3 eval/scripts/build_workflow_gold.py (defaulting to the in-repo CSV path); rule.csv and workflow.csv are drafted from the gallery and the oxo-flow-community workflows. Every row carries a provenance_url pointing at its primary source.

Rows start at review_status = draft and are skipped by capture and runner until a reviewer accepts them (approved / corrected); if no reviewable rows exist the harness fails fast instead of silently emitting an empty report.

Benchmark integrity & verification status: all 257 gold rows are cryptographically bound to an immutable audit evidence ledger (eval/review/evidence.jsonl) via SHA-256 canonical fields hashing (reviewer = wangshx, 2026-10-31, AI-assisted verification baseline under maintainer supervision). Automated reviewer placeholders (*-automated) are permanently barred by CI. Independent double-blind human review and PABAK concordance reporting remain standing requirements prior to manuscript submission. python3 eval/scripts/verify_gold.py audits cryptographic ledger consistency, and python3 eval/scripts/check_gold.py (make eval-lint) guards consistency rules on every push. Complete methodology and statistical design are documented in eval/review/METHODOLOGY.md.

The deterministic grounding half of the tool layer is additionally guarded in CI by crates/oxo-flow-ai/tests/knowledge_grounding.rs — no API key required, runs on every push.

Capture manifests record git SHA, gold/knowledge hashes, provider/model identity, sampling settings, timestamps, and per-trial metadata so later analysis can be audited and reproduced.

Frontier benchmark (cross-path quality/cost)#

eval/frontier/ is a complementary, self-contained benchmark that measures the quality/cost frontier across generation paths rather than gold-set coverage on a single path. It runs a tiered intent set (easy / medium / hard) through two variants — the full CLI path (template --ai: rich prompt, knowledge tools, tool loop) and a single-shot naive prompt (no tools, no knowledge injection; a bare-prompt baseline, not a product path) — and judges every artifact with the three deterministic static gates (validate, dry-run, lint; exit code only, no LLM judge).

Per-run token usage comes from the AI session archives (CLI variant) or the provider response (naive-prompt variant), so each cell of the {path × model} matrix reports gate pass@1 alongside tokens, estimated cost, and wall time. This is the evidence base for the unified generation harness (see Architecture → AI Subsystem) and for sizing its cost tiers.

python3 eval/frontier/run_eval.py --variants naive-prompt,cli

See eval/frontier/README.md for the full method, price-table configuration, and known limitations.