Diagnostics Engine#
Deterministic failure analysis — no AI, pure pattern matching. Every diagnosis is reproducible given the same inputs.
Overview#
The Diagnostics Engine analyzes failed pipeline runs and returns:
- Error pattern identified (e.g., OOM, command not found, file missing)
- Likely cause with evidence
- Fix suggestions (auto-fixable or manual)
- Relevant log lines for context
It does NOT use AI. It matches error signatures (log keywords, exit codes)
against a curated library of 31 error patterns (tool, resource, data,
system, and config classes — see
crates/oxo-flow-core/src/diagnostics.rs for the full list; the web
domain re-exports the same table).
Matching semantics#
stderr_patternsare case-insensitive regular expressions, compiled once per engine and shared process-wide. Entries must stay specific (loader lines, tool error prefixes) — near-miss words must not fire.- A pattern without exit codes fires on any non-zero or unknown exit code — never on exit 0. A rule that exited 0 has nothing to diagnose.
- The AI monitor's failure classifier (
monitor_agent::classify_failure) runs the same pattern table, so the AI monitor and the diagnostics service never disagree on a classification.
Error Pattern Library#
31 patterns total across five categories — Tool (5), Resource (5), Data (7),
System (6), Config (8). The tables below cover the most common; the
authoritative list is crates/oxo-flow-core/src/diagnostics.rs (the web
crate re-exports it via
crates/oxo-flow-web/src/domains/execution/diagnostics.rs).
Tool Errors#
| Pattern | Signature | Auto-Fix |
|---|---|---|
| Command not found | exit 127, "command not found" | ✅ Suggest conda install |
| Version incompatibility | "version mismatch/conflict", "incompatible … version", "unsupported version" | ❌ Suggest version switch |
| Tool crash (SIGSEGV) | exit 139, "segmentation fault" | ❌ Suggest tool update |
| Tool crash (SIGILL) | exit 132, "illegal instruction" | ❌ Check CPU compatibility |
| Tool crash (SIGBUS) | exit 138, "bus error" | ❌ Check filesystem integrity |
Resource Errors#
| Pattern | Signature | Auto-Fix |
|---|---|---|
| Out of memory | exit 137/9, "out of memory", "bad_alloc", \boom\b |
✅ Increase memory limit |
| Timeout | "timed out", exit 124 | ✅ Increase time_limit |
| Disk full | ENOSPC, "no space left on device" | ❌ Suggest cleanup |
| Too many open files | "too many open files", EMFILE, "ulimit" + failure word | ✅ Raise ulimit -n before starting |
| CPU limit exceeded | "cpu time limit", "cpu limit", EAGAIN | ✅ Increase thread/CPU allocation |
Data Errors#
| Pattern | Signature | Auto-Fix |
|---|---|---|
| Input file missing | "no such file or directory", "cannot open", "file not found", exit 1 | ❌ Point to missing path |
| File truncated | "truncated", "unexpected end", "premature" | ❌ Suggest re-download |
| Corrupt gzip file | "not in gzip format", "invalid compressed data", exit 1 | ❌ Verify file is valid gzip |
| FASTQ quality low | "per base sequence quality … fail/warn", "low quality", "poor quality" | ✅ Suggest fastp insertion |
| Empty file | "empty file", "zero length", "is empty" | ❌ Check upstream rule |
| BAM truncated | "truncated file", "EOF marker", "bgzf", exit 1 | ✅ Suggest samtools index |
| BAM index missing | "could not open/retrieve index file", "no index available", exit 1 | ✅ Suggest samtools index |
System Errors#
| Pattern | Signature | Auto-Fix |
|---|---|---|
| Permission denied | exit 126/13, "permission denied", EACCES | ❌ Fix file permissions |
| Network error | "connection refused", "could not resolve host", "network is unreachable" | ❌ Check network |
| Shared library missing | "error while loading shared libraries", "cannot open shared object", "undefined symbol" (exit 127/1) | ✅ Suggest conda install |
| Broken pipe (SIGPIPE) | exit 141/13, "broken pipe" | ❌ Check pipeline commands |
| Signal kill (SIGTERM) | exit 143/15, "terminated", \bkilled\b |
❌ Check system logs |
| Signal interrupt (SIGINT) | exit 130/2, "interrupt", "cancelled" | ❌ Re-run when ready |
Config Errors#
| Pattern | Signature | Auto-Fix |
|---|---|---|
| Invalid parameter | "invalid option", "unrecognized", "unknown option", exit 1 | ❌ Check tool documentation |
| Missing required parameter | "arguments are required", "missing required |
❌ Add the missing parameter |
| Wildcard expanded empty | "no files/matches", "wildcard … did not match" | ❌ Check naming |
| Unbound wildcard in output | "contains unbound wildcard" (engine fails the rule before the shell runs) | ❌ Declare a value source (sample_pattern, [[values]], [[sample_groups]], input_groups) or remove the wildcard |
| Conda environment failed | "CondaError", "Solving environment: failed", "environment.yml", exit 1 | ❌ Verify conda env name |
| Docker failed | exit 125/126, "docker daemon", "error response from daemon" | ❌ Check Docker daemon |
| Singularity/Apptainer failed | exit 255, "singularity", "apptainer", "image not found" | ❌ Pull the image |
| Shell syntax error | exit 2, "syntax error", "unexpected token", "parse error" | ❌ Review the command |
API#
GET /api/runs/{run_id}/diagnostics
Response:
{
failed_nodes: [{
rule: "star_align",
error_pattern: "oom_killed",
likely_cause: "STAR alignment needs ~32GB; currently 16GB",
suggestions: ["Increase memory to 32GB", "Use --limitBAMsortRAM"],
relevant_log_lines: ["FATAL: out of memory", "EXITING: 137"]
}],
warnings: [{
rule: "qualimap",
pattern: "skipped",
suggestion: "This rule was skipped due to upstream failure."
}],
resource_bottlenecks: []
}
resource_bottlenecks lists rules whose measured memory use pressed against
their declared limit:
resource_bottlenecks: [{
"rule": "markdup",
"metric": "max_memory_mb",
"actual": 31.0,
"limit": 32.0
}]
actualis the rule's sampled peak RSS in MiB (recorded by the local executor into the checkpoint's benchmark records) andlimitis its declared memory limit (memory/resources.memory, resolved at execution time).- A rule is flagged when
actual ≥ 80% × limit— a conservative threshold because the peak is sampled every 200 ms across the rule's process subtree, not an exactgetrusagemaximum; sub-interval spikes can be missed. Cluster-executed rules carry the scheduler accounting store's peak RSS when it reports one (SLURMsacct's maxMaxRSSacross task rows); the value isNoneonly for schedulers without accounting (e.g. LSF) or a missing accounting row. Legacy checkpoints degrade to an empty list.
Failure-panel actions in the web monitor#
A failed node may carry auto_fixable: true and a fix_action
(description, optional config_change, optional command). The
fix_action is advisory — the diagnostics engine never reads the live
configuration, so config_change.path / old_value / new_value describe
the intent of the change with placeholder values (typically
"current" → "increase"), and today no producer sets command. No
server endpoint applies it: POST /api/runs/{id}/retry accepts only
skip_succeeded and from_rule.
Consequently the monitor's diagnostics tab renders the suggestion as
"Suggested fix (apply manually)" (path and any command shown for
copying) and labels its action "Retry unchanged": clicking it re-runs
the same workflow without mutating anything, so a rerun of an unmodified
pipeline fails the same way. Apply the change yourself first — edit the
workflow/config and create a new run, or use the CLI's verified repair
loop (oxo-flow run --ai-recover, which gates apply_fix on validation
and writes the workflow atomically). The monitor never claims a repair was
applied when none was. The AI monitor's alert cards get the same action
label ("Retry unchanged") for the same reason — their suggestion is
likewise prose, not an applicable mutation — but they render no
"Suggested fix (apply manually)" caption.
Regression coverage: frontend/e2e/monitor-apply-retry.spec.ts
(advisory fix is displayed and Retry unchanged sends only the retry
request, no repair payload: no suggestion block, Retry unchanged still
sends no repair request, alert card action is relabelled Retry unchanged
without the manual-fix caption) asserts the displayed payload is shown as
advisory, the alert card carries only the action relabel (no manual-fix
caption), and the retry POST is the only mutation sent.
Extending the Pattern Library#
Add new patterns in crates/oxo-flow-core/src/diagnostics.rs (inside
DiagnosticsEngine::new()):
Pattern {
id: "new_pattern_id",
category: ErrorCategory::Tool,
exit_codes: vec![134],
// Case-insensitive regexes — keep them specific enough that benign
// output cannot match ("tool version 1.2" must not fire).
stderr_patterns: compile(&["specific error text"]),
likely_cause: "Description of the likely cause",
auto_fixable: false,
fix_desc: Some("Suggested fix"),
fix_config_path: None,
}
Every new pattern must ship with two test rows in the module's tests:
- a canonical trigger line in
CANONICAL_TRIGGERS(proves the pattern is not dead — the test fails if a pattern has no row, or a row no pattern), - and, when near-miss words exist, a benign line in
BENIGN_LOG_LINESthat mentions the vocabulary without being a real error.
Scientific Preflight & Biological Contract Catalog#
While runtime diagnostics analyze failed executions, the Scientific Preflight Engine (crates/oxo-flow-core/src/scientific_preflight.rs) and Biological Semantic Subtyping System (crates/oxo-flow-core/src/types/) inspect workflow DAGs prior to execution. They catch scientifically unsound configurations (e.g. training VQSR on single samples, unstranded quantification on stranded libraries, or uncompressed tabix inputs) before compute resources are wasted.
Biological Subtyping Violations (TYPE-*)#
The biological type system verifies formal format and coordination contracts across DAG edges.
| Diagnostic Code | Category | Biological Violation | Automated Coercion / Remediation |
|---|---|---|---|
TYPE-001 |
Format | Data format incompatibility across producer and consumer channels (e.g. SAM passed into BAM consumer) | Synthesizes conversion rules (e.g. samtools view -b) |
TYPE-002 |
Sort Order | Sort order conflict (e.g. queryname-sorted BAM piped into coordinate-indexed consumer) | Inserts sort step (samtools sort) |
TYPE-003 |
Index | Missing companion index artifact (e.g. BAM without .bai, VCF without .tbi) |
Generates indexer rule (samtools index, tabix -p vcf) |
TYPE-004 |
Assembly | Reference build conflict (e.g. GRCh38 BAM consumed by GRCh37 variant caller) | Remediation hint only — realign against the matching reference build or liftover (no synthesized command) |
TYPE-005 |
Metadata | Missing read group @RG metadata fields required by downstream tools |
Synthesizes gatk AddOrReplaceReadGroups … (read groups can also be added at alignment, e.g. bwa mem -R) |
TYPE-006 |
Compression | Block compression stream mismatch (e.g. plain gzip passed where random-access BGZF is required) | Re-compresses stream (bgzip) |
TYPE-007 |
Attributes | Extensible attribute contract mismatch: the consumer's declared [types.*] attributes (e.g. variant_type=germline implied by gvcf) are missing or hold different values on the producer side |
Declares the missing key=value attributes on the producer output type |
TYPE-UNKNOWN |
Resolution | A types.<name> reference on a rule input/output does not match any [types.*] declaration (or inline spec fails to parse) |
Fixes the type name or adds the missing [types.*] declaration |
TYPE-008 |
Unverifiable edge (advisory) | A typed consumer reads from a producer that declares no [types] at all, so the edge cannot be checked — the contract is unknown rather than violated |
Declares the producer's output type under [rules.types].outputs (or [types.<name>]) so its biological output can be verified |
TYPE-009 |
Unverifiable build (advisory) | The consumer names a reference_build while the producer omits it, so alignment compatibility cannot be checked (the build check only fires when both sides name a build) |
Declares the producer's reference_build so its alignment against the consumer's required build can be verified |
TYPE-001 .. TYPE-007 are proven violations: they fail oxo-flow validate and abort oxo-flow run in preflight. TYPE-008 and TYPE-009 are advisory — they report an unverifiable edge (a missing contract rather than a mismatch), surface as warnings (e.g. in the scientific preflight catalog), and never fail a run.
Curated Scientific Preflight Catalog (SCI-*)#
Deterministic, evidence-backed rules covering genomics, transcriptomics, epigenomics, metagenomics, and DAG hardware execution.
1. Germline & Somatic Variant Calling#
SCI-VQSR-COHORT: VariantRecalibrator runs on fewer than 30 samples; GATK recommends minimum ~30 samples for WGS/WES to train Gaussian mixture models. Use hard filtering (VariantFiltration) for small cohorts.SCI-BQSR-NO-KNOWN-SITES: BaseRecalibrator invoked without--known-sitesresources (e.g. dbSNP, Mills); recalibration cannot model true variation without baseline polymorphic sites.SCI-MUTECT2-TUMOR-ONLY: Mutect2 executed without matched normal sample; somatic calls depend strictly on population allele frequencies and miss private germline variants and mapping artifacts.SCI-MUTECT2-NO-PON: Mutect2 in tumor-only mode missing Panel of Normals (--panel-of-normals); recurrent sequencing artifacts will be called as somatic mutations.SCI-MUTECT2-NO-GERMLINE: Mutect2 in tumor-only mode missing--germline-resource; population allele frequency priors cannot be applied to filter common germline leakage.SCI-HAPLOTYPECALLER-GVCF-ERC: HaplotypeCaller producing a.g.vcfpath without-ERC GVCFor--emit-ref-confidence GVCF; emits flat VCF without reference confidence blocks.SCI-GENOTYPEGVCFS-NO-NEW-QUAL: GenotypeGVCFs invoked without explicit quality scoring flags; legacy Phred calculations inflate variant scores in large cohorts.SCI-DEEPVARIANT-MODEL-TYPE: DeepVariant executed without--model_type; CNN weights must match sequencing chemistry (WGS,WES,PACBIO,ONT).SCI-MANTA-NO-CONFIG: Manta structural variant caller configured without reference genome FASTA specification.SCI-STRELKA-NO-MANTA: Somatic Strelka execution omitting--indelCandidatesfrom upstream Manta; small somatic indel sensitivity is substantially reduced.SCI-STRELKA-TARGETS-UNCOMPRESSED: Strelka target region BED is uncompressed; Strelka requires bgzip-compressed and tabix-indexed target regions (.bed.gz).SCI-VEP-OFFLINE-CACHE: Ensembl VEP running without local--cacheor--offlineflags; live REST queries break offline reproducibility and throttle batch jobs.
2. Alignment & Coordinate Sorting#
SCI-BWA-MEM-NO-RG:bwa memalignment lacks read group specification (-R); missing@RGheaders prevent downstream GATK/Picard processing.SCI-BOWTIE2-NO-RG:bowtie2alignment lacks read group specification (--rg-id/--rg).SCI-MINIMAP2-PRESET:minimap2invoked without technology preset (-x); default parameters degrade alignment accuracy on PacBio, ONT, or short reads.SCI-SAMTOOLS-INDEX-UNSORTED:samtools indexapplied to SAM or unsorted BAM; requires coordinate-sorted BAM/CRAM.SCI-MARKDUPLICATES-REMOVE-DUPS: Picard/GATKMarkDuplicatesconfigured withREMOVE_DUPLICATES=true; permanently discards reads needed for discordant pair and split-read CNV/SV analyses.SCI-HISAT2-NO-SPLICE-SITES:hisat2RNA-seq alignment executed without--known-splicesite-infile; reduces junction discovery sensitivity across annotated exons.SCI-FASTP-DEDUPLICATION:fastpPCR deduplication enabled without UMI barcodes on RNA-seq; removes real biological read depth for highly expressed transcripts.
3. Bulk & Single-Cell RNA-seq#
SCI-FEATURECOUNTS-STRAND:featureCountsexecuted without explicit strandness flag (-s); defaults to unstranded (-s 0), miscounting stranded libraries.SCI-FEATURECOUNTS-MULTI-OVERLAP:featureCountsruns with-O(multi-overlapping); causes duplicate assignments for overlapping genes on opposite strands.SCI-STAR-SJDB-OVERHANG:STARindex generation with annotation omits--sjdbOverhang; default 100 degrades junction mapping for 150bp or 75bp reads.SCI-STAR-GENOME-LOAD:STARuses--genomeLoad LoadAndKeepin shared batch clusters; shared memory segments risk node memory exhaustion and eviction.SCI-SALMON-LIBTYPE:salmon quantuses automatic library type detection (-l A); can select inconsistent strandedness models on shallow-coverage libraries.SCI-SALMON-NO-GC-BIAS:salmon quantexecuted without--gcBiascorrection on Illumina reads; fails to correct for sequence-specific coverage bias.SCI-KALLISTO-SINGLE-END-FRAG:kallisto quant --singleexecuted without mean fragment length (-l) or standard deviation (-s).SCI-RSEM-NO-STRAND:rsem-calculate-expressionlacks explicit--strand-specificsetting.SCI-STRINGTIE-NO-GUIDE:stringtienovel assembly running without reference guide (-G); leads to fragmented transcript models.SCI-DESEQ2-MIN-REPLICATES: Differential expression analysis configured with fewer than 3 biological replicates; insufficient degrees of freedom for negative binomial dispersion estimation.SCI-CELLRANGER-INTRONS:cellranger counton single-nucleus RNA-seq (snRNA-seq) lacks--include-introns; pre-mRNA intronic reads account for up to 50% of nuclear signal.SCI-SEURAT-NO-NORMALIZATION: Single-cell clustering or dimensionality reduction executed on raw counts without priorNormalizeDataorSCTransform.
4. Epigenomics, ChIP-seq & ATAC-seq#
SCI-MACS2-GENOME-SIZE:macs2 callpeakmissing explicit effective genome size (-g); default human (hs) invalidates peak p-values and FDR on other organisms.SCI-MACS2-BROAD-PEAKS:macs2 callpeaktargeting broad marks (H3K27me3, H3K9me3) without--broad; fragments diffuse enrichment into artificial narrow peaks.SCI-MACS2-PAIRED-END-FORMAT:macs2 callpeakon paired-end data specifying-f BAMinstead of-f BAMPE; discards exact paired fragment length modeling.SCI-CHIP-NO-CONTROL:macs2 callpeakrunning without matched input control (-c); causes elevated false-positive peak calling due to chromatin accessibility biases.SCI-ATAC-SHIFT-COORDINATES: ATAC-seq peak calling executed on unshifted BAMs; reads must be shifted (+4 bp on +, -5 bp on - strand) to center on Tn5 cut sites.SCI-BISMARK-NON-DIRECTIONAL:bismarkalignment on non-directional bisulfite library (PBAT) lacks--non_directional; discards over 50% of sequencing reads.SCI-BOWTIE-BISULFITE-MISMATCH: Bisulfite alignment uses default strict mismatch scoring; bisulfite conversion artifacts require relaxed mismatch tolerance.SCI-BEDTOOLS-GENOME-FILE:bedtools slop/flankexecuted without chromosome length definitions (-g).
5. Metagenomics & Microbiome#
SCI-KRAKEN2-CONFIDENCE:kraken2runs with default confidence threshold of 0.0; shared k-mers cause high false-positive genus and species classifications.SCI-BRACKEN-READ-LEN:brackenabundance re-estimation lacks read length parameter (-r) matching the database distribution.SCI-QIIME2-DENOISE-TRIM: QIIME2 DADA2 denoising configured without 3' quality truncation (--p-trunc-len 0); degraded read ends inflate error models and spurious ASVs.SCI-HUMANN-NUCLEOTIDE-DB:humannmetabolic profiling executed without explicit ChocoPhlAn database path (--nucleotide-database).SCI-METAPHLAN-UNKNOWN-ESTIMATION:metaphlanwith--unknown_estimationlacks explicit marker database index (--bowtie2db).SCI-CHECKM-NO-LINEAGE:checkm lineage_wfmissing genome FASTA extension flag (-x).
6. Workflow Concurrency, Hardware & Dataflow#
SCI-AGG-RACE: Rule fans out across samples, pairs, or values dimensions but declares instance-independent output paths; concurrent instances race and overwrite each other. Statically detectable, sovalidatereports it too (warning, exit 0), not justdry-run.SCI-SORT-THREADS-MEM-PRODUCT:samtools sort -mper-thread memory multiplied by thread count exceeds the rule resource budget; leads to kernel OOM kills.SCI-TMPDIR-NOT-SCRATCH: Heavy I/O tools write intermediate files to system/tmpinstead of engine-managed scratch space (${TMPDIR}).SCI-PIPE-FAIL-DISABLED: Multi-stage shell pipelines (cmd1 | cmd2 | cmd3) lackset -o pipefail; upstream tool crashes are silently masked.SCI-ZERO-BYTE-COMPLETION: Rule masks failures by creating empty binary outputs (|| touch {output}), causing downstream tools to crash on corrupt streams.SCI-MULTI-TENANT-PORT-COLLISION: Interactive server rules specify hardcoded server ports (e.g.--port 8888), causing collision failures on multi-tenant nodes.