How samarth6/failure-forensics-tool works
Failure Forensics Tool runs insurance-claim documents through a four-stage AI pipeline, records each step, diagnoses the earliest independently failing step, and verifies suspected causes by replaying downstream work with a correction. Reviewers can turn confirmed failures into step-level regression cases and inspect runs, evidence, and analytics in the explorer. The diagram follows the documented workflow and sampled references; some API and frontend wiring is represented at subsystem level because their full implementations were not excerpted.
Main entry points: runner.py Pipeline runner · App.tsx Trace explorer · diagnose.py Root-cause diagnosis · api.py Explorer API
Components 28
Claim pipeline
- Intake
ffx/pipeline/steps.py - Extraction
ffx/pipeline/steps.py - Classification
ffx/pipeline/steps.py - Summarization
ffx/pipeline/steps.py - Pipeline runner
ffx/pipeline/runner.py - Failure corpus
ffx/corpus/generate.py - Local inference
ffx/llm/ollama_client.py
Forensic analysis
- Deterministic signals
ffx/analyzer/signals.py - Value grounding
ffx/analyzer/grounding.py - LLM judge
ffx/analyzer/judge.py - Root-cause diagnosis
ffx/analyzer/diagnose.py - Counterfactual replay
ffx/analyzer/counterfactual.py
Tracing and storage
- Trace capture
ffx/tracing/tracer.py - Trace store
ffx/tracing/store.py - OTel export
ffx/tracing/otel.py
Regression evaluation
- Case generation
ffx/evals/generate.py - Eval cases
ffx/evals/cases.py - Regression runner
ffx/evals/runner.py - Eval analytics
ffx/evals/analytics.py
Review explorer
- Explorer API
ffx/api.py - Trace explorer
web/src/App.tsx - Trace views
web/src/components/TraceGraph.tsx - Review flags
web/src/components/FlagPanel.tsx - Eval views
web/src/components/EvalsView.tsx - Analytics view
web/src/components/AnalyticsView.tsx
Other components
- Claim document
- Human reviewer
- Cloud judge services
Connections 33
- Claim document → Pipeline runner: runs through
README.md - Failure corpus → Pipeline runner: supplies documents
README.md - Pipeline runner → Intake: executes steps
ffx/pipeline/runner.py - Pipeline runner → Extraction: executes steps
ffx/pipeline/runner.py - Pipeline runner → Classification: executes steps
ffx/pipeline/runner.py - Pipeline runner → Summarization: executes steps
ffx/pipeline/runner.py - Pipeline runner → Local inference: requests inference
ffx/pipeline/runner.py - Pipeline runner → Trace capture: records execution
ffx/pipeline/runner.py - Trace capture → Trace store: persists traces
ffx/tracing/tracer.py - Trace capture → OTel export: exports spans
README.md - Trace store → Root-cause diagnosis: provides trace data
ffx/analyzer/diagnose.py - Root-cause diagnosis → Deterministic signals: assesses spans
ffx/analyzer/diagnose.py - Deterministic signals → Value grounding: checks evidence
ffx/analyzer/signals.py - Root-cause diagnosis → LLM judge: adjudicates ambiguity
ffx/analyzer/diagnose.py - LLM judge → Cloud judge services: queries providers
README.md - Root-cause diagnosis → Counterfactual replay: tests candidates
ffx/analyzer/diagnose.py - Counterfactual replay → Pipeline runner: replays downstream
ffx/analyzer/counterfactual.py - Human reviewer → Trace explorer: reviews failures
README.md - Trace explorer → Explorer API: requests data
- Explorer API → Root-cause diagnosis: requests diagnosis
ffx/api.py - Explorer API → Trace store: reads and writes
ffx/api.py - Trace explorer → Trace views: renders traces
web/src/App.tsx - Trace explorer → Review flags: renders review
web/src/App.tsx - Trace explorer → Eval views: renders evaluations
web/src/App.tsx - Trace explorer → Analytics view: renders analytics
web/src/App.tsx - Human reviewer → Case generation: confirms failure
README.md - Case generation → Eval cases: saves validated cases
ffx/evals/generate.py - Eval cases → Regression runner: provides cases
ffx/evals/runner.py - Regression runner → Pipeline runner: executes step
ffx/evals/runner.py - Regression runner → Trace store: records results
ffx/evals/runner.py - Eval analytics → Trace store: queries run data
ffx/evals/analytics.py - Eval views → Explorer API: requests eval data
- Analytics view → Explorer API: requests analytics