How sylesh29/voice-agent-eval works
Voice Agent Eval is a local evaluation platform for simulated healthcare voice-agent workflows. A CLI run loads a scenario and agent configuration, simulates caller interaction and tool-driven changes to a ground-truth world, then evaluates the resulting trace and writes run results. The API exposes stored runs for comparison and inspection; the frontend presents run, scenario, failure-pattern, and evaluator-calibration views. The README documents these capabilities, but several implementation paths are unsampled, so uncertain links are omitted rather than inferred.
Main entry points: runner.py Evaluation runner · rule_agent.py Rule agent · closure.py Closure evaluation · ui.tsx Metric components
Components 25
Scenario simulation
- Run CLI
backend/voiceval/cli.py - Evaluation runner
backend/voiceval/runner.py - Scenario contracts
backend/voiceval/scenario.py - Rule agent
backend/voiceval/agents/rule_agent.py - Agent versions
backend/voiceval/agents/configs.py - Caller simulator
backend/voiceval/caller.py - Fault-injected tools
backend/voiceval/tools.py - Simulated world
backend/voiceval/world.py - Run trace
backend/voiceval/trace.py
Evaluation engine
- State metrics
backend/voiceval/evaluators/core.py - Closure evaluation
backend/voiceval/evaluators/closure.py - Language judge
backend/voiceval/evaluators/judge.py - Metric aggregation
backend/voiceval/evaluators/aggregate.py - Evaluator calibration
backend/voiceval/calibrate.py - Mix sensitivity
backend/voiceval/reweight.py
API and persistence
- Read/write API
backend/app/main.py - Run database
backend/app/db.py
Results interface
- Run views
frontend/app/runs - Run comparison
frontend/app/compare/page.tsx - Failure patterns
frontend/app/patterns/page.tsx - Calibration view
frontend/app/calibration/page.tsx - Metric components
frontend/app/ui.tsx - Metric definitions
frontend/app/metrics.ts
Other components
- Operator
- Reviewer
Connections 33
- Operator → Run CLI: starts run
README.md - Run CLI → Evaluation runner: invokes
backend/voiceval/cli.py - Evaluation runner → Scenario contracts: loads
backend/voiceval/runner.py - Evaluation runner → Agent versions: selects version
backend/voiceval/runner.py - Evaluation runner → Rule agent: runs
backend/voiceval/runner.py - Evaluation runner → Caller simulator: simulates
backend/voiceval/runner.py - Evaluation runner → Fault-injected tools: constructs
backend/voiceval/runner.py - Evaluation runner → Simulated world: builds
backend/voiceval/runner.py - Evaluation runner → Run trace: records
backend/voiceval/runner.py - Rule agent → Agent versions: reads policy
backend/voiceval/agents/rule_agent.py - Rule agent → Caller simulator: exchanges turns
backend/voiceval/agents/rule_agent.py - Rule agent → Fault-injected tools: calls tools
backend/voiceval/agents/rule_agent.py - Rule agent → Run trace: records outcome
backend/voiceval/agents/rule_agent.py - Fault-injected tools → Simulated world: reads and mutates
backend/voiceval/tools.py - Scenario contracts → Simulated world: builds fixture
backend/voiceval/scenario.py - Evaluation runner → State metrics: evaluates trace
- Evaluation runner → Closure evaluation: evaluates closure
- Closure evaluation → State metrics: uses state checks
backend/voiceval/evaluators/closure.py - Closure evaluation → Language judge: requests judgment
backend/voiceval/evaluators/closure.py - Evaluation runner → Metric aggregation: aggregates results
- Evaluator calibration → Closure evaluation: calibrates evaluator
- Mix sensitivity → Metric aggregation: analyzes mix
- Read/write API → Run database: reads and writes
backend/app/main.py - Reviewer → Run views: reviews runs
README.md - Reviewer → Run comparison: compares runs
README.md - Reviewer → Failure patterns: inspects patterns
README.md - Reviewer → Calibration view: reviews calibration
README.md - Run views → Metric components: renders
frontend/app/runs/[runId]/page.tsx - Run comparison → Metric components: renders
frontend/app/compare/page.tsx - Run comparison → Metric definitions: uses labels
frontend/app/compare/page.tsx - Failure patterns → Metric components: renders
frontend/app/patterns/page.tsx - Failure patterns → Metric definitions: uses labels
frontend/app/patterns/page.tsx - Calibration view → Metric components: renders
frontend/app/calibration/page.tsx