How inferedge-ai/evalkit works
evalkit is a self-hosted agent-improvement platform: production traces and agent runs become test cases, experts review and promote them, and experiments run those cases to produce scores and trajectories for iterative fixes. The README documents the main loop and SDK, UI, trace, evaluator, workspace, and promotion capabilities. Internal edges are limited to relationships visible in the README or the supplied source index/excerpts; unsampled implementation details are not inferred.
Main entry points: review.py Case Review · runner.py Experiment Runner · app.tsx Web UI · app.py HTTP API
Components 21
User Experience
- Web UI
ui/src/app.tsx - CLI
src/evalkit/cli/main.py - Python SDK
src/evalkit/sdk/runner.py - UI API Client
ui/src/lib/api.ts - UI Query Cache
ui/src/lib/query.ts
Case Lifecycle
- Trace Ingestion
src/evalkit/core/traces - Case Datasets
src/evalkit/contracts/cases.py - Case Review
src/evalkit/core/review.py - Case Promotion
src/evalkit/core/promotion.py
Evaluation
- Experiment Runner
src/evalkit/core/runner.py - Evaluators
src/evalkit/core/evaluators - Trajectories
src/evalkit/core/trajectory.py - Scorecard
src/evalkit/core/scorecard.py
Platform Services
- HTTP API
src/evalkit/api/app.py - Case Persistence
src/evalkit/db/cases.py - Store Contract
src/evalkit/store/base.py - Client API
src/evalkit/client/api.py - Workspaces
src/evalkit/core/workspace.py
Other components
- Developer
- Domain Expert
- Coding Agent
Connections 21
- Developer → CLI: runs commands
README.md - Developer → Web UI: uses
README.md - Domain Expert → Web UI: reviews cases
README.md - Coding Agent → Case Datasets: drafts cases
README.md - Coding Agent → Experiment Runner: runs evaluations
README.md - Trace Ingestion → Case Datasets: mines failures
README.md - Web UI → UI Query Cache: queries through
ui/src/lib/query.ts - UI Query Cache → UI API Client: calls
ui/src/lib/query.ts - UI API Client → HTTP API: sends requests
README.md - CLI → Client API: calls
src/evalkit/cli/main.py - Client API → HTTP API: calls
README.md - Python SDK → Experiment Runner: runs experiments
README.md - HTTP API → Case Review: routes review
src/evalkit/api/app.py - Case Review → Case Persistence: reads and writes
src/evalkit/api/review.py - Case Review → Case Promotion: promotes approved cases
src/evalkit/api/review.py - Case Persistence → Store Contract: uses contract
src/evalkit/db/cases.py - Experiment Runner → Evaluators: scores cases
src/evalkit/core/runner.py - Experiment Runner → Trajectories: keeps trajectories
README.md - Experiment Runner → Scorecard: produces scores
README.md - Web UI → Case Review: presents reviews
README.md - Case Review → Workspaces: checks workspace scope
src/evalkit/api/review.py