# inferedge-ai/evalkit architecture

> Architecture diagram of the GitHub repository inferedge-ai/evalkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/inferedge-ai/evalkit
- Repository: https://github.com/inferedge-ai/evalkit

## Overview

evalkit is a self-hosted agent-improvement platform: production traces and agent runs become test cases, experts review and promote them, and experiments run those cases to produce scores and trajectories for iterative fixes. The README documents the main loop and SDK, UI, trace, evaluator, workspace, and promotion capabilities. Internal edges are limited to relationships visible in the README or the supplied source index/excerpts; unsampled implementation details are not inferred.

Main entry points: [`src/evalkit/core/review.py`](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/review.py) (Case Review), [`src/evalkit/core/runner.py`](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/runner.py) (Experiment Runner), [`ui/src/app.tsx`](https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/app.tsx) (Web UI), [`src/evalkit/api/app.py`](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/api/app.py) (HTTP API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_experience["User Experience"]
  node_web_ui["Web UI<br/>[app.tsx]"]
  node_cli["CLI<br/>[main.py]"]
  node_sdk["Python SDK<br/>[runner.py]"]
  node_ui_api["UI API Client<br/>[api.ts]"]
  node_ui_queries["UI Query Cache<br/>[query.ts]"]
end

subgraph group_case_lifecycle["Case Lifecycle"]
  node_trace_ingestion["Trace Ingestion"]
  node_case_authoring["Case Datasets<br/>[cases.py]"]
  node_case_review["Case Review<br/>[review.py]"]
  node_promotion["Case Promotion<br/>[promotion.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_experiment_runner["Experiment Runner<br/>[runner.py]"]
  node_evaluators["Evaluators"]
  node_trajectories["Trajectories<br/>[trajectory.py]"]
  node_scorecard["Scorecard<br/>[scorecard.py]"]
end

subgraph group_platform["Platform Services"]
  node_http_api["HTTP API<br/>[app.py]"]
  node_case_db[("Case Persistence<br/>[cases.py]")]
  node_store_contract["Store Contract<br/>[base.py]"]
  node_client_api["Client API<br/>[api.py]"]
  node_workspace_service["Workspaces<br/>[workspace.py]"]
end

node_developer(("Developer"))
node_domain_expert(("Domain Expert"))
node_coding_agent(("Coding Agent"))

node_developer -->|"runs commands"| node_cli
node_developer -->|"uses"| node_web_ui
node_domain_expert -->|"reviews cases"| node_web_ui
node_coding_agent -->|"drafts cases"| node_case_authoring
node_coding_agent -->|"runs evaluations"| node_experiment_runner
node_trace_ingestion -->|"mines failures"| node_case_authoring
node_web_ui -->|"queries through"| node_ui_queries
node_ui_queries -->|"calls"| node_ui_api
node_ui_api -->|"sends requests"| node_http_api
node_cli -->|"calls"| node_client_api
node_client_api -->|"calls"| node_http_api
node_sdk -->|"runs experiments"| node_experiment_runner
node_http_api -->|"routes review"| node_case_review
node_case_review -->|"reads and writes"| node_case_db
node_case_review -->|"promotes approved cases"| node_promotion
node_case_db -->|"uses contract"| node_store_contract
node_experiment_runner -->|"scores cases"| node_evaluators
node_experiment_runner -->|"keeps trajectories"| node_trajectories
node_experiment_runner -->|"produces scores"| node_scorecard
node_web_ui -->|"presents reviews"| node_case_review
node_case_review -->|"checks workspace scope"| node_workspace_service

click node_web_ui "https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/app.tsx"
click node_cli "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/cli/main.py"
click node_sdk "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/sdk/runner.py"
click node_http_api "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/api/app.py"
click node_trace_ingestion "https://github.com/inferedge-ai/evalkit/tree/develop/src/evalkit/core/traces"
click node_case_authoring "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/contracts/cases.py"
click node_case_review "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/review.py"
click node_promotion "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/promotion.py"
click node_experiment_runner "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/runner.py"
click node_evaluators "https://github.com/inferedge-ai/evalkit/tree/develop/src/evalkit/core/evaluators"
click node_trajectories "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/trajectory.py"
click node_scorecard "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/scorecard.py"
click node_case_db "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/db/cases.py"
click node_store_contract "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/store/base.py"
click node_client_api "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/client/api.py"
click node_workspace_service "https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/workspace.py"
click node_ui_api "https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/lib/api.ts"
click node_ui_queries "https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/lib/query.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_web_ui,node_cli,node_sdk,node_ui_api,node_ui_queries toneBlue
class node_trace_ingestion,node_case_authoring,node_case_review,node_promotion toneAmber
class node_experiment_runner,node_evaluators,node_trajectories,node_scorecard toneMint
class node_http_api,node_case_db,node_store_contract,node_client_api,node_workspace_service toneRose
class node_developer,node_domain_expert,node_coding_agent toneIndigo
```

## Components

### User Experience

- [Web UI](https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/app.tsx): `ui/src/app.tsx`
- [CLI](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/cli/main.py): `src/evalkit/cli/main.py`
- [Python SDK](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/sdk/runner.py): `src/evalkit/sdk/runner.py`
- [UI API Client](https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/lib/api.ts): `ui/src/lib/api.ts`
- [UI Query Cache](https://github.com/inferedge-ai/evalkit/blob/develop/ui/src/lib/query.ts): `ui/src/lib/query.ts`

### Case Lifecycle

- [Trace Ingestion](https://github.com/inferedge-ai/evalkit/tree/develop/src/evalkit/core/traces): `src/evalkit/core/traces`
- [Case Datasets](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/contracts/cases.py): `src/evalkit/contracts/cases.py`
- [Case Review](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/review.py): `src/evalkit/core/review.py`
- [Case Promotion](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/promotion.py): `src/evalkit/core/promotion.py`

### Evaluation

- [Experiment Runner](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/runner.py): `src/evalkit/core/runner.py`
- [Evaluators](https://github.com/inferedge-ai/evalkit/tree/develop/src/evalkit/core/evaluators): `src/evalkit/core/evaluators`
- [Trajectories](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/trajectory.py): `src/evalkit/core/trajectory.py`
- [Scorecard](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/scorecard.py): `src/evalkit/core/scorecard.py`

### Platform Services

- [HTTP API](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/api/app.py): `src/evalkit/api/app.py`
- [Case Persistence](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/db/cases.py): `src/evalkit/db/cases.py`
- [Store Contract](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/store/base.py): `src/evalkit/store/base.py`
- [Client API](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/client/api.py): `src/evalkit/client/api.py`
- [Workspaces](https://github.com/inferedge-ai/evalkit/blob/develop/src/evalkit/core/workspace.py): `src/evalkit/core/workspace.py`

### Other components

- **Developer**
- **Domain Expert**
- **Coding Agent**

## Connections

- Developer → CLI: runs commands (evidence: `README.md`)
- Developer → Web UI: uses (evidence: `README.md`)
- Domain Expert → Web UI: reviews cases (evidence: `README.md`)
- Coding Agent → Case Datasets: drafts cases (evidence: `README.md`)
- Coding Agent → Experiment Runner: runs evaluations (evidence: `README.md`)
- Trace Ingestion → Case Datasets: mines failures (evidence: `README.md`)
- Web UI → UI Query Cache: queries through (evidence: `ui/src/lib/query.ts`)
- UI Query Cache → UI API Client: calls (evidence: `ui/src/lib/query.ts`)
- UI API Client → HTTP API: sends requests (evidence: `README.md`)
- CLI → Client API: calls (evidence: `src/evalkit/cli/main.py`)
- Client API → HTTP API: calls (evidence: `README.md`)
- Python SDK → Experiment Runner: runs experiments (evidence: `README.md`)
- HTTP API → Case Review: routes review (evidence: `src/evalkit/api/app.py`)
- Case Review → Case Persistence: reads and writes (evidence: `src/evalkit/api/review.py`)
- Case Review → Case Promotion: promotes approved cases (evidence: `src/evalkit/api/review.py`)
- Case Persistence → Store Contract: uses contract (evidence: `src/evalkit/db/cases.py`)
- Experiment Runner → Evaluators: scores cases (evidence: `src/evalkit/core/runner.py`)
- Experiment Runner → Trajectories: keeps trajectories (evidence: `README.md`)
- Experiment Runner → Scorecard: produces scores (evidence: `README.md`)
- Web UI → Case Review: presents reviews (evidence: `README.md`)
- Case Review → Workspaces: checks workspace scope (evidence: `src/evalkit/api/review.py`)