# ukgovernmentbeis/inspect_ai architecture

> Architecture diagram of the GitHub repository ukgovernmentbeis/inspect_ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ukgovernmentbeis/inspect_ai
- Repository: https://github.com/ukgovernmentbeis/inspect_ai

## Overview

#### Purpose and entry points
Inspect is a Python framework for running and analyzing LLM evaluations, including prompts, multi-turn solvers/agents, tools, sandboxes, and scoring. The principal user entry point is the `inspect` CLI via `src/inspect_ai/__main__.py` and `src/inspect_ai/_cli/main.py`; public Python APIs are exposed through package-level modules such as `src/inspect_ai/model/__init__.py` and `src/inspect_ai/solver/__init__.py`.

#### Evaluation orchestration
The evaluation layer discovers task definitions, resolves configuration and eval sets, then schedules task/sample execution. Core flow enters CLI `eval`, loads tasks and datasets, and runs per-sample solver/model work under concurrency and limits. Key paths: `src/inspect_ai/_eval/eval.py`, `src/inspect_ai/_eval/loader.py`, `src/inspect_ai/_eval/task/run.py`.

#### Task execution, solvers, and scoring
A task combines dataset samples, solver chains, model calls, tools, and scorers. Solver state carries the evolving conversation/transcript; scorer outputs are reduced into task results. This separates evaluation mechanics from user-authored task behavior and extension registration. Key paths: `src/inspect_ai/solver/_solver.py`, `src/inspect_ai/solver/_task_state.py`, `src/inspect_ai/scorer/_scorer.py`.

#### Model abstraction and providers
The model subsystem normalizes messages, generation configuration, streaming/tool calls, retries, caching, token accounting, and compaction behind a common model API. Provider adapters form the external-service boundary for OpenAI-compatible APIs, Anthropic, Google, Bedrock, Azure, Hugging Face, local inference, and others; batch APIs are also supported. Key paths: `src/inspect_ai/model/_model.py`, `src/inspect_ai/model/_providers/providers.py`, `src/inspect_ai/model/_compaction/_compaction.py`.

#### Agents, tools, and isolation
Agents can run as solvers or tools, including ReAct, deep-agent/subagent workflows, human participation, and Agent Client Protocol (ACP) sessions. Tool invocation supports built-ins, MCP servers, browser/computer interaction, and approval gates. Execution is isolated through local or Docker/Compose sandboxes, with JSON-RPC support binaries/services for remote tool operations. Key paths: `src/inspect_ai/agent/_run.py`, `src/inspect_ai/tool/_mcp/server.py`, `src/inspect_ai/util/_sandbox/service.py`.

#### Events, logs, recovery, and checkpoints
Execution produces typed events and persistent evaluation logs, supporting streaming, buffered/chunked recording, transcript reconstruction, score edits, retries, and recovery after interruption. Checkpointing snapshots sample/evaluation state and sandbox state, using restic-backed snapshot facilities where configured. These artifacts are the boundary between execution, resumption, analysis, and visualization. Key paths: `src/inspect_ai/event/_event.py`, `src/inspect_ai/log/_recorders/recorder.py`, `src/inspect_ai/util/_checkpoint/checkpointer_impl.py`.

#### Control plane and visualization
A runtime control server exposes live evaluation state for pause, cancel, requeue, limits, scoring, and task/sample inspection; `inspect ctl` is its operator client. Separately, `inspect view` serves a FastAPI backend, OpenAPI contract, and bundled TypeScript/React log viewer; the frontend source is a git submodule. Key paths: `src/inspect_ai/_control/server.py`, `src/inspect_ai/_cli/ctl/_http.py`, `src/inspect_ai/_view/fastapi_server.py`.

#### Runtimes and delivery
The primary runtime is Python, with async execution and optional Docker/Compose sandbox infrastructure. The viewer is a FastAPI-served static React bundle; docs use Quarto. This monorepo also ships separately structured sandbox-tool and tool-support Python packages under `src/inspect_sandbox_tools/` and `src/inspect_tool_support/`, used to execute privileged/remote tool capabilities across the sandbox boundary.

Main entry points: [`src/inspect_ai/_eval/task/run.py`](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/task/run.py) (Task Sample Runner), [`src/inspect_ai/solver/_solver.py`](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/solver/_solver.py) (Solver Chains), [`src/inspect_ai/_eval/eval.py`](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/eval.py) (Evaluation Orchestrator), [`src/inspect_ai/model/_model.py`](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_model.py) (Model API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry Points"]
  node_cli["inspect CLI<br/>[main.py]"]
  node_python_api["Public Python APIs<br/>package exports<br/>[__init__.py]"]
end

subgraph group_evaluation["Evaluation Runtime"]
  node_eval["Evaluation Orchestrator<br/>evaluation engine<br/>[eval.py]"]
  node_loader["Task &amp; Dataset Loader<br/>discovery<br/>[loader.py]"]
  node_task_runner["Task Sample Runner<br/>async scheduler<br/>[run.py]"]
  node_solver["Solver Chains<br/>extension API<br/>[_solver.py]"]
  node_task_state["Solver State<br/>conversation state<br/>[_task_state.py]"]
  node_scorer["Scorers<br/>extension API<br/>[_scorer.py]"]
end

subgraph group_execution["Agent &amp; Tool Execution"]
  node_agent_runtime["Agent Runtime<br/>agent execution<br/>[_run.py]"]
  node_mcp["MCP Tool Server<br/>tool protocol<br/>[server.py]"]
  node_sandbox{{"Sandbox Service<br/>isolation boundary<br/>[service.py]"}}
  node_tool_packages["Sandbox Tool Packages<br/>privileged tools"]
end

subgraph group_model["Model Layer"]
  node_model_api["Model API<br/>model abstraction<br/>[_model.py]"]
  node_providers{{"Provider Adapters<br/>external API adapters<br/>[providers.py]"}}
  node_compaction["Context Compaction<br/>context management<br/>[_compaction.py]"]
end

subgraph group_artifacts["Artifacts &amp; Operations"]
  node_events["Typed Events<br/>event stream<br/>[_event.py]"]
  node_logs[("Evaluation Logs<br/>persistent artifacts<br/>[recorder.py]")]
  node_checkpoints[("Checkpoints<br/>recovery snapshots")]
  node_control_server["Runtime Control Server<br/>control plane<br/>[server.py]"]
  node_ctl_client["inspect ctl Client<br/>operator client<br/>[_http.py]"]
end

subgraph group_viewer["Visualization"]
  node_view_server["View Server<br/>FastAPI backend<br/>[fastapi_server.py]"]
  node_react_viewer["React Log Viewer<br/>static frontend"]
end

node_cli -->|"eval"| node_eval
node_cli -->|"ctl"| node_ctl_client
node_cli -->|"view"| node_view_server
node_python_api -->|"exports"| node_model_api
node_python_api -->|"exports"| node_solver
node_eval -->|"discovers"| node_loader
node_eval -->|"schedules"| node_task_runner
node_task_runner -->|"runs"| node_solver
node_solver -->|"updates"| node_task_state
node_task_runner -->|"scores"| node_scorer
node_solver -->|"generates"| node_model_api
node_solver -->|"invokes"| node_agent_runtime
node_agent_runtime -->|"uses tools"| node_mcp
node_agent_runtime -->|"isolates execution"| node_sandbox
node_sandbox -->|"executes"| node_tool_packages
node_model_api -->|"manages context"| node_compaction
node_model_api -->|"dispatches"| node_providers
node_task_runner -->|"emits"| node_events
node_events -->|"records"| node_logs
node_task_runner -->|"snapshots"| node_checkpoints
node_control_server -->|"controls live runs"| node_eval
node_ctl_client -->|"HTTP control"| node_control_server
node_view_server -->|"reads"| node_logs
node_view_server -->|"serves"| node_react_viewer

click node_cli "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_cli/main.py"
click node_python_api "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/__init__.py"
click node_eval "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/eval.py"
click node_loader "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/loader.py"
click node_task_runner "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/task/run.py"
click node_solver "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/solver/_solver.py"
click node_task_state "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/solver/_task_state.py"
click node_scorer "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/scorer/_scorer.py"
click node_agent_runtime "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/agent/_run.py"
click node_mcp "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/tool/_mcp/server.py"
click node_sandbox "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/util/_sandbox/service.py"
click node_model_api "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_model.py"
click node_providers "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_providers/providers.py"
click node_compaction "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_compaction/_compaction.py"
click node_events "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/event/_event.py"
click node_logs "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/log/_recorders/recorder.py"
click node_checkpoints "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/util/_checkpoint/checkpointer_impl.py"
click node_control_server "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_control/server.py"
click node_ctl_client "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_cli/ctl/_http.py"
click node_view_server "https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_view/fastapi_server.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_python_api toneBlue
class node_eval,node_loader,node_task_runner,node_solver,node_task_state,node_scorer toneAmber
class node_agent_runtime,node_mcp,node_sandbox,node_tool_packages toneMint
class node_model_api,node_providers,node_compaction toneRose
class node_events,node_logs,node_checkpoints,node_control_server,node_ctl_client toneIndigo
class node_view_server,node_react_viewer toneTeal
```

## Components

### Entry Points

- [inspect CLI](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_cli/main.py): CLI · Commands for eval, ctl, and view. · `src/inspect_ai/_cli/main.py`
- [Public Python APIs](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/__init__.py): package exports · `src/inspect_ai/model/__init__.py`

### Evaluation Runtime

- [Evaluation Orchestrator](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/eval.py): evaluation engine · Resolves evaluation configuration and coordinates runs. · `src/inspect_ai/_eval/eval.py`
- [Task & Dataset Loader](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/loader.py): discovery · `src/inspect_ai/_eval/loader.py`
- [Task Sample Runner](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_eval/task/run.py): async scheduler · Executes samples under concurrency and resource limits. · `src/inspect_ai/_eval/task/run.py`
- [Solver Chains](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/solver/_solver.py): extension API · `src/inspect_ai/solver/_solver.py`
- [Solver State](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/solver/_task_state.py): conversation state · Carries the evolving sample transcript and state. · `src/inspect_ai/solver/_task_state.py`
- [Scorers](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/scorer/_scorer.py): extension API · Produces scores reduced into task results. · `src/inspect_ai/scorer/_scorer.py`

### Agent & Tool Execution

- [Agent Runtime](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/agent/_run.py): agent execution · Runs agents as solvers or tools, including subagents and ACP. · `src/inspect_ai/agent/_run.py`
- [MCP Tool Server](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/tool/_mcp/server.py): tool protocol · Connects tool invocation to MCP servers. · `src/inspect_ai/tool/_mcp/server.py`
- [Sandbox Service](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/util/_sandbox/service.py): isolation boundary · Runs local or Docker/Compose isolated execution. · `src/inspect_ai/util/_sandbox/service.py`
- **Sandbox Tool Packages**: privileged tools · Separately structured packages for remote sandbox capabilities.

### Model Layer

- [Model API](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_model.py): model abstraction · Normalizes generation, streaming, tools, retries, caching, and token accounting. · `src/inspect_ai/model/_model.py`
- [Provider Adapters](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_providers/providers.py): external API adapters · Supports hosted, local, and batch model providers. · `src/inspect_ai/model/_providers/providers.py`
- [Context Compaction](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/model/_compaction/_compaction.py): context management · `src/inspect_ai/model/_compaction/_compaction.py`

### Artifacts & Operations

- [Typed Events](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/event/_event.py): event stream · `src/inspect_ai/event/_event.py`
- [Evaluation Logs](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/log/_recorders/recorder.py): persistent artifacts · Records execution for recovery, analysis, and transcript reconstruction. · `src/inspect_ai/log/_recorders/recorder.py`
- [Checkpoints](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/util/_checkpoint/checkpointer_impl.py): recovery snapshots · Snapshots evaluation and sandbox state for resumption. · `src/inspect_ai/util/_checkpoint/checkpointer_impl.py`
- [Runtime Control Server](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_control/server.py): control plane · Exposes live state and pause, cancel, requeue, limit, and scoring controls. · `src/inspect_ai/_control/server.py`
- [inspect ctl Client](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_cli/ctl/_http.py): operator client · `src/inspect_ai/_cli/ctl/_http.py`

### Visualization

- [View Server](https://github.com/ukgovernmentbeis/inspect_ai/blob/main/src/inspect_ai/_view/fastapi_server.py): FastAPI backend · Serves logs, the OpenAPI contract, and the bundled React viewer. · `src/inspect_ai/_view/fastapi_server.py`
- **React Log Viewer**: static frontend · Bundled TypeScript/React log visualization.

## Connections

- inspect CLI → Evaluation Orchestrator: eval
- inspect CLI → inspect ctl Client: ctl
- inspect CLI → View Server: view
- Public Python APIs → Model API: exports
- Public Python APIs → Solver Chains: exports
- Evaluation Orchestrator → Task & Dataset Loader: discovers
- Evaluation Orchestrator → Task Sample Runner: schedules
- Task Sample Runner → Solver Chains: runs
- Solver Chains → Solver State: updates
- Task Sample Runner → Scorers: scores
- Solver Chains → Model API: generates
- Solver Chains → Agent Runtime: invokes
- Agent Runtime → MCP Tool Server: uses tools
- Agent Runtime → Sandbox Service: isolates execution
- Sandbox Service → Sandbox Tool Packages: executes
- Model API → Context Compaction: manages context
- Model API → Provider Adapters: dispatches
- Task Sample Runner → Typed Events: emits
- Typed Events → Evaluation Logs: records
- Task Sample Runner → Checkpoints: snapshots
- Runtime Control Server → Evaluation Orchestrator: controls live runs
- inspect ctl Client → Runtime Control Server: HTTP control
- View Server → Evaluation Logs: reads
- View Server → React Log Viewer: serves
