How ukgovernmentbeis/inspect_ai works
Purpose and entry points
Inspect is a Python framework for running and analyzing LLM evaluations, including prompts, multi-turn solvers/agents, tools, sandboxes, and scoring. The principal user entry point is the inspect CLI via src/inspect_ai/__main__.py and src/inspect_ai/_cli/main.py; public Python APIs are exposed through package-level modules such as src/inspect_ai/model/__init__.py and src/inspect_ai/solver/__init__.py.
Evaluation orchestration
The evaluation layer discovers task definitions, resolves configuration and eval sets, then schedules task/sample execution. Core flow enters CLI eval, loads tasks and datasets, and runs per-sample solver/model work under concurrency and limits. Key paths: src/inspect_ai/_eval/eval.py, src/inspect_ai/_eval/loader.py, src/inspect_ai/_eval/task/run.py.
Task execution, solvers, and scoring
A task combines dataset samples, solver chains, model calls, tools, and scorers. Solver state carries the evolving conversation/transcript; scorer outputs are reduced into task results. This separates evaluation mechanics from user-authored task behavior and extension registration. Key paths: src/inspect_ai/solver/_solver.py, src/inspect_ai/solver/_task_state.py, src/inspect_ai/scorer/_scorer.py.
Model abstraction and providers
The model subsystem normalizes messages, generation configuration, streaming/tool calls, retries, caching, token accounting, and compaction behind a common model API. Provider adapters form the external-service boundary for OpenAI-compatible APIs, Anthropic, Google, Bedrock, Azure, Hugging Face, local inference, and others; batch APIs are also supported. Key paths: src/inspect_ai/model/_model.py, src/inspect_ai/model/_providers/providers.py, src/inspect_ai/model/_compaction/_compaction.py.
Agents, tools, and isolation
Agents can run as solvers or tools, including ReAct, deep-agent/subagent workflows, human participation, and Agent Client Protocol (ACP) sessions. Tool invocation supports built-ins, MCP servers, browser/computer interaction, and approval gates. Execution is isolated through local or Docker/Compose sandboxes, with JSON-RPC support binaries/services for remote tool operations. Key paths: src/inspect_ai/agent/_run.py, src/inspect_ai/tool/_mcp/server.py, src/inspect_ai/util/_sandbox/service.py.
Events, logs, recovery, and checkpoints
Execution produces typed events and persistent evaluation logs, supporting streaming, buffered/chunked recording, transcript reconstruction, score edits, retries, and recovery after interruption. Checkpointing snapshots sample/evaluation state and sandbox state, using restic-backed snapshot facilities where configured. These artifacts are the boundary between execution, resumption, analysis, and visualization. Key paths: src/inspect_ai/event/_event.py, src/inspect_ai/log/_recorders/recorder.py, src/inspect_ai/util/_checkpoint/checkpointer_impl.py.
Control plane and visualization
A runtime control server exposes live evaluation state for pause, cancel, requeue, limits, scoring, and task/sample inspection; inspect ctl is its operator client. Separately, inspect view serves a FastAPI backend, OpenAPI contract, and bundled TypeScript/React log viewer; the frontend source is a git submodule. Key paths: src/inspect_ai/_control/server.py, src/inspect_ai/_cli/ctl/_http.py, src/inspect_ai/_view/fastapi_server.py.
Runtimes and delivery
The primary runtime is Python, with async execution and optional Docker/Compose sandbox infrastructure. The viewer is a FastAPI-served static React bundle; docs use Quarto. This monorepo also ships separately structured sandbox-tool and tool-support Python packages under src/inspect_sandbox_tools/ and src/inspect_tool_support/, used to execute privileged/remote tool capabilities across the sandbox boundary.
Main entry points: run.py Task Sample Runner · _solver.py Solver Chains · eval.py Evaluation Orchestrator · _model.py Model API
Components 22
Entry Points
- inspect CLI · Commands for eval, ctl, and view.
src/inspect_ai/_cli/main.py - Public Python APIs
src/inspect_ai/model/__init__.py
Evaluation Runtime
- Evaluation Orchestrator · Resolves evaluation configuration and coordinates runs.
src/inspect_ai/_eval/eval.py - Task & Dataset Loader
src/inspect_ai/_eval/loader.py - Task Sample Runner · Executes samples under concurrency and resource limits.
src/inspect_ai/_eval/task/run.py - Solver Chains
src/inspect_ai/solver/_solver.py - Solver State · Carries the evolving sample transcript and state.
src/inspect_ai/solver/_task_state.py - Scorers · Produces scores reduced into task results.
src/inspect_ai/scorer/_scorer.py
Agent & Tool Execution
- Agent Runtime · Runs agents as solvers or tools, including subagents and ACP.
src/inspect_ai/agent/_run.py - MCP Tool Server · Connects tool invocation to MCP servers.
src/inspect_ai/tool/_mcp/server.py - Sandbox Service · Runs local or Docker/Compose isolated execution.
src/inspect_ai/util/_sandbox/service.py - Sandbox Tool Packages · Separately structured packages for remote sandbox capabilities.
Model Layer
- Model API · Normalizes generation, streaming, tools, retries, caching, and token accounting.
src/inspect_ai/model/_model.py - Provider Adapters · Supports hosted, local, and batch model providers.
src/inspect_ai/model/_providers/providers.py - Context Compaction
src/inspect_ai/model/_compaction/_compaction.py
Artifacts & Operations
- Typed Events
src/inspect_ai/event/_event.py - Evaluation Logs · Records execution for recovery, analysis, and transcript reconstruction.
src/inspect_ai/log/_recorders/recorder.py - Checkpoints · Snapshots evaluation and sandbox state for resumption.
src/inspect_ai/util/_checkpoint/checkpointer_impl.py - Runtime Control Server · Exposes live state and pause, cancel, requeue, limit, and scoring controls.
src/inspect_ai/_control/server.py - inspect ctl Client
src/inspect_ai/_cli/ctl/_http.py
Visualization
- View Server · Serves logs, the OpenAPI contract, and the bundled React viewer.
src/inspect_ai/_view/fastapi_server.py - React Log Viewer · Bundled TypeScript/React log visualization.
Connections 24
- inspect CLI → Evaluation Orchestrator: eval
- inspect CLI → inspect ctl Client: ctl
- inspect CLI → View Server: view
- Public Python APIs → Model API: exports
- Public Python APIs → Solver Chains: exports
- Evaluation Orchestrator → Task & Dataset Loader: discovers
- Evaluation Orchestrator → Task Sample Runner: schedules
- Task Sample Runner → Solver Chains: runs
- Solver Chains → Solver State: updates
- Task Sample Runner → Scorers: scores
- Solver Chains → Model API: generates
- Solver Chains → Agent Runtime: invokes
- Agent Runtime → MCP Tool Server: uses tools
- Agent Runtime → Sandbox Service: isolates execution
- Sandbox Service → Sandbox Tool Packages: executes
- Model API → Context Compaction: manages context
- Model API → Provider Adapters: dispatches
- Task Sample Runner → Typed Events: emits
- Typed Events → Evaluation Logs: records
- Task Sample Runner → Checkpoints: snapshots
- Runtime Control Server → Evaluation Orchestrator: controls live runs
- inspect ctl Client → Runtime Control Server: HTTP control
- View Server → Evaluation Logs: reads
- View Server → React Log Viewer: serves