# ukgovernmentbeis/inspect_evals architecture

> Architecture diagram of the GitHub repository ukgovernmentbeis/inspect_evals, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ukgovernmentbeis/inspect_evals
- Repository: https://github.com/ukgovernmentbeis/inspect_evals

## Overview

Inspect Evals is a Python library of Inspect AI benchmark tasks. A user selects or imports an evaluation, runs it against a configured model, and reviews the resulting logs; individual evaluations supply their own datasets, solvers, and scoring logic. The repository also includes trajectory-analysis scanners and a register-submission workflow for sharing evaluations. The graph preserves the major sampled benchmark, safety, and registration stages. Many benchmark modules are unsampled, so their internal connections are omitted unless documented or listed as references in the source index.

Main entry points: [`src/inspect_evals/mmlu/mmlu.py`](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/mmlu/mmlu.py) (MMLU evaluation), [`src/inspect_evals/utils/huggingface.py`](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/utils/huggingface.py) (Hugging Face utilities), [`src/register_tooling/engine.py`](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/engine.py) (Submission engine), [`src/inspect_evals/apps/apps.py`](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/apps/apps.py) (APPS task)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_evals["Evaluation tasks"]
  node_apps["APPS task<br/>[apps.py]"]
end

subgraph group_benchmarks["Benchmark data"]
  node_apps_data["APPS dataset<br/>[apps.py]"]
  node_bbeh_data["BBEH dataset<br/>[data.py]"]
  node_mmlu["MMLU evaluation<br/>[mmlu.py]"]
  node_livecode_data["LiveCodeBench data<br/>[dataset.py]"]
  node_stereoset["StereoSet evaluation<br/>[stereoset.py]"]
  node_sciknow["SciKnowEval task<br/>[sciknoweval.py]"]
  node_sciknow_scorer["SciKnowEval scorer<br/>[scorer.py]"]
  node_abstention_prompts["Abstention judge prompts"]
end

subgraph group_safety["Safety and agents"]
  node_osworld_controller["OSWorld controller<br/>[controller.py]"]
  node_gdm_pipeline["Stealth solver<br/>[solver_pipeline.py]"]
  node_gdm_env["Customer support environment"]
  node_gdm_oversight["Oversight rules<br/>[oversight.py]"]
  node_gdm_ticket["Support tickets<br/>[ticket.py]"]
  node_scanners["Trajectory scanners<br/>[scanners.py]"]
end

subgraph group_shared["Shared runtime"]
  node_metadata["Eval metadata<br/>[metadata.py]"]
  node_hf_utils["Hugging Face utilities<br/>[huggingface.py]"]
  node_osworld_logging["OSWorld logging<br/>[logging.py]"]
end

subgraph group_register["Evaluation register"]
  node_register_engine["Submission engine<br/>[engine.py]"]
  node_register_arxiv["Paper metadata client<br/>[arxiv_client.py]"]
  node_register_deriver["Evaluation derivation<br/>[deriver.py]"]
end

node_user(("Evaluator"))
node_inspect["Inspect AI"]
node_logs["Evaluation logs"]

node_user -->|"runs tasks"| node_inspect
node_inspect -->|"runs evaluation"| node_apps
node_inspect -->|"runs evaluation"| node_mmlu
node_inspect -->|"writes logs"| node_logs
node_apps -->|"loads dataset"| node_apps_data
node_bbeh_data -->|"uses utilities"| node_hf_utils
node_mmlu -->|"uses metadata"| node_metadata
node_mmlu -->|"uses utilities"| node_hf_utils
node_livecode_data -->|"uses utilities"| node_hf_utils
node_sciknow -->|"uses scorer"| node_sciknow_scorer
node_gdm_pipeline -->|"uses environment"| node_gdm_env
node_osworld_controller -->|"logs actions"| node_osworld_logging
node_register_engine -->|"fetches paper data"| node_register_arxiv
node_register_engine -->|"derives listing"| node_register_deriver
node_register_deriver -->|"reads paper data"| node_register_arxiv

click node_apps "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/apps/apps.py"
click node_apps_data "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/apps/huggingface_artifact/apps.py"
click node_bbeh_data "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/bbeh/data.py"
click node_mmlu "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/mmlu/mmlu.py"
click node_livecode_data "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/livecodebench_pro/dataset.py"
click node_stereoset "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/stereoset/stereoset.py"
click node_sciknow "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/sciknoweval/sciknoweval.py"
click node_sciknow_scorer "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/sciknoweval/scorer.py"
click node_abstention_prompts "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/abstention_bench/recipe/evaluation_judge_prompts.py"
click node_metadata "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/metadata.py"
click node_hf_utils "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/utils/huggingface.py"
click node_osworld_controller "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/osworld/container/code/controller.py"
click node_osworld_logging "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/osworld/container/code/logging.py"
click node_gdm_pipeline "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver_pipeline.py"
click node_gdm_env "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/customer_support_env.py"
click node_gdm_oversight "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/oversight.py"
click node_gdm_ticket "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/ticket.py"
click node_scanners "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/agent_artefacts/trajectory_analysis/inspect_scout/scanners.py"
click node_register_engine "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/engine.py"
click node_register_arxiv "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/arxiv_client.py"
click node_register_deriver "https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/deriver.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_apps toneBlue
class node_apps_data,node_bbeh_data,node_mmlu,node_livecode_data,node_stereoset,node_sciknow,node_sciknow_scorer,node_abstention_prompts toneAmber
class node_osworld_controller,node_gdm_pipeline,node_gdm_env,node_gdm_oversight,node_gdm_ticket,node_scanners toneMint
class node_metadata,node_hf_utils,node_osworld_logging toneRose
class node_register_engine,node_register_arxiv,node_register_deriver,node_user,node_inspect,node_logs toneIndigo
```

## Components

### Evaluation tasks

- [APPS task](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/apps/apps.py): `src/inspect_evals/apps/apps.py`

### Benchmark data

- [APPS dataset](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/apps/huggingface_artifact/apps.py): `src/inspect_evals/apps/huggingface_artifact/apps.py`
- [BBEH dataset](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/bbeh/data.py): `src/inspect_evals/bbeh/data.py`
- [MMLU evaluation](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/mmlu/mmlu.py): `src/inspect_evals/mmlu/mmlu.py`
- [LiveCodeBench data](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/livecodebench_pro/dataset.py): `src/inspect_evals/livecodebench_pro/dataset.py`
- [StereoSet evaluation](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/stereoset/stereoset.py): `src/inspect_evals/stereoset/stereoset.py`
- [SciKnowEval task](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/sciknoweval/sciknoweval.py): `src/inspect_evals/sciknoweval/sciknoweval.py`
- [SciKnowEval scorer](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/sciknoweval/scorer.py): `src/inspect_evals/sciknoweval/scorer.py`
- [Abstention judge prompts](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/abstention_bench/recipe/evaluation_judge_prompts.py): `src/inspect_evals/abstention_bench/recipe/evaluation_judge_prompts.py`

### Safety and agents

- [OSWorld controller](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/osworld/container/code/controller.py): `src/inspect_evals/osworld/container/code/controller.py`
- [Stealth solver](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver_pipeline.py): `src/inspect_evals/gdm_stealth/solver_pipeline.py`
- [Customer support environment](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/customer_support_env.py): `src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/customer_support_env.py`
- [Oversight rules](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/oversight.py): `src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/oversight.py`
- [Support tickets](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/ticket.py): `src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/ticket.py`
- [Trajectory scanners](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/agent_artefacts/trajectory_analysis/inspect_scout/scanners.py): `agent_artefacts/trajectory_analysis/inspect_scout/scanners.py`

### Shared runtime

- [Eval metadata](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/metadata.py): `src/inspect_evals/metadata.py`
- [Hugging Face utilities](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/utils/huggingface.py): `src/inspect_evals/utils/huggingface.py`
- [OSWorld logging](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/inspect_evals/osworld/container/code/logging.py): `src/inspect_evals/osworld/container/code/logging.py`

### Evaluation register

- [Submission engine](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/engine.py): `src/register_tooling/engine.py`
- [Paper metadata client](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/arxiv_client.py): `src/register_tooling/arxiv_client.py`
- [Evaluation derivation](https://github.com/UKGovernmentBEIS/inspect_evals/blob/main/src/register_tooling/deriver.py): `src/register_tooling/deriver.py`

### Other components

- **Evaluator**
- **Inspect AI**
- **Evaluation logs**

## Connections

- Evaluator → Inspect AI: runs tasks (evidence: `README.md`)
- Inspect AI → APPS task: runs evaluation (evidence: `README.md`)
- Inspect AI → MMLU evaluation: runs evaluation (evidence: `README.md`)
- Inspect AI → Evaluation logs: writes logs (evidence: `README.md`)
- APPS task → APPS dataset: loads dataset (evidence: `src/inspect_evals/apps/apps.py`)
- BBEH dataset → Hugging Face utilities: uses utilities (evidence: `src/inspect_evals/bbeh/data.py`)
- MMLU evaluation → Eval metadata: uses metadata (evidence: `src/inspect_evals/mmlu/mmlu.py`)
- MMLU evaluation → Hugging Face utilities: uses utilities (evidence: `src/inspect_evals/mmlu/mmlu.py`)
- LiveCodeBench data → Hugging Face utilities: uses utilities (evidence: `src/inspect_evals/livecodebench_pro/dataset.py`)
- SciKnowEval task → SciKnowEval scorer: uses scorer (evidence: `src/inspect_evals/sciknoweval/sciknoweval.py`)
- Stealth solver → Customer support environment: uses environment (evidence: `src/inspect_evals/gdm_stealth/solver_pipeline.py`)
- OSWorld controller → OSWorld logging: logs actions (evidence: `src/inspect_evals/osworld/container/code/controller.py`)
- Submission engine → Paper metadata client: fetches paper data (evidence: `src/register_tooling/engine.py`)
- Submission engine → Evaluation derivation: derives listing (evidence: `src/register_tooling/engine.py`)
- Evaluation derivation → Paper metadata client: reads paper data (evidence: `src/register_tooling/deriver.py`)