How ukgovernmentbeis/inspect_evals works
Inspect Evals is a Python library of Inspect AI benchmark tasks. A user selects or imports an evaluation, runs it against a configured model, and reviews the resulting logs; individual evaluations supply their own datasets, solvers, and scoring logic. The repository also includes trajectory-analysis scanners and a register-submission workflow for sharing evaluations. The graph preserves the major sampled benchmark, safety, and registration stages. Many benchmark modules are unsampled, so their internal connections are omitted unless documented or listed as references in the source index.
Main entry points: mmlu.py MMLU evaluation · huggingface.py Hugging Face utilities · engine.py Submission engine · apps.py APPS task
Components 24
Evaluation tasks
- APPS task
src/inspect_evals/apps/apps.py
Benchmark data
- APPS dataset
src/inspect_evals/apps/huggingface_artifact/apps.py - BBEH dataset
src/inspect_evals/bbeh/data.py - MMLU evaluation
src/inspect_evals/mmlu/mmlu.py - LiveCodeBench data
src/inspect_evals/livecodebench_pro/dataset.py - StereoSet evaluation
src/inspect_evals/stereoset/stereoset.py - SciKnowEval task
src/inspect_evals/sciknoweval/sciknoweval.py - SciKnowEval scorer
src/inspect_evals/sciknoweval/scorer.py - Abstention judge prompts
src/inspect_evals/abstention_bench/recipe/evaluation_judge_prompts.py
Safety and agents
- OSWorld controller
src/inspect_evals/osworld/container/code/controller.py - Stealth solver
src/inspect_evals/gdm_stealth/solver_pipeline.py - Customer support environment
src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/customer_support_env.py - Oversight rules
src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/oversight.py - Support tickets
src/inspect_evals/gdm_stealth/solver/strategic_rule_breaking/ticket.py - Trajectory scanners
agent_artefacts/trajectory_analysis/inspect_scout/scanners.py
Shared runtime
- Eval metadata
src/inspect_evals/metadata.py - Hugging Face utilities
src/inspect_evals/utils/huggingface.py - OSWorld logging
src/inspect_evals/osworld/container/code/logging.py
Evaluation register
- Submission engine
src/register_tooling/engine.py - Paper metadata client
src/register_tooling/arxiv_client.py - Evaluation derivation
src/register_tooling/deriver.py
Other components
- Evaluator
- Inspect AI
- Evaluation logs
Connections 15
- Evaluator → Inspect AI: runs tasks
README.md - Inspect AI → APPS task: runs evaluation
README.md - Inspect AI → MMLU evaluation: runs evaluation
README.md - Inspect AI → Evaluation logs: writes logs
README.md - APPS task → APPS dataset: loads dataset
src/inspect_evals/apps/apps.py - BBEH dataset → Hugging Face utilities: uses utilities
src/inspect_evals/bbeh/data.py - MMLU evaluation → Eval metadata: uses metadata
src/inspect_evals/mmlu/mmlu.py - MMLU evaluation → Hugging Face utilities: uses utilities
src/inspect_evals/mmlu/mmlu.py - LiveCodeBench data → Hugging Face utilities: uses utilities
src/inspect_evals/livecodebench_pro/dataset.py - SciKnowEval task → SciKnowEval scorer: uses scorer
src/inspect_evals/sciknoweval/sciknoweval.py - Stealth solver → Customer support environment: uses environment
src/inspect_evals/gdm_stealth/solver_pipeline.py - OSWorld controller → OSWorld logging: logs actions
src/inspect_evals/osworld/container/code/controller.py - Submission engine → Paper metadata client: fetches paper data
src/register_tooling/engine.py - Submission engine → Evaluation derivation: derives listing
src/register_tooling/engine.py - Evaluation derivation → Paper metadata client: reads paper data
src/register_tooling/deriver.py