How ukgovernmentbeis/control-arena works
Based on our review of both the file tree and the README, ControlArena is a research prototype framework aimed at mimicking real-world, high-stakes deployment environments and supporting control evaluations across multiple complex tasks. It incorporates a modular design with several major sectors such as protocols, settings (for different environments like Kubernetes-based “Infra”, as well as “Bash” and “Apps”), CLI tools, task definitions, and supporting libraries for logging, auditing, and utility functions.
Main Components
• Overall “ControlArena” core, which includes:
• The protocols layer (in control_arena/protocols) that defines how agents communicate, monitor activities, and enforce policies.
• The settings layer (in control_arena/settings) that provides definitions for different evaluation environments. Notably:
⁃ “Infra” for Kubernetes infrastructure (with a rich hierarchy of main and side tasks),
⁃ “Bash” for executing bash commands,
⁃ “Apps” for APPS-based coding tasks.
• The CLI and tooling components. For example, control_arena/cli hosts the main CLI functionality and there is a dedicated “human-trajectories” CLI in the scripts/human_trajectories directory.
• Log management modules (control_arena/log_store) which likely provide mechanisms for logging to GitHub, servers, etc.
• Utility helpers and testing frameworks (including tests directories, conftest.py, and utility scripts).
• Devcontainer and other configuration files that support environment setup, dependency management (like uv and Dockerfiles) and CI/CD (GitHub workflows).
Relationships and Interactions
• The CLI interacts with various components of the framework:
• It triggers environment setup, evaluation runs (e.g. via “uv run inspect eval …”), and launches the different settings.
• It also connects with protocol modules to choose the evaluation protocol (e.g. simple protocol, tm protocol, etc.) and with policies (attack, honest, fixed).
• Protocol modules and task definitions should be drawn as separate submodules, illustrating that protocols are built using smaller parts (like monitors and policies) that reside in protocol_parts.
• Bidirectional data flows:
• For example, control loop and task execution feed data into logging modules which record evaluations.
• The human trajectories CLI tools extract and generate evaluation trajectories that are later consumed by the evaluation engine.
Important Architectural Patterns
• Modularity: The design separates domain-specific logic into clearly isolated directories (protocols vs settings vs CLI vs tooling).
• Layering: There is a separation between core logic (control_arena) and environment/task-specific configuration (control_arena/settings), and a clear division between runtime interaction (CLI and human trajectories) and infrastructure tasks.
• Extensibility: The protocols module is built to be extended via subcomponents (as seen in the protocol_parts folder) so drawing these as “pluggable” modules is useful.
Relevant Technologies and Frameworks
• Python as the primary language, with dependency management by “uv”.
• Infrastructure tools: Docker (incorporated via Dockerfiles, devcontainer configuration, and infra/docker folder), Kubernetes (kubectl, kind, helm), and CI/CD pipelines (GitHub workflows).
• Testing and code quality via pre-commit configurations, pytest tests (tests directories), and linting configurations (e.g. .markdownlint.jsonc).
Capturing Essential Architectural Detail
• Separate and detail as many components as possible instead of lumping functionalities together.
• A block representing “control_arena” (with sub-blocks for protocols, settings, cli, log_store).
• A block for “scripts” and “tests”.
• Configuration blocks for “.devcontainer”, Dockerfiles, CI/CD files.
• Interactions: for example, the CLI triggers evaluations that use protocols defined in control_arena/protocols, which then interact with settings in control_arena/settings/infra, and results are logged in control_arena/log_store.
This breakdown should help ensure all essential architectural elements are captured in an easily understandable visual format.