# openai/preparedness architecture

> Architecture diagram of the GitHub repository openai/preparedness, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/preparedness
- Repository: https://github.com/openai/preparedness

## Overview

##### Project Type and Purpose
- The repository is an evaluation framework containing multiple “evals” (e.g., PaperBench and forthcoming SWELancer and MLE-bench). It leverages two key engines: nanoeval (an evaluation engine with asynchronous task management, solvers, and various utility modules) and alcatraz (which appears responsible for cluster or execution environments).
- PaperBench is a specific eval system with extensive submodules for agents, experiments, grading, judging, GUI, and data handling.

##### File Structure
- Top-Level Projects: The “project” directory contains several subprojects:
- alcatraz (with modules for clusters and utilities)
- nanoeval (an evaluation and task management engine including async utilities, solvers, and monitoring)
- nanoeval_alcatraz (a bridging module between nanoeval and alcatraz)
- paperbench (a comprehensive evaluation framework that appears to handle experiments, agents, grading, judging, GUI, and data storage)
- Other directories include configuration files (.github workflows, pre-commit, pyright, etc.) that indicate continuous integration, code linting, and quality checks.
- The presence of Dockerfiles (for agents and graders) shows containerization for deployment of some system components.

##### Key Architectural Components
- Evaluation Engines:
- Nanoeval: Core evaluation logic managing tasks, asynchronous processing (e.g., monitor, executor worker, persistent DB), and solvers (including computer tasks and MCQ solvers).
- Alcatraz: Cluster management and orchestration (handling container processes, serialization, different cluster modules).
- The Bridge (nanoeval_alcatraz) that ties together the evaluation logic of nanoeval with alcatraz’s execution frameworks.
- PaperBench Framework:
- Agents: Several agent types (e.g., aisi-basic-agent, dummy, human) for running evaluation experiments.
- Judge Module: Responsible for grading and evaluation logic which coordinates with data from experiments.
- GUI: The paperbench GUI front-end for display and interactive use.
- Supporting Modules: Monitor, rubric, scripts, and experiment directories to run performance, cost, and judge evaluations.
- Supporting Infrastructure:
- CI/CD: GitHub workflows (ci.yml, pyright.yml, pytest.yml) handling testing, type checking and continuous integration.
- Configuration and tooling: Pre-commit settings, linting configurations, and Docker configurations.

##### Relationships and Interactions
- Data Flow: An evaluation task starts within the nanoeval engine, possibly mediated by alcatraz in a cluster/distributed environment. The nanoeval_alcatraz bridge ensures smooth integration between task execution and environment management.
- PaperBench then utilizes the evaluation outputs to run comprehensive experiments. Its agent modules and judge modules interact with nanoeval to fetch data, execute tasks, and compare results against expected outputs (as seen in the “data” directories with grading outcomes).
- The GUI, monitor, and rubric modules in paperbench present the results and metrics, closing the loop with user inputs and potential feedback.
- CI/CD and deployment tools ensure that all these components work together with consistent code quality and integration.

##### Final Elements
- Clear labels for each major component (e.g., “Nanoeval Engine”, “Alcatraz Clusters”, “PaperBench Agents”, “PaperBench Judge”, “GUI & Monitor”, “CI/CD Pipeline”).

## Diagram (Mermaid)

```mermaid
flowchart TD
    A("Nanoeval Engine"):::evaluation
    B("Nanoeval-Alcatraz Bridge"):::integration
    C("Alcatraz Cluster Manager"):::evaluation
    J("CI/CD Pipeline"):::ci

    subgraph "PaperBench Framework"
        D("PaperBench Agents"):::app
        E("Judge Module"):::app
        F("GUI Front-End"):::app
        G("Monitor Module"):::app
        H("Rubric & Related Tasks"):::app
        I("Experiments & Data Handling"):::app
    end

    A -->|"bridges"| B
    B -->|"executes"| C
    A -->|"sends_tasks"| D
    D -->|"reports_to"| E
    I -->|"supplies_data"| E
    H -->|"supports"| E
    E -->|"updates"| F
    G -->|"monitors"| F

    J -->|"tests"| A
    J -->|"tests"| C
    J -->|"tests"| D
    J -->|"integrates"| E
    J -->|"ensures_quality"| F

    classDef evaluation fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef integration fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef app fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef ci fill:#F5B7B1,stroke:#922B21,stroke-width:2px;

    click A "https://github.com/openai/preparedness/tree/main/project/nanoeval/nanoeval"
    click B "https://github.com/openai/preparedness/tree/main/project/nanoeval_alcatraz/nanoeval_alcatraz"
    click C "https://github.com/openai/preparedness/tree/main/project/alcatraz/alcatraz"
    click D "https://github.com/openai/preparedness/tree/main/project/paperbench/paperbench/agents"
    click E "https://github.com/openai/preparedness/tree/main/project/paperbench/paperbench/judge"
    click F "https://github.com/openai/preparedness/tree/main/project/paperbench/paperbench/gui"
    click G "https://github.com/openai/preparedness/tree/main/project/paperbench/paperbench/monitor"
    click H "https://github.com/openai/preparedness/tree/main/project/paperbench/paperbench/rubric"
    click I "https://github.com/openai/preparedness/tree/main/project/paperbench/experiments"
    click J "https://github.com/openai/preparedness/tree/main/.github/workflows"
```
