# openai/mle-bench architecture

> Architecture diagram of the GitHub repository openai/mle-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/mle-bench
- Repository: https://github.com/openai/mle-bench

## Overview

##### Project Type and Purpose
- MLE-bench is a Python-based CLI framework and evaluation platform for benchmarking ML-engineered agents on Kaggle-style competitions. It provides commands to prepare datasets, run agents in isolated Docker environments, and grade their submissions.

##### File Structure Analysis
- Top-level directories:
- `mlebench/`: core library (CLI entrypoint, competition registry, data prep, grading logic)
- `agents/`: modular agent implementations, each in its own folder with Dockerfile, config, and runner scripts
- `environment/`: base Docker image and grading server code
- `experiments/`, `extras/`, `examples/`: supplementary scripts and utilities
- `runs/`: recorded outputs and grading reports
- `tests/`: unit and integration tests
- Docker-based microservices: each agent and the environment are defined by Dockerfiles (in `agents/*/Dockerfile` and `environment/Dockerfile`).
- CLI is implemented in `mlebench/cli.py` and exposed via `pyproject.toml` entry points.

##### Core Components
a. Frontend (User Interface)
- CLI (`mlebench prepare`, `mlebench grade`, etc.)
- Entrypoint: `mlebench/cli.py` → dispatches to subcommands
b. Core Library
- Competition registry & config (`mlebench/registry.py`)
- Data preparation modules (`mlebench/data.py`, `mlebench/utils.py`)
- Grading modules (`mlebench/grade.py`, `mlebench/grade_helpers.py`)
c. Data Storage
- Local filesystem / Git-LFS cache for raw Kaggle datasets
- Prepared train/test splits under cache directory
- Splits definitions (`experiments/splits/*.txt`)
d. External Services
- Kaggle API (external HTTP service) for downloading raw data
e. Docker Environments
- Base environment image (`environment/Dockerfile`, `grading_server.py`)
- Agent containers (`agents/*/Dockerfile`, `run_agent.py`)
f. Orchestration & Runner
- `run_agent.py` and `agents/run.py` launch agent containers against prepared data
- Grading server container validates submission structure
g. Outputs
- Submission JSONL files
- Grading reports (`runs/.../*.json`)

##### Relationships and Interactions
- CLI → (1) invokes data-prep modules → connects to Kaggle API → writes prepared datasets to local cache
- CLI → (2) builds/spawns base Docker image (environment) and grading server
- CLI → (3) builds & runs each agent’s Docker container, mounting prepared data and invoking agent logic
- Agent container → outputs submission CSVs/JSONL → CLI collects into a unified file
- CLI → (4) invokes grading modules against grading server → produces grading reports stored under `runs/`
- Tests → exercise CLI commands and core modules without Docker (integration/unit)

##### Architectural Patterns & Design Principles
- Plugin pattern: agents are discoverable via `agents/registry.py` and can be added without modifying core code
- Microservices via Docker: each agent and the grading server run in isolated containers
- CLI-orchestrator: the CLI acts as a thin orchestration layer over the core Python library and Docker tasks
- Separation of concerns: data prep, agent execution, grading logic, and experiments are clearly partitioned

##### Relevant Technologies & Frameworks
- Python 3.10+, Click or argparse for CLI
- Docker & Dockerfile for containerization
- Git-LFS for large dataset storage
- Kaggle API for external data retrieval
- Pytest for tests
- YAML/JSON for config and grading reports

## Diagram (Mermaid)

```mermaid
graph TB
    %% CLI Layer
    subgraph "CLI Layer"
        CLI["mlebench CLI"]:::cli
        PyProj["pyproject.toml"]:::cli
    end

    %% Core Library
    subgraph "Core Library (Backend Service)"
        Reg["Competition Registry & Config"]:::core
        CompDefs["Competition Definitions & Grading Hooks"]:::core
        DataPrep["Data Preparation Module"]:::core
        Utils["Utility Functions"]:::core
        GradeCore["Grading Core Logic"]:::core
        GradeHelp["Grading Helper Functions"]:::core
        Metrics["Metrics Utilities"]:::core
    end

    %% Data Storage
    subgraph "Local Data Storage"
        RawCache["Raw Kaggle Data (Git-LFS Cache)"]:::storage
        PreparedCache["Prepared Splits"]:::storage
        Splits["Experiment Splits Definitions"]:::storage
    end

    %% External Service
    Kaggle["Kaggle API"]:::external

    %% Docker Environments
    subgraph "Docker Environments"
        subgraph "Base Environment"
            EnvImage["Base Env Image"]:::container
            GradingServ["Grading Server"]:::container
        end
        subgraph "Agent Containers"
            AgentReg["Agent Plugin Registry & Runner"]:::container
            AgentsRun["Agent Launch Script"]:::container
            AIDE["Agent: aide"]:::container
            Dummy["Agent: dummy"]:::container
            MLAgent["Agent: mlagentbench"]:::container
            OpenDevIn["Agent: opendevin"]:::container
            AgentUtils["Agent Utility Functions"]:::container
        end
    end

    %% Outputs & Reports
    Submission["Submission Files (JSONL/CSV)"]:::storage
    Reports["Grading Reports"]:::storage
    RunsStore["Recorded Outputs & Reports Store"]:::storage

    %% Tests
    subgraph "Tests"
        UnitTests["Unit Tests"]:::core
        IntTests["Integration Tests"]:::core
    end

    %% Relationships
    CLI -->|"invokes data-prep"| DataPrep
    DataPrep -->|"downloads raw data"| Kaggle
    Kaggle -->|"stores raw data"| RawCache
    DataPrep -->|"writes splits"| PreparedCache
    PreparedCache --> Splits

    CLI -->|"builds & runs env"| EnvImage
    CLI -->|"builds & runs grading server"| GradingServ

    CLI -->|"builds & runs agents"| AgentReg
    AgentReg --> AgentsRun
    AgentsRun --> AIDE
    AgentsRun --> Dummy
    AgentsRun --> MLAgent
    AgentsRun --> OpenDevIn
    AIDE --> Submission
    Dummy --> Submission
    MLAgent --> Submission
    OpenDevIn --> Submission

    CLI -->|"collects submissions"| Submission
    CLI -->|"invokes grading"| GradeCore
    GradeCore --> GradingServ
    GradingServ -->|"returns report"| Reports
    Reports --> RunsStore

    %% Core internal links
    CLI --> Reg
    CLI --> CompDefs
    CLI --> Utils
    CLI --> GradeHelp
    CLI --> Metrics

    %% Tests relations
    UnitTests --> DataPrep
    UnitTests --> GradeCore
    IntTests --> CLI
    IntTests --> GradingServ

    %% Click Events
    click CLI "https://github.com/openai/mle-bench/blob/main/mlebench/cli.py"
    click PyProj "https://github.com/openai/mle-bench/blob/main/pyproject.toml"
    click Reg "https://github.com/openai/mle-bench/blob/main/mlebench/registry.py"
    click CompDefs "https://github.com/openai/mle-bench/tree/main/mlebench/competitions/"
    click DataPrep "https://github.com/openai/mle-bench/blob/main/mlebench/data.py"
    click Utils "https://github.com/openai/mle-bench/blob/main/mlebench/utils.py"
    click GradeCore "https://github.com/openai/mle-bench/blob/main/mlebench/grade.py"
    click GradeHelp "https://github.com/openai/mle-bench/blob/main/mlebench/grade_helpers.py"
    click Metrics "https://github.com/openai/mle-bench/blob/main/mlebench/metrics.py"
    click EnvImage "https://github.com/openai/mle-bench/tree/main/environment/Dockerfile"
    click GradingServ "https://github.com/openai/mle-bench/blob/main/environment/grading_server.py"
    click AgentReg "https://github.com/openai/mle-bench/blob/main/agents/registry.py"
    click AgentsRun "https://github.com/openai/mle-bench/blob/main/agents/run.py"
    click AgentUtils "https://github.com/openai/mle-bench/blob/main/agents/utils.py"
    click AIDE "https://github.com/openai/mle-bench/tree/main/agents/aide/Dockerfile"
    click Dummy "https://github.com/openai/mle-bench/tree/main/agents/dummy/Dockerfile"
    click MLAgent "https://github.com/openai/mle-bench/tree/main/agents/mlagentbench/Dockerfile"
    click OpenDevIn "https://github.com/openai/mle-bench/tree/main/agents/opendevin/Dockerfile"
    click RunsStore "https://github.com/openai/mle-bench/tree/main/runs/"

    %% Styles
    classDef cli fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef core fill:#90EE90,stroke:#333,stroke-width:1px
    classDef storage fill:#FFA500,stroke:#333,stroke-width:1px
    classDef container fill:#D8BFD8,stroke:#333,stroke-width:1px
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:1px
```
