How openai/mle-bench works
Project Type and Purpose
• MLE-bench is a Python-based CLI framework and evaluation platform for benchmarking ML-engineered agents on Kaggle-style competitions. It provides commands to prepare datasets, run agents in isolated Docker environments, and grade their submissions.
File Structure Analysis
• Top-level directories:
• mlebench/: core library (CLI entrypoint, competition registry, data prep, grading logic)
• agents/: modular agent implementations, each in its own folder with Dockerfile, config, and runner scripts
• environment/: base Docker image and grading server code
• experiments/, extras/, examples/: supplementary scripts and utilities
• runs/: recorded outputs and grading reports
• tests/: unit and integration tests
• Docker-based microservices: each agent and the environment are defined by Dockerfiles (in agents/*/Dockerfile and environment/Dockerfile).
• CLI is implemented in mlebench/cli.py and exposed via pyproject.toml entry points.
Core Components
a. Frontend (User Interface)
• CLI (mlebench prepare, mlebench grade, etc.)
• Entrypoint: mlebench/cli.py → dispatches to subcommands
b. Core Library
• Competition registry & config (mlebench/registry.py)
• Data preparation modules (mlebench/data.py, mlebench/utils.py)
• Grading modules (mlebench/grade.py, mlebench/grade_helpers.py)
c. Data Storage
• Local filesystem / Git-LFS cache for raw Kaggle datasets
• Prepared train/test splits under cache directory
• Splits definitions (experiments/splits/*.txt)
d. External Services
• Kaggle API (external HTTP service) for downloading raw data
e. Docker Environments
• Base environment image (environment/Dockerfile, grading_server.py)
• Agent containers (agents/*/Dockerfile, run_agent.py)
f. Orchestration & Runner
• run_agent.py and agents/run.py launch agent containers against prepared data
• Grading server container validates submission structure
g. Outputs
• Submission JSONL files
• Grading reports (runs/.../*.json)
Relationships and Interactions
• CLI → (1) invokes data-prep modules → connects to Kaggle API → writes prepared datasets to local cache
• CLI → (2) builds/spawns base Docker image (environment) and grading server
• CLI → (3) builds & runs each agent’s Docker container, mounting prepared data and invoking agent logic
• Agent container → outputs submission CSVs/JSONL → CLI collects into a unified file
• CLI → (4) invokes grading modules against grading server → produces grading reports stored under runs/
• Tests → exercise CLI commands and core modules without Docker (integration/unit)
Architectural Patterns & Design Principles
• Plugin pattern: agents are discoverable via agents/registry.py and can be added without modifying core code
• Microservices via Docker: each agent and the grading server run in isolated containers
• CLI-orchestrator: the CLI acts as a thin orchestration layer over the core Python library and Docker tasks
• Separation of concerns: data prep, agent execution, grading logic, and experiments are clearly partitioned
Relevant Technologies & Frameworks
• Python 3.10+, Click or argparse for CLI
• Docker & Dockerfile for containerization
• Git-LFS for large dataset storage
• Kaggle API for external data retrieval
• Pytest for tests
• YAML/JSON for config and grading reports