# openai/swelancer-benchmark architecture

> Architecture diagram of the GitHub repository openai/swelancer-benchmark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openai/swelancer-benchmark
- Repository: https://github.com/openai/swelancer-benchmark

## Overview

The SWE-Lancer project is a research and evaluation framework built around real-world freelance software engineering tasks, as evidenced by the README and by the well‐structured subprojects (nanoeval, alcatraz, nanoeval_alcatraz) within the “project” directory. Its purpose is to study whether advanced language models can succeed—and even potentially earn $1 million—in freelance software engineering. While the repository also includes extensive issue directories to manage patches and reproducible bugs, the core of the system is built on three interrelated submodules and an orchestrating agent combined with Docker-based deployment.

##### Major Components
- Docker/Deployment Environment:
– Show Docker images (for different architectures, as swelancer and swelancer_x86) that encapsulate the entire runtime.
- Virtual Environment & Package Manager:
- Sub-modules (Projects):
– nanoeval: This subproject contains the core evaluation logic, wrappers for tasks, metric computations, and supporting libraries (e.g., evaluation scripts, examples, and helpers like json_recorder, async utilities, and a CLI interface).
– alcatraz: This module (with its clustering and computer interface components) implements the “ComputerInterface” that abstracts low-level compute tasks such as executing shell commands, file transfers, and environment management.
– nanoeval_alcatraz: An integration layer that combines nanoeval’s evaluation logic with alcatraz’s computer orchestration. It also provides its own tests and a clean interface for linking a user’s infrastructure (for example, customizing the ComputerInterface).
- Orchestration & Entry Points:
– swelancer.py and swelancer_agent.py serve as the orchestrators that kick off the workflow. They initialize compute tasks (via a custom ComputerInterface) and load configurations/environment variables (with help from sample.env and requirements.txt).
- Runtime Scripts:
- Environment and Configuration Management:
– Environment variables (via sample.env renamed to .env) ensure that sensitive information like API keys, Pusher credentials, etc., are injected into the system.

##### Interactions and Data Flows
- How the user (or an automated system) builds and runs the Docker container (selecting either the Dockerfile for ARM64 or the Dockerfile_x86 for Intel) and then executes “run_swelancer.py”.
- That the swelancer_agent (and its _start_computer function) interacts with the ComputerInterface implementations from alcatraz (or a custom implementation, if extended) to execute shell commands (upload/download/cleanup) on external compute infrastructure.
- Nanoeval and nanoeval_alcatraz modules to show that evaluation routines (like computed tasks, metrics, and recordings) flow into alcatraz’s compute orchestration layer, which then returns results to the orchestrator.

##### Architectural Patterns and Principles
- Modularity: Segmenting the project into distinct, autonomous modules (nanoeval, alcatraz, nanoeval_alcatraz) makes the architecture modular and extensible.
- Abstraction Layers: The ComputerInterface abstract class is key; it decouples the task orchestration (swelancer_agent) from any one specific compute backend.
- Containerization: The system uses Docker to encapsulate deployment ensuring consistency between development and production.
- External Integration: Environment variables and prebuilt virtualenvs (using uv) are used to interface securely with external APIs (e.g., OpenAI API, pusher) and networking resources.

##### Technologies, Frameworks, and Tools
- Python 3.11 (mandatory version)
- Docker and buildx for creating container images tailored to the target architecture.
- uv for lightweight package management and virtual environment handling.
- Standard libraries and asynchronous programming (async/await) for running computer tasks.
- YAML configuration files in the runtime_scripts folder for deployment workflows.

## Diagram (Mermaid)

```mermaid
graph TD
    A["User/CI"]:::user

    subgraph "Deployment Layer"
        L["Dockerfile"]:::deployment
        M["Dockerfile_x86"]:::deployment
        B["Docker Build Environment"]:::deployment
        C["Docker Container"]:::deployment
        D["Virtual Environment (uv)"]:::deployment
        subgraph "Configuration"
            E1["Environment (.env)"]:::deployment
            E2["Requirements"]:::deployment
        end
        F["Runtime Scripts"]:::deployment
    end

    subgraph "Submodules"
        G["nanoeval Module"]:::submodule
        I["nanoeval_alcatraz Integration"]:::submodule
        H["alcatraz Module"]:::submodule
    end

    subgraph "Orchestration"
        J["swelancer.py"]:::orchestration
        K["swelancer_agent.py"]:::orchestration
    end

    A -->|"triggers"| B
    L -->|"defines"| B
    M -->|"defines"| B
    B -->|"builds"| C
    E1 -->|"injects"| C
    E2 -->|"injects"| C
    C -->|"hosts"| D
    C -->|"runs"| F
    C -->|"initializes"| J
    C -->|"initializes"| K

    G -->|"evaluation"| I
    I -->|"integration"| H
    H -->|"compute"| K
    F -->|"supports"| J

    classDef user fill:#f9f,stroke:#333,stroke-width:2px;
    classDef deployment fill:#bbf,stroke:#333,stroke-width:2px;
    classDef orchestration fill:#bfb,stroke:#333,stroke-width:2px;
    classDef submodule fill:#fbf,stroke:#333,stroke-width:2px;

click L "https://github.com/openai/swelancer-benchmark/tree/main/Dockerfile"
click M "https://github.com/openai/swelancer-benchmark/tree/main/Dockerfile_x86"
click D "https://github.com/openai/swelancer-benchmark/blob/main/uv.lock"
click G "https://github.com/openai/swelancer-benchmark/tree/main/project/nanoeval/"
click H "https://github.com/openai/swelancer-benchmark/tree/main/project/alcatraz/"
click I "https://github.com/openai/swelancer-benchmark/tree/main/project/nanoeval_alcatraz/"
click J "https://github.com/openai/swelancer-benchmark/blob/main/swelancer.py"
click K "https://github.com/openai/swelancer-benchmark/blob/main/swelancer_agent.py"
click F "https://github.com/openai/swelancer-benchmark/tree/main/runtime_scripts/"
click E1 "https://github.com/openai/swelancer-benchmark/blob/main/sample.env"
click E2 "https://github.com/openai/swelancer-benchmark/blob/main/requirements.txt"
```
