# swe-bench/swe-bench architecture

> Architecture diagram of the GitHub repository swe-bench/swe-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swe-bench/swe-bench
- Repository: https://github.com/swe-bench/swe-bench

## Overview

##### Main Components Identification

a) Collection System (/swebench/collect/)
- Task collection pipeline
- Dataset building components
- Repository management

b) Inference System (/swebench/inference/)
- Model integration (LLaMA support)
- Dataset processing
- API runners

c) Evaluation Harness (/swebench/harness/)
- Docker-based evaluation environment
- Test execution
- Results processing

**┌─────────────────────────────────────────────┐**
- User Interface              │
- (CLI Commands / Python API / HuggingFace)  │
- ┘
- ↓
**┌─────────────────────────────────────────────┐**
- Core SWE-bench System            │
- ┬─────────────┬────────────────┤
- Collection │  Inference  │    Harness     │
- System    │   System    │    System      │
- ┴─────────────┴────────────────┘
- ↓
**┌─────────────────────────────────────────────┐**
- External Dependencies             │
- (Docker, GitHub, HuggingFace, PyPI)     │
- ┘

##### Component Details

For each major subsystem, include these elements:

Collection System (Blue Components):
- GitHub issue crawler
- Dataset builder
- Repository manager
- Version control integration

Inference System (Green Components):
- LLaMA model integration
- Dataset processors
- BM25 retrieval system
- API runners

Evaluation Harness (Orange Components):
- Docker container manager
- Test executor
- Results processor
- Grading system

##### Data Flow Representation

- Issue collection → Dataset creation
- Model inference → Patch generation
- Evaluation execution → Results processing

##### External Integration Points

Connections to:
- Docker containers
- GitHub repositories
- HuggingFace datasets
- Language models (LLaMA, GPT-4)

##### Additional Elements

Include:
- Version information
- System requirements
- Key metrics collection points

##### Technical Details

Specify:
- Docker container architecture
- Model integration points
- Dataset processing pipeline
- Evaluation metrics collection

Remember to include the system requirements mentioned in the README (120GB storage, 16GB RAM, 8 CPU cores) as technical constraints in a notes section.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Styles
    classDef ui fill:#e1e1e1,stroke:#333,stroke-width:2px
    classDef collection fill:#a8d1f0,stroke:#333,stroke-width:2px
    classDef inference fill:#a8f0c6,stroke:#333,stroke-width:2px
    classDef harness fill:#f0cba8,stroke:#333,stroke-width:2px
    classDef external fill:#d8d8d8,stroke:#333,stroke-width:2px
    
    %% User Interface Layer
    UI["User Interface<br/>(CLI/Python API/HuggingFace)"]:::ui
    
    %% Core Systems
    subgraph CoreSystems["Core SWE-bench System"]
        %% Collection System
        subgraph CollectionSystem["Collection System"]
            TCP["Task Collection Pipeline"]:::collection
            DB["Dataset Builder"]:::collection
            RM["Repository Manager"]:::collection
            VCI["Version Control Integration"]:::collection
            CU["Cleanup Utilities"]:::collection
        end
        
        %% Inference System
        subgraph InferenceSystem["Inference System"]
            LMI["LLaMA Model Integration"]:::inference
            DPS["Dataset Processing System"]:::inference
            BM25["BM25 Retrieval System"]:::inference
            APIRun["API Runners"]:::inference
            LIR["Live Inference Runner"]:::inference
            DCT["Dataset Creation Tools"]:::inference
        end
        
        %% Evaluation System
        subgraph HarnessSystem["Evaluation Harness"]
            DCM["Docker Container Manager"]:::harness
            TE["Test Executor"]:::harness
            RP["Results Processor"]:::harness
            DU["Docker Utilities"]:::harness
        end
    end
    
    %% External Dependencies
    GH["GitHub"]:::external
    Docker["Docker"]:::external
    HF["HuggingFace"]:::external
    PyPI["PyPI"]:::external
    
    %% Relationships
    UI --> TCP
    UI --> APIRun
    UI --> TE
    
    TCP --> DB
    DB --> RM
    RM --> VCI
    
    DPS --> BM25
    BM25 --> DCT
    DCT --> APIRun
    APIRun --> LIR
    LMI --> APIRun
    
    DCM --> TE
    TE --> RP
    DCM --> DU
    
    GH --> TCP
    Docker --> DCM
    HF --> LMI
    
    %% Click Events
    click TCP "https://github.com/swe-bench/SWE-bench/blob/main/swebench/collect/get_tasks_pipeline.py"
    click DB "https://github.com/swe-bench/SWE-bench/blob/main/swebench/collect/build_dataset.py"
    click RM "https://github.com/swe-bench/SWE-bench/tree/main/swebench/collect/make_repo/"
    click LMI "https://github.com/swe-bench/SWE-bench/tree/main/swebench/inference/llamao/"
    click DPS "https://github.com/swe-bench/SWE-bench/tree/main/swebench/inference/make_datasets/"
    click BM25 "https://github.com/swe-bench/SWE-bench/blob/main/swebench/inference/make_datasets/bm25_retrieval.py"
    click APIRun "https://github.com/swe-bench/SWE-bench/blob/main/swebench/inference/run_api.py"
    click DCM "https://github.com/swe-bench/SWE-bench/blob/main/swebench/harness/docker_build.py"
    click TE "https://github.com/swe-bench/SWE-bench/blob/main/swebench/harness/run_evaluation.py"
    click RP "https://github.com/swe-bench/SWE-bench/blob/main/swebench/harness/grading.py"
    click DU "https://github.com/swe-bench/SWE-bench/blob/main/swebench/harness/docker_utils.py"
    click VCI "https://github.com/swe-bench/SWE-bench/tree/main/swebench/versioning/"
    click DCT "https://github.com/swe-bench/SWE-bench/blob/main/swebench/inference/make_datasets/create_instance.py"
    click LIR "https://github.com/swe-bench/SWE-bench/blob/main/swebench/inference/run_live.py"
    click CU "https://github.com/swe-bench/SWE-bench/tree/main/swebench/collect/cleanup/"
    
    %% Legend
    subgraph Legend
        L1["UI Components"]:::ui
        L2["Collection Components"]:::collection
        L3["Inference Components"]:::inference
        L4["Harness Components"]:::harness
        L5["External Services"]:::external
    end
```
