# aider-ai/aider-swe-bench architecture

> Architecture diagram of the GitHub repository aider-ai/aider-swe-bench, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aider-ai/aider-swe-bench
- Repository: https://github.com/aider-ai/aider-swe-bench

## Overview

##### Project type and purpose
- This is a command‑line benchmarking harness for running and evaluating the “aider” AI coding assistant on the SWE Bench Lite suite of programming problems.
- Its goal is to repeatedly invoke aider on each problem’s repository (using different LLM models), record the outputs (diffs and metadata), then evaluate those outputs against held‑out acceptance tests and produce summary reports.

##### File structure
- Top‑level scripts:
– harness.py: core agent that clones repos, runs aider, retries until a plausible solution is found, and writes per‑instance JSON predictions.
– report.py: aggregates predictions into a JSONL, runs acceptance tests via SWE Bench Docker, and emits logs and a summary results.json.
– compare.py, dump.py, table.py: auxiliary CLI tools for comparing runs, dumping data, and generating result tables.
– run_eval.sh: shell wrapper around the evaluation workflow.
– tests.py: unit tests for the harness and utilities.
– utils.py: shared helper functions (filesystem paths, subprocess calls, JSON I/O, etc.).
- Directory swebench_docker: checked‑in or submodule Docker images and scripts for running the SWE Bench testbed.
- predictions/: filesystem storage for all raw JSON prediction files, per‑model and per‑problem subdirectories.

##### README for additional insights
- Explains methodology: alternating models (gpt‑4o, opus), retry logic, acceptance criteria (no syntax/lint/test errors), selection of “least bad” solution when all attempts fail.
- Distinguishes two phases:
1) prediction generation (harness.py)
2) evaluation/reporting (report.py)

##### Main components and interactions
a. Components
- CLI Layer: harness.py, report.py, compare.py, dump.py, table.py, run_eval.sh
- Agent Engine (harness): git checkout → invoke aider → parse aider outputs → write prediction JSON
- Evaluation Engine (report): read JSONL → spin up Docker containers via swebench_docker → run acceptance tests → collect eval logs → write results.json
- Storage: local filesystem under predictions/ and logs/
- External Services: LLM model endpoints (OpenAI, Anthropic, etc.) accessed by aider
- Utilities: utils.py, tests.py for internal checks
b. Relationships and data flow
1. User runs harness.py (or run_eval.sh).
2. For each SWE Bench problem:
– harness clones problem repo into temp directory
– harness calls aider with model, test command, and “accept all” flag
– aider communicates with external LLM APIs, returns diffs and test outcomes
– harness writes a per‑instance JSON prediction file to predictions/
3. When all predictions generated, user runs report.py:
– report.py reads all per‑instance JSONs, bundles into all_preds.jsonl
– report.py invokes Docker‐based testbed (swebench_docker) to run acceptance tests on each predicted solution
– Docker returns pass/fail logs → report.py writes logs/ and results.json
4. Auxiliary scripts (compare.py, dump.py, table.py) read predictions/ and results.json to produce summaries or tables.
c. Architectural patterns and design principles
- Pipeline/pipeline‐stage pattern: separation of prediction generation and evaluation stages.
- Batch processing via filesystem as queue (one JSON file per instance).
- Modularity: each CLI tool has a single responsibility.
- Retry logic and fallback selection in harness.py encapsulate robustness principles.
d. Key technologies and frameworks
- Python 3 CLI scripts
- Docker for isolated evaluation (swebench_docker)
- External LLM APIs via aider (OpenAI GPT‑4o, Anthropic Claude, etc.)
- pytest for harness unit tests
- JSON/JSONL for data interchange
- Shell scripting for orchestration (run_eval.sh)

##### Detail without overthinking
- Break the system into as many logical components as needed to show responsibilities and data flows clearly.
- To show both file system artifacts and external dependencies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer" 
        direction TB
        RunEval["run_eval.sh"]:::cli
        Harness["Harness Agent\n(harness.py)"]:::compute
        Report["Report Generator\n(report.py)"]:::compute
        Auxiliary["Auxiliary Tools"]:::cli
        Replay["Replay Tool\n(replay.py)"]:::cli
    end

    subgraph "Auxiliary Tools"
        direction TB
        Compare["compare.py"]:::cli
        Dump["dump.py"]:::cli
        Table["table.py"]:::cli
    end

    %% Agent Engine internals
    subgraph "Agent Engine"
        direction TB
        Harness
        Aider["Aider (LLM API)"]:::external
        Utils["Utilities\n(utils.py)"]:::util
        Tests["Unit Tests\n(tests.py)"]:::tests
    end

    %% Storage
    subgraph "Storage"
        direction TB
        Predictions["Predictions\n(predictions/)"]:::storage
        LogsResults["Logs & Results\n(logs/, results.json)"]:::storage
    end

    %% Evaluation Engine
    subgraph "Evaluation Engine"
        direction TB
        Report
        Docker["Docker Testbed\n(swebench_docker/)"]:::external
    end

    %% Config & Metadata
    subgraph "Config & Metadata"
        direction TB
        Requirements["requirements.txt"]:::config
        Flake8[".flake8"]:::config
        PreCommit[".pre-commit-config.yaml"]:::config
        Readme["README.md"]:::meta
        Contrib["CONTRIBUTING.md"]:::meta
        License["LICENSE.txt"]:::meta
    end

    %% Connections
    RunEval -->|starts| Harness
    Harness -->|"writes JSON"| Predictions
    Harness -->|invokes| Aider
    Harness -->|uses| Utils
    Harness -->|tested by| Tests

    Predictions -->|input| Report
    Report -->|spins up| Docker
    Docker -->|returns pass/fail logs| LogsResults
    Report -->|writes results.json| LogsResults

    %% Auxiliary flows
    Compare -->|reads| Predictions
    Compare -->|reads| LogsResults
    Dump -->|reads| Predictions
    Table -->|reads| LogsResults
    Replay -->|re-runs| Harness

    %% Class Definitions
    classDef cli fill:#E8F1FA,stroke:#005B96,color:#005B96;
    classDef compute fill:#FFF3E0,stroke:#FB8C00,color:#E65100;
    classDef storage fill:#E8F5E9,stroke:#2E7D32,color:#1B5E20;
    classDef external fill:#F3E5F5,stroke:#6A1B9A,color:#4A148C,stroke-dasharray: 5 5;
    classDef util fill:#E0F7FA,stroke:#006064,color:#004D40;
    classDef tests fill:#FFEBEE,stroke:#C62828,color:#B71C1C,stroke-dasharray: 2 2;
    classDef config fill:#ECEFF1,stroke:#37474F,color:#263238;
    classDef meta fill:#FFFDE7,stroke:#F9A825,color:#F57F17;

    %% Click Events
    click Harness "https://github.com/aider-ai/aider-swe-bench/blob/main/harness.py"
    click Report "https://github.com/aider-ai/aider-swe-bench/blob/main/report.py"
    click Compare "https://github.com/aider-ai/aider-swe-bench/blob/main/compare.py"
    click Dump "https://github.com/aider-ai/aider-swe-bench/blob/main/dump.py"
    click Table "https://github.com/aider-ai/aider-swe-bench/blob/main/table.py"
    click RunEval "https://github.com/aider-ai/aider-swe-bench/blob/main/run_eval.sh"
    click Docker "https://github.com/aider-ai/aider-swe-bench/tree/main/swebench_docker/"
    click Predictions "https://github.com/aider-ai/aider-swe-bench/tree/main/predictions/"
    click Utils "https://github.com/aider-ai/aider-swe-bench/blob/main/utils.py"
    click Tests "https://github.com/aider-ai/aider-swe-bench/blob/main/tests.py"
    click Replay "https://github.com/aider-ai/aider-swe-bench/blob/main/replay.py"
    click Requirements "https://github.com/aider-ai/aider-swe-bench/blob/main/requirements.txt"
    click Flake8 "https://github.com/aider-ai/aider-swe-bench/blob/main/.flake8"
    click PreCommit "https://github.com/aider-ai/aider-swe-bench/blob/main/.pre-commit-config.yaml"
    click Readme "https://github.com/aider-ai/aider-swe-bench/blob/main/README.md"
    click Contrib "https://github.com/aider-ai/aider-swe-bench/blob/main/CONTRIBUTING.md"
    click License "https://github.com/aider-ai/aider-swe-bench/blob/main/LICENSE.txt"
```
