How aider-ai/aider-swe-bench works
Project type and purpose
• This is a command‑line benchmarking harness for running and evaluating the “aider” AI coding assistant on the SWE Bench Lite suite of programming problems.
• Its goal is to repeatedly invoke aider on each problem’s repository (using different LLM models), record the outputs (diffs and metadata), then evaluate those outputs against held‑out acceptance tests and produce summary reports.
File structure
• Top‑level scripts:
– harness.py: core agent that clones repos, runs aider, retries until a plausible solution is found, and writes per‑instance JSON predictions.
– report.py: aggregates predictions into a JSONL, runs acceptance tests via SWE Bench Docker, and emits logs and a summary results.json.
– compare.py, dump.py, table.py: auxiliary CLI tools for comparing runs, dumping data, and generating result tables.
– run_eval.sh: shell wrapper around the evaluation workflow.
– tests.py: unit tests for the harness and utilities.
– utils.py: shared helper functions (filesystem paths, subprocess calls, JSON I/O, etc.).
• Directory swebench_docker: checked‑in or submodule Docker images and scripts for running the SWE Bench testbed.
• predictions/: filesystem storage for all raw JSON prediction files, per‑model and per‑problem subdirectories.
README for additional insights
• Explains methodology: alternating models (gpt‑4o, opus), retry logic, acceptance criteria (no syntax/lint/test errors), selection of “least bad” solution when all attempts fail.
• Distinguishes two phases:
1) prediction generation (harness.py)
2) evaluation/reporting (report.py)
Main components and interactions
a. Components
• CLI Layer: harness.py, report.py, compare.py, dump.py, table.py, run_eval.sh
• Agent Engine (harness): git checkout → invoke aider → parse aider outputs → write prediction JSON
• Evaluation Engine (report): read JSONL → spin up Docker containers via swebench_docker → run acceptance tests → collect eval logs → write results.json
• Storage: local filesystem under predictions/ and logs/
• External Services: LLM model endpoints (OpenAI, Anthropic, etc.) accessed by aider
• Utilities: utils.py, tests.py for internal checks
b. Relationships and data flow
1. User runs harness.py (or run_eval.sh).
2. For each SWE Bench problem:
– harness clones problem repo into temp directory
– harness calls aider with model, test command, and “accept all” flag
– aider communicates with external LLM APIs, returns diffs and test outcomes
– harness writes a per‑instance JSON prediction file to predictions/
3. When all predictions generated, user runs report.py:
– report.py reads all per‑instance JSONs, bundles into all_preds.jsonl
– report.py invokes Docker‐based testbed (swebench_docker) to run acceptance tests on each predicted solution
– Docker returns pass/fail logs → report.py writes logs/ and results.json
4. Auxiliary scripts (compare.py, dump.py, table.py) read predictions/ and results.json to produce summaries or tables.
c. Architectural patterns and design principles
• Pipeline/pipeline‐stage pattern: separation of prediction generation and evaluation stages.
• Batch processing via filesystem as queue (one JSON file per instance).
• Modularity: each CLI tool has a single responsibility.
• Retry logic and fallback selection in harness.py encapsulate robustness principles.
d. Key technologies and frameworks
• Python 3 CLI scripts
• Docker for isolated evaluation (swebench_docker)
• External LLM APIs via aider (OpenAI GPT‑4o, Anthropic Claude, etc.)
• pytest for harness unit tests
• JSON/JSONL for data interchange
• Shell scripting for orchestration (run_eval.sh)
Detail without overthinking
• Break the system into as many logical components as needed to show responsibilities and data flows clearly.
• To show both file system artifacts and external dependencies.