How swe-bench/swe-bench works
Main Components Identification
a) Collection System (/swebench/collect/)
• Task collection pipeline
• Dataset building components
• Repository management
b) Inference System (/swebench/inference/)
• Model integration (LLaMA support)
• Dataset processing
• API runners
c) Evaluation Harness (/swebench/harness/)
• Docker-based evaluation environment
• Test execution
• Results processing
┌─────────────────────────────────────────────┐
• User Interface │
• (CLI Commands / Python API / HuggingFace) │
• ┘
• ↓
┌─────────────────────────────────────────────┐
• Core SWE-bench System │
• ┬─────────────┬────────────────┤
• Collection │ Inference │ Harness │
• System │ System │ System │
• ┴─────────────┴────────────────┘
• ↓
┌─────────────────────────────────────────────┐
• External Dependencies │
• (Docker, GitHub, HuggingFace, PyPI) │
• ┘
Component Details
For each major subsystem, include these elements:
Collection System (Blue Components):
• GitHub issue crawler
• Dataset builder
• Repository manager
• Version control integration
Inference System (Green Components):
• LLaMA model integration
• Dataset processors
• BM25 retrieval system
• API runners
Evaluation Harness (Orange Components):
• Docker container manager
• Test executor
• Results processor
• Grading system
Data Flow Representation
• Issue collection → Dataset creation
• Model inference → Patch generation
• Evaluation execution → Results processing
External Integration Points
Connections to:
• Docker containers
• GitHub repositories
• HuggingFace datasets
• Language models (LLaMA, GPT-4)
Additional Elements
Include:
• Version information
• System requirements
• Key metrics collection points
Technical Details
Specify:
• Docker container architecture
• Model integration points
• Dataset processing pipeline
• Evaluation metrics collection
Remember to include the system requirements mentioned in the README (120GB storage, 16GB RAM, 8 CPU cores) as technical constraints in a notes section.