# dreamyang-liu/storm architecture

> Architecture diagram of the GitHub repository dreamyang-liu/storm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dreamyang-liu/storm
- Repository: https://github.com/dreamyang-liu/storm

## Overview

STORM is a multi-agent benchmark runner for code implementation (Commit0) and paper reproduction (PaperBench), built on the OpenHands SDK. The principal workflow selects a task and configuration, runs either a single-agent or multi-agent experiment in a workspace, then records results and— for PaperBench—can evaluate them with a judge. The tree and README establish the product branches; the source index confirms the inference entry point’s task, manager, and patch dependencies. Several internal links are documented at a high level because their implementation excerpts were not provided.

Main entry points: [`STORM/run_infer.py`](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/run_infer.py) (Inference runner), [`STORM/core/manager.py`](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/manager.py) (Agent manager), [`STORM/tasks/base.py`](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/base.py) (Task interface), [`software-agent-sdk/openhands-sdk/openhands/sdk/__init__.py`](https://github.com/dreamyang-liu/storm/blob/mainline/software-agent-sdk/openhands-sdk/openhands/sdk/__init__.py) (OpenHands SDK)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Experiment entry"]
  node_inference["Inference runner<br/>[run_infer.py]"]
  node_configuration["Run configuration<br/>[config.py]"]
end

subgraph group_tasks["Benchmark tasks"]
  node_base_task["Task interface<br/>[base.py]"]
  node_commit0["Commit0 task<br/>[commit0.py]"]
  node_paperbench["PaperBench task<br/>[paperbench.py]"]
end

subgraph group_orchestration["Agent orchestration"]
  node_manager["Agent manager<br/>[manager.py]"]
  node_subagent["Engineer subagents<br/>[subagent.py]"]
  node_patches["SDK patches<br/>[patches.py]"]
  node_utilities["Run utilities<br/>[utils.py]"]
  node_openhands["OpenHands SDK<br/>[__init__.py]"]
  node_workspace["Task workspace<br/>[workspace.py]"]
end

subgraph group_results["Results and evaluation"]
  node_output["Experiment outputs<br/>[run_infer.py]"]
  node_judge_runner["Judge runner<br/>[judge_runner.py]"]
  node_judge_service["PaperBench judge<br/>[create_judge.py]"]
  node_commit_report["Commit0 report"]
end

node_user(("Experimenter"))
node_llm{{"LLM provider"}}

node_user -->|"starts run"| node_inference
node_inference -->|"loads settings"| node_configuration
node_inference -->|"selects task"| node_base_task
node_base_task -.->|"implements benchmark"| node_commit0
node_base_task -.->|"implements benchmark"| node_paperbench
node_inference -->|"starts orchestration"| node_manager
node_manager -.->|"assigns work"| node_subagent
node_manager -->|"uses agent SDK"| node_openhands
node_manager -.->|"executes commands"| node_workspace
node_manager -.->|"uses compatibility patches"| node_patches
node_inference -->|"uses helpers"| node_utilities
node_openhands -.->|"requests completions"| node_llm
node_inference -->|"records run results"| node_output
node_inference -.->|"supports re-judging"| node_judge_runner
node_judge_runner -.->|"evaluates papers"| node_judge_service
node_inference -.->|"grades implementation"| node_commit_report

click node_inference "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/run_infer.py"
click node_configuration "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/config.py"
click node_base_task "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/base.py"
click node_commit0 "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/commit0.py"
click node_paperbench "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/paperbench.py"
click node_manager "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/manager.py"
click node_subagent "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/subagent.py"
click node_patches "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/patches.py"
click node_utilities "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/utils.py"
click node_openhands "https://github.com/dreamyang-liu/storm/blob/mainline/software-agent-sdk/openhands-sdk/openhands/sdk/__init__.py"
click node_workspace "https://github.com/dreamyang-liu/storm/blob/mainline/software-agent-sdk/openhands-workspace/openhands/workspace/docker/workspace.py"
click node_output "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/run_infer.py"
click node_judge_runner "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/judge/judge_runner.py"
click node_judge_service "https://github.com/dreamyang-liu/storm/blob/mainline/frontier-evals/project/paperbench/paperbench/judge/create_judge.py"
click node_commit_report "https://github.com/dreamyang-liu/storm/blob/mainline/STORM/judge/judge_runner_initial.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_inference,node_configuration toneBlue
class node_base_task,node_commit0,node_paperbench toneAmber
class node_manager,node_subagent,node_patches,node_utilities,node_openhands,node_workspace toneMint
class node_output,node_judge_runner,node_judge_service,node_commit_report toneRose
class node_user,node_llm toneIndigo
```

## Components

### Experiment entry

- [Inference runner](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/run_infer.py): `STORM/run_infer.py`
- [Run configuration](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/config.py): `STORM/config.py`

### Benchmark tasks

- [Task interface](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/base.py): `STORM/tasks/base.py`
- [Commit0 task](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/commit0.py): `STORM/tasks/commit0.py`
- [PaperBench task](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/tasks/paperbench.py): `STORM/tasks/paperbench.py`

### Agent orchestration

- [Agent manager](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/manager.py): `STORM/core/manager.py`
- [Engineer subagents](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/subagent.py): `STORM/core/subagent.py`
- [SDK patches](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/patches.py): `STORM/core/patches.py`
- [Run utilities](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/core/utils.py): `STORM/core/utils.py`
- [OpenHands SDK](https://github.com/dreamyang-liu/storm/blob/mainline/software-agent-sdk/openhands-sdk/openhands/sdk/__init__.py): `software-agent-sdk/openhands-sdk/openhands/sdk/__init__.py`
- [Task workspace](https://github.com/dreamyang-liu/storm/blob/mainline/software-agent-sdk/openhands-workspace/openhands/workspace/docker/workspace.py): `software-agent-sdk/openhands-workspace/openhands/workspace/docker/workspace.py`

### Results and evaluation

- [Experiment outputs](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/run_infer.py): `STORM/run_infer.py`
- [Judge runner](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/judge/judge_runner.py): `STORM/judge/judge_runner.py`
- [PaperBench judge](https://github.com/dreamyang-liu/storm/blob/mainline/frontier-evals/project/paperbench/paperbench/judge/create_judge.py): `frontier-evals/project/paperbench/paperbench/judge/create_judge.py`
- [Commit0 report](https://github.com/dreamyang-liu/storm/blob/mainline/STORM/judge/judge_runner_initial.py): `STORM/judge/judge_runner_initial.py`

### Other components

- **Experimenter**
- **LLM provider**

## Connections

- Experimenter → Inference runner: starts run (evidence: `README.md`)
- Inference runner → Run configuration: loads settings (evidence: `STORM/run_infer.py`)
- Inference runner → Task interface: selects task (evidence: `STORM/run_infer.py`)
- Task interface → Commit0 task: implements benchmark (evidence: `README.md`)
- Task interface → PaperBench task: implements benchmark (evidence: `README.md`)
- Inference runner → Agent manager: starts orchestration (evidence: `STORM/run_infer.py`)
- Agent manager → Engineer subagents: assigns work (evidence: `README.md`)
- Agent manager → OpenHands SDK: uses agent SDK (evidence: `STORM/core/manager.py`)
- Agent manager → Task workspace: executes commands (evidence: `README.md`)
- Agent manager → SDK patches: uses compatibility patches (evidence: `STORM/run_infer.py`)
- Inference runner → Run utilities: uses helpers (evidence: `STORM/run_infer.py`)
- OpenHands SDK → LLM provider: requests completions (evidence: `README.md`)
- Inference runner → Experiment outputs: records run results (evidence: `README.md`)
- Inference runner → Judge runner: supports re-judging (evidence: `README.md`)
- Judge runner → PaperBench judge: evaluates papers (evidence: `README.md`)
- Inference runner → Commit0 report: grades implementation (evidence: `README.md`)
