How dreamyang-liu/storm works
STORM is a multi-agent benchmark runner for code implementation (Commit0) and paper reproduction (PaperBench), built on the OpenHands SDK. The principal workflow selects a task and configuration, runs either a single-agent or multi-agent experiment in a workspace, then records results and— for PaperBench—can evaluate them with a judge. The tree and README establish the product branches; the source index confirms the inference entry point’s task, manager, and patch dependencies. Several internal links are documented at a high level because their implementation excerpts were not provided.
Main entry points: run_infer.py Inference runner · manager.py Agent manager · base.py Task interface · __init__.py OpenHands SDK
Components 17
Experiment entry
- Inference runner
STORM/run_infer.py - Run configuration
STORM/config.py
Benchmark tasks
- Task interface
STORM/tasks/base.py - Commit0 task
STORM/tasks/commit0.py - PaperBench task
STORM/tasks/paperbench.py
Agent orchestration
- Agent manager
STORM/core/manager.py - Engineer subagents
STORM/core/subagent.py - SDK patches
STORM/core/patches.py - Run utilities
STORM/core/utils.py - OpenHands SDK
software-agent-sdk/openhands-sdk/openhands/sdk/__init__.py - Task workspace
software-agent-sdk/openhands-workspace/openhands/workspace/docker/workspace.py
Results and evaluation
- Experiment outputs
STORM/run_infer.py - Judge runner
STORM/judge/judge_runner.py - PaperBench judge
frontier-evals/project/paperbench/paperbench/judge/create_judge.py - Commit0 report
STORM/judge/judge_runner_initial.py
Other components
- Experimenter
- LLM provider
Connections 16
- Experimenter → Inference runner: starts run
README.md - Inference runner → Run configuration: loads settings
STORM/run_infer.py - Inference runner → Task interface: selects task
STORM/run_infer.py - Task interface → Commit0 task: implements benchmark
README.md - Task interface → PaperBench task: implements benchmark
README.md - Inference runner → Agent manager: starts orchestration
STORM/run_infer.py - Agent manager → Engineer subagents: assigns work
README.md - Agent manager → OpenHands SDK: uses agent SDK
STORM/core/manager.py - Agent manager → Task workspace: executes commands
README.md - Agent manager → SDK patches: uses compatibility patches
STORM/run_infer.py - Inference runner → Run utilities: uses helpers
STORM/run_infer.py - OpenHands SDK → LLM provider: requests completions
README.md - Inference runner → Experiment outputs: records run results
README.md - Inference runner → Judge runner: supports re-judging
README.md - Judge runner → PaperBench judge: evaluates papers
README.md - Inference runner → Commit0 report: grades implementation
README.md