# stanford-oval/datastorm architecture

> Architecture diagram of the GitHub repository stanford-oval/datastorm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/stanford-oval/datastorm
- Repository: https://github.com/stanford-oval/datastorm

## Overview

DataSTORM accepts a research question, explores it through a database-oriented search tree, consolidates findings, and generates a cited report; web research can add context. The README identifies the SQL agent and research pipeline as the two core components. The graph preserves those stages and the documented optional SUQL, sandbox, and upload capabilities. Some relationships are documented rather than visible in sampled implementation excerpts, so those edges cite the README; unsampled subsystem internals are represented only at supported directory paths.

Main entry points: [`knowledge_storm/datatalk_agent/agent.py`](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/agent.py) (SQL Agent), [`datatalk_basic_article_generation_costorm.py`](https://github.com/stanford-oval/datastorm/blob/main/datatalk_basic_article_generation_costorm.py) (CLI / API), [`knowledge_storm/datastorm/modules/knowledge_curation.py`](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/knowledge_curation.py) (Knowledge Curation), [`knowledge_storm/datastorm/engine.py`](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/engine.py) (DataSTORM Engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_research["Research Pipeline"]
  node_entry["CLI / API"]
  node_storm_engine["DataSTORM Engine<br/>[engine.py]"]
  node_curation["Knowledge Curation"]
  node_tree["Tree Simulation<br/>[tree_simulator.py]"]
  node_search_utils["Tree Search"]
  node_retriever["Retriever<br/>[retriever.py]"]
  node_insights["Research State<br/>[storm_dataclass.py]"]
  node_costorm["Co-STORM Engine<br/>[engine.py]"]
  node_warmstart["Warm Start"]
end

subgraph group_sql["SQL Agent"]
  node_sql_agent["SQL Agent<br/>[agent.py]"]
  node_agent_library["Agent Library<br/>[library.py]"]
  node_agent_state["SQL State<br/>[state.py]"]
  node_sandbox["Python Sandbox<br/>[sandbox_runner.py]"]
end

subgraph group_generation["Report Generation"]
  node_report["Report Generation"]
  node_output["Research Artifacts"]
end

subgraph group_integrations["External Integrations"]
  node_database[("Domain Database")]
  node_suql["SUQL Services"]
  node_web_search["Web Search<br/>[rm.py]"]
end

node_analyst(("Researcher"))

node_analyst -->|"submits question"| node_entry
node_entry -.->|"starts research"| node_storm_engine
node_entry -->|"configures runner"| node_costorm
node_entry -->|"uses curation"| node_curation
node_entry -->|"uses generation"| node_report
node_storm_engine -->|"runs curation"| node_curation
node_storm_engine -->|"constructs retriever"| node_retriever
node_curation -->|"uses simulator"| node_tree
node_tree -->|"uses utilities"| node_search_utils
node_curation -->|"updates research state"| node_insights
node_retriever -.->|"queries sources"| node_web_search
node_curation -->|"asks database questions"| node_sql_agent
node_sql_agent -->|"updates state"| node_agent_state
node_agent_library -->|"invokes agent"| node_sql_agent
node_sql_agent -->|"executes SQL"| node_database
node_sql_agent -.->|"uses semantic queries"| node_suql
node_sql_agent -->|"delegates analysis"| node_sandbox
node_storm_engine -.->|"generates report"| node_report
node_report -->|"writes artifacts"| node_output
node_costorm -->|"runs warm start"| node_warmstart

click node_entry "https://github.com/stanford-oval/datastorm/blob/main/datatalk_basic_article_generation_costorm.py"
click node_storm_engine "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/engine.py"
click node_curation "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/knowledge_curation.py"
click node_tree "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/tree_simulator.py"
click node_search_utils "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/tree_search_utils.py"
click node_retriever "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/retriever.py"
click node_insights "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/storm_dataclass.py"
click node_sql_agent "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/agent.py"
click node_agent_library "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/library.py"
click node_agent_state "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/state.py"
click node_sandbox "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/python_modules/sandbox_runner.py"
click node_web_search "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/rm.py"
click node_report "https://github.com/stanford-oval/datastorm/blob/main/final_report_gen_utils.py"
click node_output "https://github.com/stanford-oval/datastorm/blob/main/datatalk_basic_article_generation_costorm.py"
click node_costorm "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/collaborative_storm/engine.py"
click node_warmstart "https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/collaborative_storm/modules/warmstart_hierarchical_chat.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_entry,node_storm_engine,node_curation,node_tree,node_search_utils,node_retriever,node_insights,node_costorm,node_warmstart toneBlue
class node_sql_agent,node_agent_library,node_agent_state,node_sandbox toneAmber
class node_report,node_output toneMint
class node_database,node_suql,node_web_search toneRose
class node_analyst toneIndigo
```

## Components

### Research Pipeline

- [CLI / API](https://github.com/stanford-oval/datastorm/blob/main/datatalk_basic_article_generation_costorm.py): `datatalk_basic_article_generation_costorm.py`
- [DataSTORM Engine](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/engine.py): `knowledge_storm/datastorm/engine.py`
- [Knowledge Curation](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/knowledge_curation.py): `knowledge_storm/datastorm/modules/knowledge_curation.py`
- [Tree Simulation](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/tree_simulator.py): `knowledge_storm/datastorm/modules/tree_simulator.py`
- [Tree Search](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/tree_search_utils.py): `knowledge_storm/datastorm/modules/tree_search_utils.py`
- [Retriever](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/retriever.py): `knowledge_storm/datastorm/modules/retriever.py`
- [Research State](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datastorm/modules/storm_dataclass.py): `knowledge_storm/datastorm/modules/storm_dataclass.py`
- [Co-STORM Engine](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/collaborative_storm/engine.py): `knowledge_storm/collaborative_storm/engine.py`
- [Warm Start](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/collaborative_storm/modules/warmstart_hierarchical_chat.py): `knowledge_storm/collaborative_storm/modules/warmstart_hierarchical_chat.py`

### SQL Agent

- [SQL Agent](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/agent.py): `knowledge_storm/datatalk_agent/agent.py`
- [Agent Library](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/library.py): `knowledge_storm/datatalk_agent/library.py`
- [SQL State](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/state.py): `knowledge_storm/datatalk_agent/state.py`
- [Python Sandbox](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/datatalk_agent/python_modules/sandbox_runner.py): `knowledge_storm/datatalk_agent/python_modules/sandbox_runner.py`

### Report Generation

- [Report Generation](https://github.com/stanford-oval/datastorm/blob/main/final_report_gen_utils.py): `final_report_gen_utils.py`
- [Research Artifacts](https://github.com/stanford-oval/datastorm/blob/main/datatalk_basic_article_generation_costorm.py): `datatalk_basic_article_generation_costorm.py`

### External Integrations

- **Domain Database**
- **SUQL Services**
- [Web Search](https://github.com/stanford-oval/datastorm/blob/main/knowledge_storm/rm.py): `knowledge_storm/rm.py`

### Other components

- **Researcher**

## Connections

- Researcher → CLI / API: submits question
- CLI / API → DataSTORM Engine: starts research
- CLI / API → Co-STORM Engine: configures runner (evidence: `datatalk_basic_article_generation_costorm.py`)
- CLI / API → Knowledge Curation: uses curation (evidence: `datatalk_basic_article_generation_costorm.py`)
- CLI / API → Report Generation: uses generation (evidence: `datatalk_basic_article_generation_costorm.py`)
- DataSTORM Engine → Knowledge Curation: runs curation (evidence: `knowledge_storm/datastorm/engine.py`)
- DataSTORM Engine → Retriever: constructs retriever (evidence: `knowledge_storm/datastorm/engine.py`)
- Knowledge Curation → Tree Simulation: uses simulator (evidence: `knowledge_storm/datastorm/modules/knowledge_curation.py`)
- Tree Simulation → Tree Search: uses utilities (evidence: `knowledge_storm/datastorm/modules/tree_simulator.py`)
- Knowledge Curation → Research State: updates research state (evidence: `knowledge_storm/datastorm/modules/knowledge_curation.py`)
- Retriever → Web Search: queries sources
- Knowledge Curation → SQL Agent: asks database questions
- SQL Agent → SQL State: updates state (evidence: `knowledge_storm/datatalk_agent/agent.py`)
- Agent Library → SQL Agent: invokes agent (evidence: `knowledge_storm/datatalk_agent/library.py`)
- SQL Agent → Domain Database: executes SQL
- SQL Agent → SUQL Services: uses semantic queries
- SQL Agent → Python Sandbox: delegates analysis (evidence: `knowledge_storm/datatalk_agent/agent.py`)
- DataSTORM Engine → Report Generation: generates report
- Report Generation → Research Artifacts: writes artifacts (evidence: `datatalk_basic_article_generation_costorm.py`)
- Co-STORM Engine → Warm Start: runs warm start (evidence: `knowledge_storm/collaborative_storm/engine.py`)
