# allenai/codescientist architecture

> Architecture diagram of the GitHub repository allenai/codescientist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/allenai/codescientist
- Repository: https://github.com/allenai/codescientist

## Overview

##### Project Type and Purpose
- CodeScientist is an open-source, end-to-end scientific discovery system.
- It provides ideation (LLM‐based idea generation), planning (LLM‐based operationalization into experiment code), experiment execution (in cloud containers via Modal), reporting (LaTeX/PDF), and meta-analysis (aggregating results with LLM).
- It is not a traditional web app or compiler; it’s a data-orchestration and automation tool that integrates multiple components.

##### High-Level File Structure & Architectural Clues
- Top-level scripts:
- src/CodeScientistWebServer.py – backend HTTP server
- src/CodeScientistWebInterface.py – frontend GUI
- Core modules in src/:
- IdeaStore.py – ideation and idea persistence
- ExperimentMaker.py – orchestrates experiment submission
- CodeBlockStore.py – codeblock library and prompt templates
- MetaAnalysis.py – bulk reporting & meta-analysis
- Batch*.py – batch ideation, ranking, planning for benchmarks
- llm-proxy/ – standalone proxy server to meter and route LLM calls
- codeblocks/ – repository of small example scripts (codeblocks) that LLM uses to assemble experiments
- src/modules/ModuleRunPythonInModal.py – cloud container orchestration via Modal.com
- data/, paperstore/, generated-experiments/ – JSON datastore for ideas, experiments, papers, raw results
- config_threads.json, cleaner_config.yaml – tuning and cleanup

##### Key Components & Their Responsibilities
A. Frontend GUI
- Python Flask-based UI (CodeScientistWebInterface.py)
- Presents menus: ideation, manual experiment, batch modes, monitor, meta-analysis
- Talks to backend via HTTP or local socket
B. Backend Orchestrator
- CodeScientistWebServer.py exposes REST endpoints
- Dispatches requests to:
- Ideator (IdeaStore)
- Planner/Operationalizer (BatchPopulatePlans & IdeaStore’s planning routines)
- Experiment submission (ExperimentMaker + ModuleRunPythonInModal)
- Code generation/reflection (CodeBlockStore)
- Reporting & meta-analysis (MetaAnalysis)
- Persists/reads JSON state in data/ and paperstore/
C. Codeblock Library
- codeblocks/*.py with metadata comments
- Summaries in codeblock_summaries.json
- Enables “search for appropriate code snippets” when prompting LLM
D. LLM Proxy
- llm-proxy/llm-proxy-server.py
- Intercepts LLM calls from experiment code to meter token usage and enforce cost limits
E. Cloud Sandbox Executor
- Modal.com containers launched via ModuleRunPythonInModal
- Installs pip requirements, runs generated Python experiment code
- Captures logs, outputs, LLM usage, and result artifacts
F. Data Storage
- JSON files in data/ (ideas, experiments, benchmarks)
- paperstore/paper_index.json – list of ingested ArXiv papers
- generated-experiments/ – full experiment archives

##### Component Interactions & Data Flow
1. User clicks a button in Frontend → HTTP request to Backend
2. Backend invokes:
- Ideation pipeline (LLM calls) → writes ideas to data/ideastore.json
- Planning pipeline (LLM calls + CodeBlockStore) → enriches ideas with experiment plans
- ExperimentMaker → packages plan + code snippets → calls Modal module
3. Modal executes code in sandbox, with LLM calls routed through LLM Proxy
4. Sandbox returns logs, results, PDF/LaTeX report → saved under generated-experiments/
5. Backend indexes experiment metadata in data/all-experiments.json
6. Frontend’s Experiment Monitor polls Backend for status and displays summaries
7. User triggers MetaAnalysis → Backend loads experiment JSONs → LLM meta-analysis → tabular/JSON output for download

##### Architectural Patterns & Principles
- Modular layering
- UI layer (Flask GUI)
- Orchestration layer (server + high-level pipelines)
- Execution layer (cloud containers + LLM proxy)
- Persistence layer (flat JSON files)
- “Prompt as code” pattern: codeblocks are first‐class artifacts with metadata
- Microservice‐style separation: the LLM proxy is a distinct service
- Event‐driven workflow: user actions trigger pipeline stages

##### Technologies & Frameworks
- Python 3.12, Flask (web UI & server)
- Modal.com Python SDK for container orchestration
- litellm or custom proxy for LLM calls (OpenAI, Anthropic, etc.)
- JSON for configuration and state persistence
- LaTeX (pdflatex) for report generation
- Git & Apache 2.0 license for open-source distribution

##### Essential Elements to Capture
- Shaded grouping (e.g., “Local Host” vs “Cloud Sandbox”)
- Where LLM calls occur (ideation, planning, runtime, meta-analysis)
- Cost-limiting proxy in the runtime path

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Local Host Environment
    subgraph "Local Host"
        direction TB
        UI["Frontend GUI\n(src/CodeScientistWebInterface.py)"]:::frontend
        API["Backend Orchestrator\n(src/CodeScientistWebServer.py)"]:::orchestration

        subgraph "Orchestration Layer"
            direction TB
            Ideator["Ideation Pipeline\n(src/IdeaStore.py)"]:::orchestration
            Planner["Planning Pipeline\n(BatchPopulatePlans, BatchIdeatorRanker, BatchPregeneratedIdeator)"]:::orchestration
            CodeBlocks["Codeblock Library\n(codeblocks/*.py, codeblock_summaries.json, src/CodeBlockStore.py)"]:::orchestration
            ExperimentMaker["Experiment Submission\n(src/ExperimentMaker.py)"]:::orchestration
            MetaAnalysisComp["Reporting & Meta-Analysis\n(src/MetaAnalysis.py)"]:::orchestration
            PaperStore["Paper Ingestion\n(src/PaperStore.py)"]:::orchestration
        end

        subgraph "Persistence Layer"
            direction TB
            DataStore["JSON Data Stores\n(data/, paperstore/, generated-experiments/)"]:::persistence
            Config["Config & Cleanup\n(cleaner_config.yaml, config_threads.json, requirements.txt)"]:::persistence
        end

        UI -->|HTTP/JSON| API
        API -->|invoke| Ideator
        API -->|invoke| Planner
        API -->|invoke| ExperimentMaker
        API -->|invoke| MetaAnalysisComp
        API -->|invoke| PaperStore

        Ideator -->|writes ideas to| DataStore
        Planner -->|reads/writes| DataStore
        CodeBlocks -->|provides snippets| Planner
        ExperimentMaker -->|reads plans + snippets| Planner
        ExperimentMaker -->|packages code| CodeBlocks
        ExperimentMaker -->|push| CloudExecutor
        MetaAnalysisComp -->|reads experiments| DataStore

        PaperStore -->|updates| DataStore
    end

    %% Cloud Sandbox Environment
    subgraph "Cloud Sandbox"
        direction TB
        CloudExecutor["Modal.com Sandbox\n(ModuleRunPythonInModal, ModalUtilStopAllApps)"]:::runtime
        LLMProxy["LLM Proxy Service\n(llm-proxy-server.py, llm_proxy_usage.py)"]:::runtime
        ExternalLLM["External LLM APIs\n(OpenAI, Anthropic)"]:::external
        Arxiv["ArXiv Papers"]:::external

        CloudExecutor -->|runs code with| LLMProxy
        CloudExecutor -->|executes experiments| ExternalLLM
        LLMProxy -->|routes calls| ExternalLLM
        ExternalLLM -->|returns tokens| CloudExecutor
        Arxiv -->|ingest| PaperStore
        CloudExecutor -->|outputs logs/results| DataStore
    end

    %% Monitoring Workflow
    subgraph "User Workflow"
        direction TB
        Monitor["Experiment Monitor\n(polls API for status)"]:::frontend
        User["User"]:::external
        User -->|clicks| UI
        User -->|views| Monitor
        Monitor -->|HTTP| API
    end

    %% Click Events
    click UI "https://github.com/allenai/codescientist/blob/main/src/CodeScientistWebInterface.py"
    click API "https://github.com/allenai/codescientist/blob/main/src/CodeScientistWebServer.py"
    click Ideator "https://github.com/allenai/codescientist/blob/main/src/IdeaStore.py"
    click Planner "https://github.com/allenai/codescientist/blob/main/src/BatchPopulatePlans.py"
    click Planner "https://github.com/allenai/codescientist/blob/main/src/BatchIdeatorRanker.py"
    click Planner "https://github.com/allenai/codescientist/blob/main/src/BatchPregeneratedIdeator.py"
    click ExperimentMaker "https://github.com/allenai/codescientist/blob/main/src/ExperimentMaker.py"
    click CloudExecutor "https://github.com/allenai/codescientist/blob/main/src/modules/ModuleRunPythonInModal.py"
    click CloudExecutor "https://github.com/allenai/codescientist/blob/main/src/utils/ModalUtilStopAllApps.py"
    click CodeBlocks "https://github.com/allenai/codescientist/tree/main/codeblocks/"
    click CodeBlocks "https://github.com/allenai/codescientist/blob/main/codeblocks/codeblock_summaries.json"
    click CodeBlocks "https://github.com/allenai/codescientist/blob/main/src/CodeBlockStore.py"
    click LLMProxy "https://github.com/allenai/codescientist/blob/main/llm-proxy/llm-proxy-server.py"
    click LLMProxy "https://github.com/allenai/codescientist/blob/main/llm-proxy/llm_proxy_usage.py"
    click MetaAnalysisComp "https://github.com/allenai/codescientist/blob/main/src/MetaAnalysis.py"
    click MetaAnalysisComp "https://github.com/allenai/codescientist/tree/main/metaanalysis/"
    click PaperStore "https://github.com/allenai/codescientist/blob/main/src/PaperStore.py"
    click PaperStore "https://github.com/allenai/codescientist/blob/main/paperstore/paper_index.json"
    click DataStore "https://github.com/allenai/codescientist/tree/main/data/"
    click DataStore "https://github.com/allenai/codescientist/tree/main/generated-experiments/"
    click Config "https://github.com/allenai/codescientist/blob/main/cleaner_config.yaml"
    click Config "https://github.com/allenai/codescientist/blob/main/config_threads.json"
    click Config "https://github.com/allenai/codescientist/blob/main/requirements.txt"

    %% Styles
    classDef frontend fill:#e0f7fa,stroke:#006064,color:#006064;
    classDef orchestration fill:#e8f5e9,stroke:#1b5e20,color:#1b5e20;
    classDef runtime fill:#fff3e0,stroke:#e65100,color:#e65100;
    classDef persistence fill:#eceff1,stroke:#546e7a,color:#546e7a;
    classDef external fill:#f3e5f5,stroke:#6a1b9a,color:#6a1b9a;
```
