How allenai/codescientist works
Project Type and Purpose
• CodeScientist is an open-source, end-to-end scientific discovery system.
• It provides ideation (LLM‐based idea generation), planning (LLM‐based operationalization into experiment code), experiment execution (in cloud containers via Modal), reporting (LaTeX/PDF), and meta-analysis (aggregating results with LLM).
• It is not a traditional web app or compiler; it’s a data-orchestration and automation tool that integrates multiple components.
High-Level File Structure & Architectural Clues
• Top-level scripts:
• src/CodeScientistWebServer.py – backend HTTP server
• src/CodeScientistWebInterface.py – frontend GUI
• Core modules in src/:
• IdeaStore.py – ideation and idea persistence
• ExperimentMaker.py – orchestrates experiment submission
• CodeBlockStore.py – codeblock library and prompt templates
• MetaAnalysis.py – bulk reporting & meta-analysis
• Batch*.py – batch ideation, ranking, planning for benchmarks
• llm-proxy/ – standalone proxy server to meter and route LLM calls
• codeblocks/ – repository of small example scripts (codeblocks) that LLM uses to assemble experiments
• src/modules/ModuleRunPythonInModal.py – cloud container orchestration via Modal.com
• data/, paperstore/, generated-experiments/ – JSON datastore for ideas, experiments, papers, raw results
• config_threads.json, cleaner_config.yaml – tuning and cleanup
Key Components & Their Responsibilities
A. Frontend GUI
• Python Flask-based UI (CodeScientistWebInterface.py)
• Presents menus: ideation, manual experiment, batch modes, monitor, meta-analysis
• Talks to backend via HTTP or local socket
B. Backend Orchestrator
• CodeScientistWebServer.py exposes REST endpoints
• Dispatches requests to:
• Ideator (IdeaStore)
• Planner/Operationalizer (BatchPopulatePlans & IdeaStore’s planning routines)
• Experiment submission (ExperimentMaker + ModuleRunPythonInModal)
• Code generation/reflection (CodeBlockStore)
• Reporting & meta-analysis (MetaAnalysis)
• Persists/reads JSON state in data/ and paperstore/
C. Codeblock Library
• codeblocks/*.py with metadata comments
• Summaries in codeblock_summaries.json
• Enables “search for appropriate code snippets” when prompting LLM
D. LLM Proxy
• llm-proxy/llm-proxy-server.py
• Intercepts LLM calls from experiment code to meter token usage and enforce cost limits
E. Cloud Sandbox Executor
• Modal.com containers launched via ModuleRunPythonInModal
• Installs pip requirements, runs generated Python experiment code
• Captures logs, outputs, LLM usage, and result artifacts
F. Data Storage
• JSON files in data/ (ideas, experiments, benchmarks)
• paperstore/paper_index.json – list of ingested ArXiv papers
• generated-experiments/ – full experiment archives
Component Interactions & Data Flow
1. User clicks a button in Frontend → HTTP request to Backend
2. Backend invokes:
• Ideation pipeline (LLM calls) → writes ideas to data/ideastore.json
• Planning pipeline (LLM calls + CodeBlockStore) → enriches ideas with experiment plans
• ExperimentMaker → packages plan + code snippets → calls Modal module
3. Modal executes code in sandbox, with LLM calls routed through LLM Proxy
4. Sandbox returns logs, results, PDF/LaTeX report → saved under generated-experiments/
5. Backend indexes experiment metadata in data/all-experiments.json
6. Frontend’s Experiment Monitor polls Backend for status and displays summaries
7. User triggers MetaAnalysis → Backend loads experiment JSONs → LLM meta-analysis → tabular/JSON output for download
Architectural Patterns & Principles
• Modular layering
• UI layer (Flask GUI)
• Orchestration layer (server + high-level pipelines)
• Execution layer (cloud containers + LLM proxy)
• Persistence layer (flat JSON files)
• “Prompt as code” pattern: codeblocks are first‐class artifacts with metadata
• Microservice‐style separation: the LLM proxy is a distinct service
• Event‐driven workflow: user actions trigger pipeline stages
Technologies & Frameworks
• Python 3.12, Flask (web UI & server)
• Modal.com Python SDK for container orchestration
• litellm or custom proxy for LLM calls (OpenAI, Anthropic, etc.)
• JSON for configuration and state persistence
• LaTeX (pdflatex) for report generation
• Git & Apache 2.0 license for open-source distribution
Essential Elements to Capture
• Shaded grouping (e.g., “Local Host” vs “Cloud Sandbox”)
• Where LLM calls occur (ideation, planning, runtime, meta-analysis)
• Cost-limiting proxy in the runtime path