# sakanaai/ai-scientist architecture

> Architecture diagram of the GitHub repository sakanaai/ai-scientist, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sakanaai/ai-scientist
- Repository: https://github.com/sakanaai/ai-scientist

## Overview

This repository is an end-to-end system for automated scientific ideation, experimentation, paper writing, and review generation. The architecture is centered around a small orchestration layer in `ai_scientist/` that drives many template-specific research workloads under `templates/` and stores outputs under run directories such as `example_papers/` and `review_*`.

#### 1) High-level system shape

The repo has four main layers:

1. **Core orchestration and LLM access**
   - `launch_scientist.py`
   - `ai_scientist/`
   - `experimental/launch_oe_scientist.py`

2. **Task templates for research domains**
   - `templates/` for reusable experiment/paper-generation blueprints
   - `data/` for dataset preparation used by templates

3. **Generated research artifacts**
   - `example_papers/` contains complete sample runs with code, logs, figures, LaTeX, reviews, and PDFs
   - these directories show the expected output shape of a successful run

4. **Evaluation / review benchmarking**
   - `ai_scientist/perform_review.py`
   - `review_ai_scientist/`
   - `review_iclr_bench/`

The repo is not a single application; it is a framework for launching many independent research experiments based on a template.

#### 2) Core orchestration subsystem

The `ai_scientist/` package contains the “agentic workflow” used during paper generation:

- `generate_ideas.py` handles idea generation from prompts and few-shot examples.
- `perform_experiments.py` runs or coordinates experiment execution for a selected idea.
- `perform_writeup.py` turns results into a paper-style writeup using a LaTeX template.
- `perform_review.py` generates paper reviews and loads PDFs/text for evaluation.
- `llm.py` abstracts model backends and provider-specific API access.

This core layer is the boundary between the high-level workflow and the actual template code. It is where LLM calls are made, where prompt construction happens, and where outputs are collected into the standardized run folders.

##### Important architectural boundary
The system intentionally separates:
- **model-facing logic** in `ai_scientist/`
from
- **research-domain logic** in each template’s `experiment.py` and related files.

That separation is important because the framework can swap templates while reusing the same ideation/writeup/review pipeline.

#### 3) Template subsystem

The template folders define the scientific “world” the AI Scientist can work in. Each template is a self-contained research environment with a consistent interface.

Common template structure:
- `experiment.py` — main experiment entry point
- `plot.py` — converts results into figures
- `prompt.json` — describes the task and constraints for the LLM
- `ideas.json` and `seed_ideas.json` — example seed ideas and prior candidates
- `latex/template.tex` plus style/bibliography assets — final paper shell
- optional domain-specific support code and datasets

The repo includes both official and community-contributed templates. The most important baseline templates mentioned in the README are:

- `templates/nanoGPT`
- `templates/2d_diffusion`
- `templates/grokking`

Additional templates expand the framework to other domains:
- `seir`
- `mobilenetV3`
- `sketch_rnn`
- `MACE`
- `earthquake-prediction`
- `tensorf`
- `probes`

##### Template runtime contract
The framework expects templates to produce outputs in a predictable shape:
- a `run_0` baseline and subsequent `run_*` directories
- JSON files summarizing metrics and final state
- plots for the paper
- a compiled PDF after LaTeX writeup

This contract is why the README emphasizes matching base filenames and output JSON structure when adding a new template.

#### 4) Experiment execution flow

A typical experiment runs through this sequence:

1. **Data preparation**
   - Some templates require preprocessing scripts in `data/`
   - Example:
     - `data/enwik8/prepare.py`
     - `data/shakespeare_char/prepare.py`
     - `data/text8/prepare.py`
   - These scripts create the datasets consumed by the NanoGPT-style experiments

2. **Baseline run**
   - Each template has a machine-specific baseline run, usually `run_0`
   - Example:
     - `python experiment.py --out_dir run_0`
   - Baselines are important because they anchor comparisons on the local machine’s hardware and runtime characteristics

3. **Idea generation**
   - `ai_scientist/generate_ideas.py` uses a selected LLM to propose candidate research ideas
   - It is guided by the template prompt, few-shot examples, and optionally prior seed ideas

4. **Experimentation**
   - `ai_scientist/perform_experiments.py` coordinates the selected idea into an actual code change / run loop
   - The template’s `experiment.py` is the main execution target
   - Outputs are written into run-specific directories and JSON summaries

5. **Plotting**
   - Template-specific `plot.py` reads run results and creates publication-quality figures

6. **Writeup**
   - `ai_scientist/perform_writeup.py` assembles results into a LaTeX manuscript using the template’s `latex/template.tex`
   - The output is a paper PDF in the template or example-paper folder

7. **Review**
   - `ai_scientist/perform_review.py` can score the generated paper and produce accept/reject style commentary

#### 5) LLM and external service boundary

The repository is explicitly multi-provider. `ai_scientist/llm.py` is the abstraction layer for model selection and API calls.

Supported providers called out in the README include:
- OpenAI
- Anthropic
- DeepSeek
- OpenRouter
- Google Gemini
- Claude via Bedrock or Vertex AI

This matters architecturally because the core orchestration does not depend on one vendor; the provider logic is isolated behind the LLM layer.

##### Other external services
- **Semantic Scholar** or **OpenAlex** is used for literature search / novelty checking and citation gathering during the paper-generation workflow.
- LaTeX tooling is required for PDF generation, typically via `pdflatex` from a full TeX distribution.
- The repository warns that generated code is executed, so the runtime boundary includes process execution and containerization concerns.

#### 6) Generated artifact subsystem

`example_papers/` is a rich archive of completed experiments. Each example paper directory is essentially a snapshot of the entire workflow for one idea.

A typical example-paper directory contains:
- `experiment.py` or generated `run_*.py`
- `run_*` directories with `final_info.json` and related result files
- `plot.py`
- `ideas.json`, `seed_ideas.json`, and `prompt.json`
- `notes.txt`, `log.txt`, and `review.txt`
- `latex/` with the paper source and styles
- final PDFs

This folder shows the output model of the whole system:
- raw experiment code
- experiment results
- figures
- manuscript source
- manuscript PDF
- review artifacts

That makes `example_papers/` a useful reference boundary for understanding what “done” looks like in this architecture.

#### 7) Review and benchmark subsystem

The review side is separated from paper generation.

##### `review_ai_scientist/`
This folder contains:
- parsed AI-generated papers
- model-specific review CSVs
- manual paper ratings
- notebook-based analysis and figure generation

It is essentially an offline analysis corpus for evaluating generated papers and paper reviews.

##### `review_iclr_bench/`
This is a larger benchmark dataset for paper review analysis:
- `iclr_parsed/` contains text-parsed papers
- `llm_reviews/` contains generated review CSVs from different models and configurations
- `iclr_analysis.py` performs batch review evaluation
- `ratings_subset.tsv` and notebooks support further analysis

Architecturally, this subsystem is independent from paper generation. It reuses the review logic but focuses on evaluation and benchmarking rather than producing new experiments.

#### 8) Infrastructure and packaging

Supporting infrastructure is light but important:

- `requirements.txt` lists Python dependencies
- `experimental/Dockerfile` provides containerization support
- `launch_scientist.py` is the main entrypoint for running the system
- `experimental/launch_oe_scientist.py` suggests an alternate launch path or experimental mode
- `LICENSE` and the README’s responsible-use section define the operational and legal boundaries

The architecture assumes:
- Linux
- NVIDIA GPU acceleration for most templates
- Python 3.11
- external APIs for LLMs and literature search
- local filesystem storage for all run artifacts

#### 9) Data and runtime flow summary

A diagram of this repo should show the following flow:

- Template metadata and few-shot examples feed the LLM-driven ideation layer.
- The selected idea is transformed into code-experiment runs using the template’s `experiment.py`.
- Runtime outputs are written into run folders as JSON, logs, and plots.
- The plotting layer converts results into figures.
- The writeup layer consumes results and plots to generate a LaTeX paper and PDF.
- The review layer consumes the paper text/PDF and returns scores, weaknesses, and an accept/reject decision.
- Separate benchmark folders collect generated and human-reviewed papers for offline analysis.

#### 10) Key architectural constraints

Several constraints materially shape the architecture:

- **Template rigidity:** templates must conform to the repo’s expected naming and output format.
- **Code execution risk:** the system runs LLM-generated code, so sandboxing/containerization is a major operational boundary.
- **GPU dependence:** most meaningful workloads require CUDA-capable hardware.
- **External API dependence:** the system is only as strong as the chosen LLM and literature-search providers.
- **Artifact-heavy workflow:** the system is designed around persistent run directories and generated documents, not ephemeral in-memory computation.

Overall, this repository is best understood as a modular research automation pipeline: a generic LLM orchestration core, a set of domain templates, and a paper/review artifact pipeline that turns experimental runs into publishable-style outputs.

Main entry points: [`ai_scientist/perform_experiments.py`](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/perform_experiments.py) (Experiments), [`ai_scientist/generate_ideas.py`](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/generate_ideas.py) (Ideation), [`ai_scientist/perform_review.py`](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/perform_review.py) (Review), [`ai_scientist/llm.py`](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/llm.py) (LLM layer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph "Core orchestration"
  launch_scientist["Launch<br/>entrypoint"]
  oe_launch["OE launch<br/>entrypoint"]
  llm["LLM layer<br/>provider abstraction<br/>[llm.py]"]
  generate_ideas["Ideation<br/>workflow stage<br/>[generate_ideas.py]"]
  perform_experiments["Experiments<br/>workflow stage"]
  perform_writeup["Writeup<br/>workflow stage<br/>[perform_writeup.py]"]
  perform_review["Review<br/>workflow stage<br/>[perform_review.py]"]
end

subgraph "Research templates"
  nano_gpt["nanoGPT<br/>template"]
  diffusion["2D diffusion<br/>template"]
  grokking["Grokking<br/>template"]
  tensorf["Tensorf<br/>template"]
  sketch_rnn["Sketch RNN<br/>template"]
  probes["Probes<br/>template"]
  seir["SEIR<br/>template"]
end

subgraph "Generated artifacts"
  example_papers[("Examples<br/>run artifacts")]
end

subgraph "Review benchmarking"
  review_ai[("AI reviews<br/>benchmark corpus")]
  review_bench[("ICLR bench<br/>benchmark corpus")]
end

subgraph "Support data"
  data_prep["Data prep<br/>dataset preparation"]
  dockerfile["Docker<br/>runtime scaffold"]
end

launch_scientist -->|"starts"| generate_ideas
oe_launch -.->|"alt path"| perform_experiments
generate_ideas -->|"calls"| llm
perform_experiments -->|"calls"| llm
perform_writeup -->|"calls"| llm
perform_review -->|"calls"| llm
generate_ideas -->|"targets"| nano_gpt
generate_ideas -->|"targets"| diffusion
generate_ideas -->|"targets"| grokking
perform_experiments -->|"runs"| tensorf
perform_experiments -->|"runs"| sketch_rnn
perform_experiments -->|"runs"| probes
perform_experiments -->|"runs"| seir
perform_writeup -->|"writes"| example_papers
perform_experiments -->|"outputs"| example_papers
data_prep -->|"feeds"| nano_gpt
data_prep -.->|"mirrors"| example_papers
dockerfile -.->|"supports"| perform_experiments
perform_review -->|"analyzes"| review_ai
perform_review -->|"analyzes"| review_bench
review_ai -->|"ingests"| example_papers
review_bench -.->|"benchmarks"| perform_review

click launch_scientist "https://github.com/sakanaai/ai-scientist/blob/main/launch_scientist.py"
click oe_launch "https://github.com/sakanaai/ai-scientist/blob/main/experimental/launch_oe_scientist.py"
click llm "https://github.com/sakanaai/ai-scientist/blob/main/ai_scientist/llm.py"
click generate_ideas "https://github.com/sakanaai/ai-scientist/blob/main/ai_scientist/generate_ideas.py"
click perform_experiments "https://github.com/sakanaai/ai-scientist/blob/main/ai_scientist/perform_experiments.py"
click perform_writeup "https://github.com/sakanaai/ai-scientist/blob/main/ai_scientist/perform_writeup.py"
click perform_review "https://github.com/sakanaai/ai-scientist/blob/main/ai_scientist/perform_review.py"
click nano_gpt "https://github.com/sakanaai/ai-scientist/tree/main/templates/nanoGPT"
click diffusion "https://github.com/sakanaai/ai-scientist/tree/main/templates/2d_diffusion"
click grokking "https://github.com/sakanaai/ai-scientist/tree/main/templates/grokking"
click tensorf "https://github.com/sakanaai/ai-scientist/tree/main/templates/tensorf"
click sketch_rnn "https://github.com/sakanaai/ai-scientist/tree/main/templates/sketch_rnn"
click probes "https://github.com/sakanaai/ai-scientist/tree/main/templates/probes"
click seir "https://github.com/sakanaai/ai-scientist/tree/main/templates/seir"
click example_papers "https://github.com/sakanaai/ai-scientist/tree/main/example_papers"
click data_prep "https://github.com/sakanaai/ai-scientist/tree/main/data"
click dockerfile "https://github.com/sakanaai/ai-scientist/tree/main/experimental/Dockerfile"
click review_ai "https://github.com/sakanaai/ai-scientist/tree/main/review_ai_scientist"
click review_bench "https://github.com/sakanaai/ai-scientist/tree/main/review_iclr_bench"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class launch_scientist,oe_launch,llm,generate_ideas,perform_experiments,perform_writeup,perform_review toneBlue
class nano_gpt,diffusion,grokking,tensorf,sketch_rnn,probes,seir toneAmber
class example_papers toneMint
class review_ai,review_bench toneRose
class data_prep,dockerfile toneIndigo
```

## Components

### Core orchestration

Entry points, LLM access, and workflow stages for ideation through review.

- [Launch](https://github.com/sakanaai/ai-scientist/blob/HEAD/launch_scientist.py): entrypoint · Main launcher for the scientist workflow. · `launch_scientist.py`
- [OE launch](https://github.com/sakanaai/ai-scientist/blob/HEAD/experimental/launch_oe_scientist.py): entrypoint · Experimental alternate launcher. · `experimental/launch_oe_scientist.py`
- [LLM layer](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/llm.py): provider abstraction · Backend abstraction for model and API access. · `ai_scientist/llm.py`
- [Ideation](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/generate_ideas.py): workflow stage · Prompts models to propose candidate research ideas. · `ai_scientist/generate_ideas.py`
- [Experiments](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/perform_experiments.py): workflow stage · Coordinates experiment execution for selected ideas. · `ai_scientist/perform_experiments.py`
- [Writeup](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/perform_writeup.py): workflow stage · Builds LaTeX manuscripts from results and figures. · `ai_scientist/perform_writeup.py`
- [Review](https://github.com/sakanaai/ai-scientist/blob/HEAD/ai_scientist/perform_review.py): workflow stage · Generates paper reviews and evaluation outputs. · `ai_scientist/perform_review.py`

### Research templates

Domain-specific experiment blueprints that define the scientific workload.

- [nanoGPT](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/nanoGPT): template · Baseline text-modeling template and canonical workload. · `templates/nanoGPT`
- [2D diffusion](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/2d_diffusion): template · Toy diffusion research template. · `templates/2d_diffusion`
- [Grokking](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/grokking): template · Template for grokking-style experiments. · `templates/grokking`
- [Tensorf](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/tensorf): template · 3D reconstruction / neural field template. · `templates/tensorf`
- [Sketch RNN](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/sketch_rnn): template · Sequence-to-sketch generation template. · `templates/sketch_rnn`
- [Probes](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/probes): template · Probe-based analysis template. · `templates/probes`
- [SEIR](https://github.com/sakanaai/ai-scientist/tree/HEAD/templates/seir): template · Epidemiology-style template. · `templates/seir`

### Generated artifacts

Example run directories and produced papers that show the expected output shape.

- [Examples](https://github.com/sakanaai/ai-scientist/tree/HEAD/example_papers): run artifacts · Completed run directories with code, logs, plots, LaTeX, and PDFs. · `example_papers`

### Review benchmarking

Offline corpora and analysis for paper-review generation and benchmarking.

- [AI reviews](https://github.com/sakanaai/ai-scientist/tree/HEAD/review_ai_scientist): benchmark corpus · Offline corpus of generated papers and model review scores. · `review_ai_scientist`
- [ICLR bench](https://github.com/sakanaai/ai-scientist/tree/HEAD/review_iclr_bench): benchmark corpus · Parsed papers, review CSVs, and analysis for review benchmarking. · `review_iclr_bench`

### Support data

Dataset preparation and deployment/runtime scaffolding.

- [Data prep](https://github.com/sakanaai/ai-scientist/tree/HEAD/data): dataset preparation · Preprocessing scripts for template datasets. · `data`
- [Docker](https://github.com/sakanaai/ai-scientist/tree/HEAD/experimental/Dockerfile): runtime scaffold · Containerization support for execution and reproducibility. · `experimental/Dockerfile`

## Connections

- Launch → Ideation: starts · Main entrypoint launches the ideation workflow.
- OE launch → Experiments: alt path · Experimental launcher reaches the execution stage.
- Ideation → LLM layer: calls · Idea generation relies on model backends.
- Experiments → LLM layer: calls · Experiment planning and coordination use LLM access.
- Writeup → LLM layer: calls · Writeup generation uses model assistance.
- Review → LLM layer: calls · Review generation uses model access.
- Ideation → nanoGPT: targets · Template prompt and examples feed ideation.
- Ideation → 2D diffusion: targets · Template prompt and examples feed ideation.
- Ideation → Grokking: targets · Template prompt and examples feed ideation.
- Experiments → Tensorf: runs · Experiment stage executes template code.
- Experiments → Sketch RNN: runs · Experiment stage executes template code.
- Experiments → Probes: runs · Experiment stage executes template code.
- Experiments → SEIR: runs · Experiment stage executes template code.
- Writeup → Examples: writes · Manuscripts and PDFs are materialized in run directories.
- Experiments → Examples: outputs · Runs write JSON, logs, and per-run artifacts.
- Data prep → nanoGPT: feeds · Dataset preparation scripts feed text-model templates.
- Data prep → Examples: mirrors · Example papers include prepared datasets used in runs.
- Docker → Experiments: supports · Containerization supports executing generated workloads.
- Review → AI reviews: analyzes · Review outputs are stored for offline analysis.
- Review → ICLR bench: analyzes · Benchmark review generation and scoring reuse review logic.
- AI reviews → Examples: ingests · Parsed AI papers are derived from generated artifacts.
- ICLR bench → Review: benchmarks · Review benchmark corpus evaluates review generation quality.
