How sakanaai/ai-scientist works
This repository is an end-to-end system for automated scientific ideation, experimentation, paper writing, and review generation. The architecture is centered around a small orchestration layer in ai_scientist/ that drives many template-specific research workloads under templates/ and stores outputs under run directories such as example_papers/ and review_*.
1) High-level system shape
The repo has four main layers:
1. Core orchestration and LLM access
- launch_scientist.py
- ai_scientist/
- experimental/launch_oe_scientist.py
2. Task templates for research domains
- templates/ for reusable experiment/paper-generation blueprints
- data/ for dataset preparation used by templates
3. Generated research artifacts
- example_papers/ contains complete sample runs with code, logs, figures, LaTeX, reviews, and PDFs
- these directories show the expected output shape of a successful run
4. Evaluation / review benchmarking
- ai_scientist/perform_review.py
- review_ai_scientist/
- review_iclr_bench/
The repo is not a single application; it is a framework for launching many independent research experiments based on a template.
2) Core orchestration subsystem
The ai_scientist/ package contains the “agentic workflow” used during paper generation:
• generate_ideas.py handles idea generation from prompts and few-shot examples.
• perform_experiments.py runs or coordinates experiment execution for a selected idea.
• perform_writeup.py turns results into a paper-style writeup using a LaTeX template.
• perform_review.py generates paper reviews and loads PDFs/text for evaluation.
• llm.py abstracts model backends and provider-specific API access.
This core layer is the boundary between the high-level workflow and the actual template code. It is where LLM calls are made, where prompt construction happens, and where outputs are collected into the standardized run folders.
Important architectural boundary
The system intentionally separates:
• model-facing logic in ai_scientist/
from
• research-domain logic in each template’s experiment.py and related files.
That separation is important because the framework can swap templates while reusing the same ideation/writeup/review pipeline.
3) Template subsystem
The template folders define the scientific “world” the AI Scientist can work in. Each template is a self-contained research environment with a consistent interface.
Common template structure:
• experiment.py — main experiment entry point
• plot.py — converts results into figures
• prompt.json — describes the task and constraints for the LLM
• ideas.json and seed_ideas.json — example seed ideas and prior candidates
• latex/template.tex plus style/bibliography assets — final paper shell
• optional domain-specific support code and datasets
The repo includes both official and community-contributed templates. The most important baseline templates mentioned in the README are:
• templates/nanoGPT
• templates/2d_diffusion
• templates/grokking
Additional templates expand the framework to other domains:
• seir
• mobilenetV3
• sketch_rnn
• MACE
• earthquake-prediction
• tensorf
• probes
Template runtime contract
The framework expects templates to produce outputs in a predictable shape:
• a run_0 baseline and subsequent run_* directories
• JSON files summarizing metrics and final state
• plots for the paper
• a compiled PDF after LaTeX writeup
This contract is why the README emphasizes matching base filenames and output JSON structure when adding a new template.
4) Experiment execution flow
A typical experiment runs through this sequence:
1. Data preparation
- Some templates require preprocessing scripts in data/
- Example:
- data/enwik8/prepare.py
- data/shakespeare_char/prepare.py
- data/text8/prepare.py
- These scripts create the datasets consumed by the NanoGPT-style experiments
2. Baseline run
- Each template has a machine-specific baseline run, usually run_0
- Example:
- python experiment.py --out_dir run_0
- Baselines are important because they anchor comparisons on the local machine’s hardware and runtime characteristics
3. Idea generation
- ai_scientist/generate_ideas.py uses a selected LLM to propose candidate research ideas
- It is guided by the template prompt, few-shot examples, and optionally prior seed ideas
4. Experimentation
- ai_scientist/perform_experiments.py coordinates the selected idea into an actual code change / run loop
- The template’s experiment.py is the main execution target
- Outputs are written into run-specific directories and JSON summaries
5. Plotting
- Template-specific plot.py reads run results and creates publication-quality figures
6. Writeup
- ai_scientist/perform_writeup.py assembles results into a LaTeX manuscript using the template’s latex/template.tex
- The output is a paper PDF in the template or example-paper folder
7. Review
- ai_scientist/perform_review.py can score the generated paper and produce accept/reject style commentary
5) LLM and external service boundary
The repository is explicitly multi-provider. ai_scientist/llm.py is the abstraction layer for model selection and API calls.
Supported providers called out in the README include:
• OpenAI
• Anthropic
• DeepSeek
• OpenRouter
• Google Gemini
• Claude via Bedrock or Vertex AI
This matters architecturally because the core orchestration does not depend on one vendor; the provider logic is isolated behind the LLM layer.
Other external services
• Semantic Scholar or OpenAlex is used for literature search / novelty checking and citation gathering during the paper-generation workflow.
• LaTeX tooling is required for PDF generation, typically via pdflatex from a full TeX distribution.
• The repository warns that generated code is executed, so the runtime boundary includes process execution and containerization concerns.
6) Generated artifact subsystem
example_papers/ is a rich archive of completed experiments. Each example paper directory is essentially a snapshot of the entire workflow for one idea.
A typical example-paper directory contains:
• experiment.py or generated run_*.py
• run_* directories with final_info.json and related result files
• plot.py
• ideas.json, seed_ideas.json, and prompt.json
• notes.txt, log.txt, and review.txt
• latex/ with the paper source and styles
• final PDFs
This folder shows the output model of the whole system:
• raw experiment code
• experiment results
• figures
• manuscript source
• manuscript PDF
• review artifacts
That makes example_papers/ a useful reference boundary for understanding what “done” looks like in this architecture.
7) Review and benchmark subsystem
The review side is separated from paper generation.
review_ai_scientist/
This folder contains:
• parsed AI-generated papers
• model-specific review CSVs
• manual paper ratings
• notebook-based analysis and figure generation
It is essentially an offline analysis corpus for evaluating generated papers and paper reviews.
review_iclr_bench/
This is a larger benchmark dataset for paper review analysis:
• iclr_parsed/ contains text-parsed papers
• llm_reviews/ contains generated review CSVs from different models and configurations
• iclr_analysis.py performs batch review evaluation
• ratings_subset.tsv and notebooks support further analysis
Architecturally, this subsystem is independent from paper generation. It reuses the review logic but focuses on evaluation and benchmarking rather than producing new experiments.
8) Infrastructure and packaging
Supporting infrastructure is light but important:
• requirements.txt lists Python dependencies
• experimental/Dockerfile provides containerization support
• launch_scientist.py is the main entrypoint for running the system
• experimental/launch_oe_scientist.py suggests an alternate launch path or experimental mode
• LICENSE and the README’s responsible-use section define the operational and legal boundaries
The architecture assumes:
• Linux
• NVIDIA GPU acceleration for most templates
• Python 3.11
• external APIs for LLMs and literature search
• local filesystem storage for all run artifacts
9) Data and runtime flow summary
A diagram of this repo should show the following flow:
• Template metadata and few-shot examples feed the LLM-driven ideation layer.
• The selected idea is transformed into code-experiment runs using the template’s experiment.py.
• Runtime outputs are written into run folders as JSON, logs, and plots.
• The plotting layer converts results into figures.
• The writeup layer consumes results and plots to generate a LaTeX paper and PDF.
• The review layer consumes the paper text/PDF and returns scores, weaknesses, and an accept/reject decision.
• Separate benchmark folders collect generated and human-reviewed papers for offline analysis.
10) Key architectural constraints
Several constraints materially shape the architecture:
• Template rigidity: templates must conform to the repo’s expected naming and output format.
• Code execution risk: the system runs LLM-generated code, so sandboxing/containerization is a major operational boundary.
• GPU dependence: most meaningful workloads require CUDA-capable hardware.
• External API dependence: the system is only as strong as the chosen LLM and literature-search providers.
• Artifact-heavy workflow: the system is designed around persistent run directories and generated documents, not ephemeral in-memory computation.
Overall, this repository is best understood as a modular research automation pipeline: a generic LLM orchestration core, a set of domain templates, and a paper/review artifact pipeline that turns experimental runs into publishable-style outputs.
Main entry points: perform_experiments.py Experiments · generate_ideas.py Ideation · perform_review.py Review · llm.py LLM layer
Components 19
Core orchestration
Entry points, LLM access, and workflow stages for ideation through review.
- Launch · Main launcher for the scientist workflow.
launch_scientist.py - OE launch · Experimental alternate launcher.
experimental/launch_oe_scientist.py - LLM layer · Backend abstraction for model and API access.
ai_scientist/llm.py - Ideation · Prompts models to propose candidate research ideas.
ai_scientist/generate_ideas.py - Experiments · Coordinates experiment execution for selected ideas.
ai_scientist/perform_experiments.py - Writeup · Builds LaTeX manuscripts from results and figures.
ai_scientist/perform_writeup.py - Review · Generates paper reviews and evaluation outputs.
ai_scientist/perform_review.py
Research templates
Domain-specific experiment blueprints that define the scientific workload.
- nanoGPT · Baseline text-modeling template and canonical workload.
templates/nanoGPT - 2D diffusion · Toy diffusion research template.
templates/2d_diffusion - Grokking · Template for grokking-style experiments.
templates/grokking - Tensorf · 3D reconstruction / neural field template.
templates/tensorf - Sketch RNN · Sequence-to-sketch generation template.
templates/sketch_rnn - Probes · Probe-based analysis template.
templates/probes - SEIR · Epidemiology-style template.
templates/seir
Generated artifacts
Example run directories and produced papers that show the expected output shape.
- Examples · Completed run directories with code, logs, plots, LaTeX, and PDFs.
example_papers
Review benchmarking
Offline corpora and analysis for paper-review generation and benchmarking.
- AI reviews · Offline corpus of generated papers and model review scores.
review_ai_scientist - ICLR bench · Parsed papers, review CSVs, and analysis for review benchmarking.
review_iclr_bench
Connections 22
- Launch → Ideation: starts · Main entrypoint launches the ideation workflow.
- OE launch → Experiments: alt path · Experimental launcher reaches the execution stage.
- Ideation → LLM layer: calls · Idea generation relies on model backends.
- Experiments → LLM layer: calls · Experiment planning and coordination use LLM access.
- Writeup → LLM layer: calls · Writeup generation uses model assistance.
- Review → LLM layer: calls · Review generation uses model access.
- Ideation → nanoGPT: targets · Template prompt and examples feed ideation.
- Ideation → 2D diffusion: targets · Template prompt and examples feed ideation.
- Ideation → Grokking: targets · Template prompt and examples feed ideation.
- Experiments → Tensorf: runs · Experiment stage executes template code.
- Experiments → Sketch RNN: runs · Experiment stage executes template code.
- Experiments → Probes: runs · Experiment stage executes template code.
- Experiments → SEIR: runs · Experiment stage executes template code.
- Writeup → Examples: writes · Manuscripts and PDFs are materialized in run directories.
- Experiments → Examples: outputs · Runs write JSON, logs, and per-run artifacts.
- Data prep → nanoGPT: feeds · Dataset preparation scripts feed text-model templates.
- Data prep → Examples: mirrors · Example papers include prepared datasets used in runs.
- Docker → Experiments: supports · Containerization supports executing generated workloads.
- Review → AI reviews: analyzes · Review outputs are stored for offline analysis.
- Review → ICLR bench: analyzes · Benchmark review generation and scoring reuse review logic.
- AI reviews → Examples: ingests · Parsed AI papers are derived from generated artifacts.
- ICLR bench → Review: benchmarks · Review benchmark corpus evaluates review generation quality.