# karpathy/autoresearch architecture

> Architecture diagram of the GitHub repository karpathy/autoresearch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karpathy/autoresearch
- Repository: https://github.com/karpathy/autoresearch

## Overview

Autoresearch is a single-GPU autonomous LLM pretraining loop. An AI research agent reads the instructions, prepares data and a tokenizer once, edits `train.py`, runs a fixed five-minute experiment, evaluates validation bits-per-byte, and keeps or discards the change. The repository implements preparation, batching, GPT modeling, optimization, training, and evaluation; the autonomous orchestration and experiment history are described by the README rather than sampled implementation.

Main entry points: [`train.py`](https://github.com/karpathy/autoresearch/blob/master/train.py) (Training Loop), [`prepare.py`](https://github.com/karpathy/autoresearch/blob/master/prepare.py) (Preparation Runtime), [`program.md`](https://github.com/karpathy/autoresearch/blob/master/program.md) (Experiment Instructions)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_prep["Data Preparation"]
  node_prepare_runtime["Preparation Runtime<br/>[prepare.py]"]
  node_cache[("Data Cache<br/>[prepare.py]")]
  node_tokenizer[("BPE Tokenizer<br/>[prepare.py]")]
  node_batcher["Batch Loader<br/>[prepare.py]"]
end

subgraph group_train["Training Runtime"]
  node_train_loop["Training Loop<br/>[train.py]"]
  node_gpt["GPT Model<br/>[train.py]"]
  node_blocks["Transformer Blocks<br/>[train.py]"]
  node_attention["Causal Attention<br/>[train.py]"]
  node_mlp["MLP Layers<br/>[train.py]"]
  node_optimizer["Muon AdamW<br/>[train.py]"]
end

subgraph group_eval["Evaluation"]
  node_evaluator["Fixed Evaluator<br/>[prepare.py]"]
  node_bpb["Validation BPB<br/>[prepare.py]"]
end

subgraph group_loop["Research Iteration"]
  node_instructions["Experiment Instructions<br/>[program.md]"]
end

node_agent(("Research Agent"))
node_dataset[("Dataset Service")]

node_agent -->|"reads"| node_instructions
node_agent -->|"runs"| node_prepare_runtime
node_agent -->|"edits and runs"| node_train_loop
node_dataset -->|"supplies shards"| node_cache
node_prepare_runtime -->|"downloads data"| node_dataset
node_prepare_runtime -->|"writes shards"| node_cache
node_prepare_runtime -->|"trains tokenizer"| node_tokenizer
node_batcher -->|"reads shards"| node_cache
node_batcher -->|"encodes text"| node_tokenizer
node_train_loop -->|"loads batches"| node_batcher
node_train_loop -->|"runs model"| node_gpt
node_train_loop -->|"updates weights"| node_optimizer
node_gpt -->|"runs blocks"| node_blocks
node_blocks -->|"calls attention"| node_attention
node_blocks -->|"calls MLP"| node_mlp
node_train_loop -->|"evaluates model"| node_evaluator
node_evaluator -->|"computes metric"| node_bpb
node_agent -->|"checks result"| node_bpb

click node_instructions "https://github.com/karpathy/autoresearch/blob/master/program.md"
click node_prepare_runtime "https://github.com/karpathy/autoresearch/blob/master/prepare.py"
click node_cache "https://github.com/karpathy/autoresearch/blob/master/prepare.py"
click node_tokenizer "https://github.com/karpathy/autoresearch/blob/master/prepare.py"
click node_batcher "https://github.com/karpathy/autoresearch/blob/master/prepare.py"
click node_train_loop "https://github.com/karpathy/autoresearch/blob/master/train.py"
click node_gpt "https://github.com/karpathy/autoresearch/blob/master/train.py"
click node_blocks "https://github.com/karpathy/autoresearch/blob/master/train.py"
click node_attention "https://github.com/karpathy/autoresearch/blob/master/train.py"
click node_mlp "https://github.com/karpathy/autoresearch/blob/master/train.py"
click node_optimizer "https://github.com/karpathy/autoresearch/blob/master/train.py"
click node_evaluator "https://github.com/karpathy/autoresearch/blob/master/prepare.py"
click node_bpb "https://github.com/karpathy/autoresearch/blob/master/prepare.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_prepare_runtime,node_cache,node_tokenizer,node_batcher toneBlue
class node_train_loop,node_gpt,node_blocks,node_attention,node_mlp,node_optimizer,node_dataset toneAmber
class node_evaluator,node_bpb toneMint
class node_instructions toneRose
class node_agent toneIndigo
```

## Components

### Data Preparation

- [Preparation Runtime](https://github.com/karpathy/autoresearch/blob/master/prepare.py): `prepare.py`
- [Data Cache](https://github.com/karpathy/autoresearch/blob/master/prepare.py): `prepare.py`
- [BPE Tokenizer](https://github.com/karpathy/autoresearch/blob/master/prepare.py): `prepare.py`
- [Batch Loader](https://github.com/karpathy/autoresearch/blob/master/prepare.py): `prepare.py`

### Training Runtime

- [Training Loop](https://github.com/karpathy/autoresearch/blob/master/train.py): `train.py`
- [GPT Model](https://github.com/karpathy/autoresearch/blob/master/train.py): `train.py`
- [Transformer Blocks](https://github.com/karpathy/autoresearch/blob/master/train.py): `train.py`
- [Causal Attention](https://github.com/karpathy/autoresearch/blob/master/train.py): `train.py`
- [MLP Layers](https://github.com/karpathy/autoresearch/blob/master/train.py): `train.py`
- [Muon AdamW](https://github.com/karpathy/autoresearch/blob/master/train.py): `train.py`

### Evaluation

- [Fixed Evaluator](https://github.com/karpathy/autoresearch/blob/master/prepare.py): `prepare.py`
- [Validation BPB](https://github.com/karpathy/autoresearch/blob/master/prepare.py): `prepare.py`

### Research Iteration

- [Experiment Instructions](https://github.com/karpathy/autoresearch/blob/master/program.md): `program.md`

### Other components

- **Research Agent**
- **Dataset Service**

## Connections

- Research Agent → Experiment Instructions: reads
- Research Agent → Preparation Runtime: runs
- Research Agent → Training Loop: edits and runs
- Dataset Service → Data Cache: supplies shards
- Preparation Runtime → Dataset Service: downloads data
- Preparation Runtime → Data Cache: writes shards
- Preparation Runtime → BPE Tokenizer: trains tokenizer
- Batch Loader → Data Cache: reads shards
- Batch Loader → BPE Tokenizer: encodes text
- Training Loop → Batch Loader: loads batches
- Training Loop → GPT Model: runs model
- Training Loop → Muon AdamW: updates weights
- GPT Model → Transformer Blocks: runs blocks
- Transformer Blocks → Causal Attention: calls attention
- Transformer Blocks → MLP Layers: calls MLP
- Training Loop → Fixed Evaluator: evaluates model
- Fixed Evaluator → Validation BPB: computes metric
- Research Agent → Validation BPB: checks result
