How karpathy/autoresearch works
Autoresearch is a single-GPU autonomous LLM pretraining loop. An AI research agent reads the instructions, prepares data and a tokenizer once, edits train.py, runs a fixed five-minute experiment, evaluates validation bits-per-byte, and keeps or discards the change. The repository implements preparation, batching, GPT modeling, optimization, training, and evaluation; the autonomous orchestration and experiment history are described by the README rather than sampled implementation.
Main entry points: train.py Training Loop · prepare.py Preparation Runtime · program.md Experiment Instructions
Components 15
Data Preparation
- Preparation Runtime
prepare.py - Data Cache
prepare.py - BPE Tokenizer
prepare.py - Batch Loader
prepare.py
Training Runtime
- Training Loop
train.py - GPT Model
train.py - Transformer Blocks
train.py - Causal Attention
train.py - MLP Layers
train.py - Muon AdamW
train.py
Evaluation
- Fixed Evaluator
prepare.py - Validation BPB
prepare.py
Research Iteration
- Experiment Instructions
program.md
Other components
- Research Agent
- Dataset Service
Connections 18
- Research Agent → Experiment Instructions: reads
- Research Agent → Preparation Runtime: runs
- Research Agent → Training Loop: edits and runs
- Dataset Service → Data Cache: supplies shards
- Preparation Runtime → Dataset Service: downloads data
- Preparation Runtime → Data Cache: writes shards
- Preparation Runtime → BPE Tokenizer: trains tokenizer
- Batch Loader → Data Cache: reads shards
- Batch Loader → BPE Tokenizer: encodes text
- Training Loop → Batch Loader: loads batches
- Training Loop → GPT Model: runs model
- Training Loop → Muon AdamW: updates weights
- GPT Model → Transformer Blocks: runs blocks
- Transformer Blocks → Causal Attention: calls attention
- Transformer Blocks → MLP Layers: calls MLP
- Training Loop → Fixed Evaluator: evaluates model
- Fixed Evaluator → Validation BPB: computes metric
- Research Agent → Validation BPB: checks result