How karpathy/nanogpt works
nanoGPT is a compact GPT training and text-generation repository. Its main workflow prepares a dataset into tokenized binary splits, trains or evaluates a GPT model with configurable presets, saves checkpoints, then loads a checkpoint or pretrained GPT-2 weights to generate text. The README also documents character-level and GPT-2 BPE datasets, pretrained-model evaluation, finetuning, and distributed training. Source excerpts show model internals, training, preprocessing, configuration, and sampling; some documented capabilities have no sampled implementation detail, so their links are limited to README evidence.
Main entry points: train.py Train and evaluate · sample.py Sampling runner · openwebtext/prepare.py OpenWebText prep · shakespeare/prepare.py Shakespeare BPE prep
Components 19
Data preparation
- OpenWebText source
- Shakespeare text source
- OpenWebText prep
data/openwebtext/prepare.py - Shakespeare BPE prep
data/shakespeare/prepare.py - Character prep
data/shakespeare_char/prepare.py - Tokenized data
Training and evaluation
- Train and evaluate
train.py - Config overrides
configurator.py - Training presets
config/train_gpt2.py - Finetune preset
config/finetune_shakespeare.py - GPT-2 eval presets
config/eval_gpt2_large.py - Character preset
config/train_shakespeare_char.py - Model checkpoint
GPT model
- GPT language model
model.py - Causal attention
model.py - Transformer block
model.py
Text generation
- Sampling runner
sample.py - Generated text
Other components
- Researcher
Connections 24
- Researcher → OpenWebText prep: prepares corpus
README.md - Researcher → Shakespeare BPE prep: prepares corpus
README.md - Researcher → Character prep: prepares corpus
README.md - OpenWebText source → OpenWebText prep: loads dataset
data/openwebtext/prepare.py - Shakespeare text source → Shakespeare BPE prep: downloads text
data/shakespeare/prepare.py - Shakespeare text source → Character prep: downloads text
data/shakespeare_char/prepare.py - OpenWebText prep → Tokenized data: writes token bins
data/openwebtext/prepare.py - Shakespeare BPE prep → Tokenized data: writes token bins
data/shakespeare/prepare.py - Character prep → Tokenized data: writes bins and metadata
data/shakespeare_char/prepare.py - Researcher → Train and evaluate: trains or evaluates
README.md - Training presets → Train and evaluate: configures
README.md - Finetune preset → Train and evaluate: configures
README.md - GPT-2 eval presets → Train and evaluate: configures
README.md - Character preset → Train and evaluate: configures
README.md - Train and evaluate → Config overrides: executes overrides
train.py - Train and evaluate → Tokenized data: reads batches
train.py - Train and evaluate → GPT language model: initializes and trains
train.py - GPT language model → Transformer block: runs blocks
model.py - Transformer block → Causal attention: calls attention
model.py - Train and evaluate → Model checkpoint: saves checkpoint
train.py - Researcher → Sampling runner: requests samples
README.md - Model checkpoint → Sampling runner: loads checkpoint
README.md - Sampling runner → GPT language model: generates with
sample.py - Sampling runner → Generated text: prints samples
sample.py