# karpathy/nanogpt architecture

> Architecture diagram of the GitHub repository karpathy/nanogpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/karpathy/nanogpt
- Repository: https://github.com/karpathy/nanogpt
- Video tour (about a minute): https://gitdiagram.com/karpathy/nanogpt/video

## Overview

nanoGPT is a compact GPT training and text-generation repository. Its main workflow prepares a dataset into tokenized binary splits, trains or evaluates a GPT model with configurable presets, saves checkpoints, then loads a checkpoint or pretrained GPT-2 weights to generate text. The README also documents character-level and GPT-2 BPE datasets, pretrained-model evaluation, finetuning, and distributed training. Source excerpts show model internals, training, preprocessing, configuration, and sampling; some documented capabilities have no sampled implementation detail, so their links are limited to README evidence.

Main entry points: [`train.py`](https://github.com/karpathy/nanogpt/blob/master/train.py) (Train and evaluate), [`sample.py`](https://github.com/karpathy/nanogpt/blob/master/sample.py) (Sampling runner), [`data/openwebtext/prepare.py`](https://github.com/karpathy/nanogpt/blob/master/data/openwebtext/prepare.py) (OpenWebText prep), [`data/shakespeare/prepare.py`](https://github.com/karpathy/nanogpt/blob/master/data/shakespeare/prepare.py) (Shakespeare BPE prep)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_openwebtext[("OpenWebText source")]
  node_shakespeare_source["Shakespeare text source"]
  node_owt_prep["OpenWebText prep<br/>[prepare.py]"]
  node_shakespeare_prep["Shakespeare BPE prep<br/>[prepare.py]"]
  node_char_prep["Character prep<br/>[prepare.py]"]
  node_token_data[("Tokenized data")]
end

subgraph group_training["Training and evaluation"]
  node_runner["Train and evaluate<br/>[train.py]"]
  node_configurator["Config overrides<br/>[configurator.py]"]
  node_presets["Training presets<br/>[train_gpt2.py]"]
  node_finetune_preset["Finetune preset"]
  node_eval_presets["GPT-2 eval presets<br/>[eval_gpt2_large.py]"]
  node_char_preset["Character preset"]
  node_checkpoint[("Model checkpoint")]
end

subgraph group_model["GPT model"]
  node_gpt["GPT language model<br/>[model.py]"]
  node_attention["Causal attention<br/>[model.py]"]
  node_transformer_block["Transformer block<br/>[model.py]"]
end

subgraph group_generation["Text generation"]
  node_sampler["Sampling runner<br/>[sample.py]"]
  node_generated_text["Generated text"]
end

node_user(("Researcher"))

node_user -->|"prepares corpus"| node_owt_prep
node_user -->|"prepares corpus"| node_shakespeare_prep
node_user -->|"prepares corpus"| node_char_prep
node_openwebtext -->|"loads dataset"| node_owt_prep
node_shakespeare_source -->|"downloads text"| node_shakespeare_prep
node_shakespeare_source -->|"downloads text"| node_char_prep
node_owt_prep -->|"writes token bins"| node_token_data
node_shakespeare_prep -->|"writes token bins"| node_token_data
node_char_prep -->|"writes bins and metadata"| node_token_data
node_user -->|"trains or evaluates"| node_runner
node_presets -->|"configures"| node_runner
node_finetune_preset -->|"configures"| node_runner
node_eval_presets -->|"configures"| node_runner
node_char_preset -->|"configures"| node_runner
node_runner -->|"executes overrides"| node_configurator
node_runner -->|"reads batches"| node_token_data
node_runner -->|"initializes and trains"| node_gpt
node_gpt -->|"runs blocks"| node_transformer_block
node_transformer_block -->|"calls attention"| node_attention
node_runner -->|"saves checkpoint"| node_checkpoint
node_user -->|"requests samples"| node_sampler
node_checkpoint -->|"loads checkpoint"| node_sampler
node_sampler -->|"generates with"| node_gpt
node_sampler -->|"prints samples"| node_generated_text

click node_owt_prep "https://github.com/karpathy/nanogpt/blob/master/data/openwebtext/prepare.py"
click node_shakespeare_prep "https://github.com/karpathy/nanogpt/blob/master/data/shakespeare/prepare.py"
click node_char_prep "https://github.com/karpathy/nanogpt/blob/master/data/shakespeare_char/prepare.py"
click node_runner "https://github.com/karpathy/nanogpt/blob/master/train.py"
click node_configurator "https://github.com/karpathy/nanogpt/blob/master/configurator.py"
click node_presets "https://github.com/karpathy/nanogpt/blob/master/config/train_gpt2.py"
click node_finetune_preset "https://github.com/karpathy/nanogpt/blob/master/config/finetune_shakespeare.py"
click node_eval_presets "https://github.com/karpathy/nanogpt/blob/master/config/eval_gpt2_large.py"
click node_char_preset "https://github.com/karpathy/nanogpt/blob/master/config/train_shakespeare_char.py"
click node_gpt "https://github.com/karpathy/nanogpt/blob/master/model.py"
click node_attention "https://github.com/karpathy/nanogpt/blob/master/model.py"
click node_transformer_block "https://github.com/karpathy/nanogpt/blob/master/model.py"
click node_sampler "https://github.com/karpathy/nanogpt/blob/master/sample.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_openwebtext,node_shakespeare_source,node_owt_prep,node_shakespeare_prep,node_char_prep,node_token_data toneBlue
class node_runner,node_configurator,node_presets,node_finetune_preset,node_eval_presets,node_char_preset,node_checkpoint toneAmber
class node_gpt,node_attention,node_transformer_block toneMint
class node_sampler,node_generated_text toneRose
class node_user toneIndigo
```

## Components

### Data preparation

- **OpenWebText source**
- **Shakespeare text source**
- [OpenWebText prep](https://github.com/karpathy/nanogpt/blob/master/data/openwebtext/prepare.py): `data/openwebtext/prepare.py`
- [Shakespeare BPE prep](https://github.com/karpathy/nanogpt/blob/master/data/shakespeare/prepare.py): `data/shakespeare/prepare.py`
- [Character prep](https://github.com/karpathy/nanogpt/blob/master/data/shakespeare_char/prepare.py): `data/shakespeare_char/prepare.py`
- **Tokenized data**

### Training and evaluation

- [Train and evaluate](https://github.com/karpathy/nanogpt/blob/master/train.py): `train.py`
- [Config overrides](https://github.com/karpathy/nanogpt/blob/master/configurator.py): `configurator.py`
- [Training presets](https://github.com/karpathy/nanogpt/blob/master/config/train_gpt2.py): `config/train_gpt2.py`
- [Finetune preset](https://github.com/karpathy/nanogpt/blob/master/config/finetune_shakespeare.py): `config/finetune_shakespeare.py`
- [GPT-2 eval presets](https://github.com/karpathy/nanogpt/blob/master/config/eval_gpt2_large.py): `config/eval_gpt2_large.py`
- [Character preset](https://github.com/karpathy/nanogpt/blob/master/config/train_shakespeare_char.py): `config/train_shakespeare_char.py`
- **Model checkpoint**

### GPT model

- [GPT language model](https://github.com/karpathy/nanogpt/blob/master/model.py): `model.py`
- [Causal attention](https://github.com/karpathy/nanogpt/blob/master/model.py): `model.py`
- [Transformer block](https://github.com/karpathy/nanogpt/blob/master/model.py): `model.py`

### Text generation

- [Sampling runner](https://github.com/karpathy/nanogpt/blob/master/sample.py): `sample.py`
- **Generated text**

### Other components

- **Researcher**

## Connections

- Researcher → OpenWebText prep: prepares corpus (evidence: `README.md`)
- Researcher → Shakespeare BPE prep: prepares corpus (evidence: `README.md`)
- Researcher → Character prep: prepares corpus (evidence: `README.md`)
- OpenWebText source → OpenWebText prep: loads dataset (evidence: `data/openwebtext/prepare.py`)
- Shakespeare text source → Shakespeare BPE prep: downloads text (evidence: `data/shakespeare/prepare.py`)
- Shakespeare text source → Character prep: downloads text (evidence: `data/shakespeare_char/prepare.py`)
- OpenWebText prep → Tokenized data: writes token bins (evidence: `data/openwebtext/prepare.py`)
- Shakespeare BPE prep → Tokenized data: writes token bins (evidence: `data/shakespeare/prepare.py`)
- Character prep → Tokenized data: writes bins and metadata (evidence: `data/shakespeare_char/prepare.py`)
- Researcher → Train and evaluate: trains or evaluates (evidence: `README.md`)
- Training presets → Train and evaluate: configures (evidence: `README.md`)
- Finetune preset → Train and evaluate: configures (evidence: `README.md`)
- GPT-2 eval presets → Train and evaluate: configures (evidence: `README.md`)
- Character preset → Train and evaluate: configures (evidence: `README.md`)
- Train and evaluate → Config overrides: executes overrides (evidence: `train.py`)
- Train and evaluate → Tokenized data: reads batches (evidence: `train.py`)
- Train and evaluate → GPT language model: initializes and trains (evidence: `train.py`)
- GPT language model → Transformer block: runs blocks (evidence: `model.py`)
- Transformer block → Causal attention: calls attention (evidence: `model.py`)
- Train and evaluate → Model checkpoint: saves checkpoint (evidence: `train.py`)
- Researcher → Sampling runner: requests samples (evidence: `README.md`)
- Model checkpoint → Sampling runner: loads checkpoint (evidence: `README.md`)
- Sampling runner → GPT language model: generates with (evidence: `sample.py`)
- Sampling runner → Generated text: prints samples (evidence: `sample.py`)
