# kellerjordan/modded-nanogpt architecture

> Architecture diagram of the GitHub repository kellerjordan/modded-nanogpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kellerjordan/modded-nanogpt
- Repository: https://github.com/kellerjordan/modded-nanogpt

## Overview

Modded-NanoGPT is a distributed FineWeb language-model training speedrun. The principal workflow is a track-1 run: load and schedule batches, train the GPT model, update dense and sparse parameters, then validate. Supporting branches include hashed n-gram embeddings, sparse value-embedding updates, sampled-softmax loss, CUDA-graph execution and tail averaging. README statements and source-index references establish the relationships shown; unsampled modules are represented only where the tree or README supports their role.

Main entry points: [`track_1_short/model/gpt.py`](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/gpt.py) (GPT model), [`train_gpt.py`](https://github.com/kellerjordan/modded-nanogpt/blob/master/train_gpt.py) (Track-1 trainer), [`track_1_short/data.py`](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/data.py) (Batch loading), [`track_1_short/training.py`](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/training.py) (Training loop)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_run["Training runtime"]
  node_trainer["Track-1 trainer<br/>[train_gpt.py]"]
  node_training["Training loop<br/>[training.py]"]
  node_validation["HellaSwag evaluation<br/>[hellaswag.py]"]
end

subgraph group_model["Model and data"]
  node_fineweb["FineWeb corpus<br/>[fineweb.py]"]
  node_batching["Batch loading<br/>[data.py]"]
  node_schedule["Training schedules<br/>[schedule.py]"]
  node_config["Run configuration<br/>[config.py]"]
  node_gpt["GPT model<br/>[gpt.py]"]
  node_attention["Attention layers<br/>[attention.py]"]
  node_prefix["Prefix prediction"]
  node_ngram[("Hashed n-gram table<br/>[ngram_table.py]")]
  node_sampled["Sampled softmax<br/>[sampled_softmax.py]"]
end

subgraph group_updates["Optimization"]
  node_anvil["ANVIL optimizer<br/>[anvil.py]"]
  node_sharding["Sharded row exchange<br/>[sharded_rows.py]"]
  node_valuepull["Value embedding pulls"]
  node_tail["Tail averaging<br/>[tail_average.py]"]
end

subgraph group_systems["GPU systems"]
  node_stepgraphs["Step graph replay<br/>[step_graphs.py]"]
  node_mlpkernel["FP8 MLP kernel<br/>[mlp.py]"]
  node_xentkernel["Cross-entropy kernel<br/>[cross_entropy.py]"]
end

node_user(("Run operator"))
node_distributed{{"Distributed GPUs"}}

node_user -->|"starts run"| node_trainer
node_trainer -->|"runs loop"| node_training
node_trainer -->|"loads batches"| node_batching
node_trainer -->|"builds model"| node_gpt
node_trainer -->|"validates run"| node_validation
node_fineweb -.->|"provides tokens"| node_batching
node_batching -->|"reads settings"| node_config
node_batching -->|"uses schedule"| node_schedule
node_batching -->|"prepares row ids"| node_ngram
node_training -->|"replays steps"| node_stepgraphs
node_stepgraphs -->|"runs forward"| node_gpt
node_stepgraphs -->|"updates parameters"| node_anvil
node_stepgraphs -->|"holds gradients"| node_ngram
node_gpt -->|"uses attention"| node_attention
node_gpt -->|"reads embeddings"| node_ngram
node_gpt -->|"uses loss path"| node_sampled
node_gpt -->|"runs MLP"| node_mlpkernel
node_gpt -->|"computes loss"| node_xentkernel
node_ngram -->|"exchanges rows"| node_sharding
node_anvil -->|"coordinates updates"| node_distributed
node_valuepull -->|"exchanges rows"| node_sharding
node_training -.->|"coordinates pulls"| node_valuepull
node_training -.->|"averages final weights"| node_tail
node_trainer -->|"runs across GPUs"| node_distributed
node_gpt -.->|"adds auxiliary loss"| node_prefix

click node_trainer "https://github.com/kellerjordan/modded-nanogpt/blob/master/train_gpt.py"
click node_training "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/training.py"
click node_validation "https://github.com/kellerjordan/modded-nanogpt/blob/master/evals/hellaswag.py"
click node_fineweb "https://github.com/kellerjordan/modded-nanogpt/blob/master/data/fineweb.py"
click node_batching "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/data.py"
click node_schedule "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/schedule.py"
click node_config "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/config.py"
click node_gpt "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/gpt.py"
click node_attention "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/attention.py"
click node_prefix "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/prefix_prediction.py"
click node_ngram "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/ngram_table.py"
click node_sampled "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/sampled_softmax.py"
click node_anvil "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/optim/anvil.py"
click node_sharding "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/sharded_rows.py"
click node_valuepull "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/value_embed_pull.py"
click node_tail "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/tail_average.py"
click node_stepgraphs "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/cuda_graphs/step_graphs.py"
click node_mlpkernel "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/kernels/mlp.py"
click node_xentkernel "https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/kernels/cross_entropy.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_trainer,node_training,node_validation toneBlue
class node_fineweb,node_batching,node_schedule,node_config,node_gpt,node_attention,node_prefix,node_ngram,node_sampled toneAmber
class node_anvil,node_sharding,node_valuepull,node_tail toneMint
class node_stepgraphs,node_mlpkernel,node_xentkernel toneRose
class node_user,node_distributed toneIndigo
```

## Components

### Training runtime

- [Track-1 trainer](https://github.com/kellerjordan/modded-nanogpt/blob/master/train_gpt.py): `train_gpt.py`
- [Training loop](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/training.py): `track_1_short/training.py`
- [HellaSwag evaluation](https://github.com/kellerjordan/modded-nanogpt/blob/master/evals/hellaswag.py): `evals/hellaswag.py`

### Model and data

- [FineWeb corpus](https://github.com/kellerjordan/modded-nanogpt/blob/master/data/fineweb.py): `data/fineweb.py`
- [Batch loading](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/data.py): `track_1_short/data.py`
- [Training schedules](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/schedule.py): `track_1_short/schedule.py`
- [Run configuration](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/config.py): `track_1_short/config.py`
- [GPT model](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/gpt.py): `track_1_short/model/gpt.py`
- [Attention layers](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/attention.py): `track_1_short/model/attention.py`
- [Prefix prediction](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/model/prefix_prediction.py): `track_1_short/model/prefix_prediction.py`
- [Hashed n-gram table](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/ngram_table.py): `track_1_short/ngram_table.py`
- [Sampled softmax](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/sampled_softmax.py): `track_1_short/sampled_softmax.py`

### Optimization

- [ANVIL optimizer](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/optim/anvil.py): `track_1_short/optim/anvil.py`
- [Sharded row exchange](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/sharded_rows.py): `track_1_short/sharded_rows.py`
- [Value embedding pulls](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/value_embed_pull.py): `track_1_short/perf/value_embed_pull.py`
- [Tail averaging](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/tail_average.py): `track_1_short/tail_average.py`

### GPU systems

- [Step graph replay](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/cuda_graphs/step_graphs.py): `track_1_short/perf/cuda_graphs/step_graphs.py`
- [FP8 MLP kernel](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/kernels/mlp.py): `track_1_short/perf/kernels/mlp.py`
- [Cross-entropy kernel](https://github.com/kellerjordan/modded-nanogpt/blob/master/track_1_short/perf/kernels/cross_entropy.py): `track_1_short/perf/kernels/cross_entropy.py`

### Other components

- **Run operator**
- **Distributed GPUs**

## Connections

- Run operator → Track-1 trainer: starts run (evidence: `README.md`)
- Track-1 trainer → Training loop: runs loop (evidence: `README.md`)
- Track-1 trainer → Batch loading: loads batches (evidence: `README.md`)
- Track-1 trainer → GPT model: builds model (evidence: `README.md`)
- Track-1 trainer → HellaSwag evaluation: validates run (evidence: `README.md`)
- FineWeb corpus → Batch loading: provides tokens (evidence: `README.md`)
- Batch loading → Run configuration: reads settings (evidence: `track_1_short/data.py`)
- Batch loading → Training schedules: uses schedule (evidence: `track_1_short/data.py`)
- Batch loading → Hashed n-gram table: prepares row ids (evidence: `track_1_short/data.py`)
- Training loop → Step graph replay: replays steps (evidence: `track_1_short/perf/cuda_graphs/step_graphs.py`)
- Step graph replay → GPT model: runs forward (evidence: `track_1_short/perf/cuda_graphs/step_graphs.py`)
- Step graph replay → ANVIL optimizer: updates parameters (evidence: `track_1_short/perf/cuda_graphs/step_graphs.py`)
- Step graph replay → Hashed n-gram table: holds gradients (evidence: `track_1_short/perf/cuda_graphs/step_graphs.py`)
- GPT model → Attention layers: uses attention (evidence: `track_1_short/model/gpt.py`)
- GPT model → Hashed n-gram table: reads embeddings (evidence: `track_1_short/model/gpt.py`)
- GPT model → Sampled softmax: uses loss path (evidence: `track_1_short/model/gpt.py`)
- GPT model → FP8 MLP kernel: runs MLP (evidence: `track_1_short/model/gpt.py`)
- GPT model → Cross-entropy kernel: computes loss (evidence: `track_1_short/model/gpt.py`)
- Hashed n-gram table → Sharded row exchange: exchanges rows (evidence: `track_1_short/ngram_table.py`)
- ANVIL optimizer → Distributed GPUs: coordinates updates (evidence: `track_1_short/optim/anvil.py`)
- Value embedding pulls → Sharded row exchange: exchanges rows (evidence: `track_1_short/perf/value_embed_pull.py`)
- Training loop → Value embedding pulls: coordinates pulls
- Training loop → Tail averaging: averages final weights
- Track-1 trainer → Distributed GPUs: runs across GPUs (evidence: `README.md`)
- GPT model → Prefix prediction: adds auxiliary loss (evidence: `README.md`)
