How kellerjordan/modded-nanogpt works
Modded-NanoGPT is a distributed FineWeb language-model training speedrun. The principal workflow is a track-1 run: load and schedule batches, train the GPT model, update dense and sparse parameters, then validate. Supporting branches include hashed n-gram embeddings, sparse value-embedding updates, sampled-softmax loss, CUDA-graph execution and tail averaging. README statements and source-index references establish the relationships shown; unsampled modules are represented only where the tree or README supports their role.
Main entry points: gpt.py GPT model · train_gpt.py Track-1 trainer · data.py Batch loading · training.py Training loop
Components 21
Training runtime
- Track-1 trainer
train_gpt.py - Training loop
track_1_short/training.py - HellaSwag evaluation
evals/hellaswag.py
Model and data
- FineWeb corpus
data/fineweb.py - Batch loading
track_1_short/data.py - Training schedules
track_1_short/schedule.py - Run configuration
track_1_short/config.py - GPT model
track_1_short/model/gpt.py - Attention layers
track_1_short/model/attention.py - Prefix prediction
track_1_short/model/prefix_prediction.py - Hashed n-gram table
track_1_short/ngram_table.py - Sampled softmax
track_1_short/sampled_softmax.py
Optimization
- ANVIL optimizer
track_1_short/optim/anvil.py - Sharded row exchange
track_1_short/sharded_rows.py - Value embedding pulls
track_1_short/perf/value_embed_pull.py - Tail averaging
track_1_short/tail_average.py
GPU systems
- Step graph replay
track_1_short/perf/cuda_graphs/step_graphs.py - FP8 MLP kernel
track_1_short/perf/kernels/mlp.py - Cross-entropy kernel
track_1_short/perf/kernels/cross_entropy.py
Other components
- Run operator
- Distributed GPUs
Connections 25
- Run operator → Track-1 trainer: starts run
README.md - Track-1 trainer → Training loop: runs loop
README.md - Track-1 trainer → Batch loading: loads batches
README.md - Track-1 trainer → GPT model: builds model
README.md - Track-1 trainer → HellaSwag evaluation: validates run
README.md - FineWeb corpus → Batch loading: provides tokens
README.md - Batch loading → Run configuration: reads settings
track_1_short/data.py - Batch loading → Training schedules: uses schedule
track_1_short/data.py - Batch loading → Hashed n-gram table: prepares row ids
track_1_short/data.py - Training loop → Step graph replay: replays steps
track_1_short/perf/cuda_graphs/step_graphs.py - Step graph replay → GPT model: runs forward
track_1_short/perf/cuda_graphs/step_graphs.py - Step graph replay → ANVIL optimizer: updates parameters
track_1_short/perf/cuda_graphs/step_graphs.py - Step graph replay → Hashed n-gram table: holds gradients
track_1_short/perf/cuda_graphs/step_graphs.py - GPT model → Attention layers: uses attention
track_1_short/model/gpt.py - GPT model → Hashed n-gram table: reads embeddings
track_1_short/model/gpt.py - GPT model → Sampled softmax: uses loss path
track_1_short/model/gpt.py - GPT model → FP8 MLP kernel: runs MLP
track_1_short/model/gpt.py - GPT model → Cross-entropy kernel: computes loss
track_1_short/model/gpt.py - Hashed n-gram table → Sharded row exchange: exchanges rows
track_1_short/ngram_table.py - ANVIL optimizer → Distributed GPUs: coordinates updates
track_1_short/optim/anvil.py - Value embedding pulls → Sharded row exchange: exchanges rows
track_1_short/perf/value_embed_pull.py - Training loop → Value embedding pulls: coordinates pulls
- Training loop → Tail averaging: averages final weights
- Track-1 trainer → Distributed GPUs: runs across GPUs
README.md - GPT model → Prefix prediction: adds auxiliary loss
README.md