How raiyanyahya/how-to-train-your-gpt works
How to Train Your GPT is an interactive textbook with a runnable, single-file GPT training and inference example. The principal workflow is to follow the chapters from tokenization through model construction, training, and inference, then explore the concepts in companion notebooks or run main.py. The graph includes documented standalone explainers and sampled runtime components. Runtime wiring beyond the shown excerpts is omitted where it cannot be confirmed.
Main entry points: main.py Training Runner · 00_overview.md GPT Overview · 02_tokenization.md Tokenization · 03_embeddings.md Embeddings
Components 21
Core Curriculum
- GPT Overview
chapters/00_overview.md - Tokenization
chapters/02_tokenization.md - Embeddings
chapters/03_embeddings.md - Position Encoding
chapters/04_positional_encoding.md - Attention
chapters/05_attention.md - Transformer Block
chapters/06_transformer_block.md - GPT Model
chapters/07_gpt_model.md - Training Pipeline
chapters/08_training.md - Inference
chapters/09_inference.md - Full Script Guide
chapters/10_full_script.md
Topic Explainers
- Topic Guides
explanations and examples WIP
Runnable GPT
- BPE Tokenizer
main.py - Text Dataset
main.py - GPT Model Code
main.py - Transformer Components
main.py - AdamW Optimizer
main.py - Cosine Warmup
main.py - Training Runner
main.py - Token Generation
main.py
Interactive Notebooks
- Chapter Notebooks
notebooks
Other components
- Learner
Connections 19
- Learner → GPT Overview: starts with
- GPT Overview → Tokenization: continues to
- Tokenization → Embeddings: continues to
- Embeddings → Position Encoding: continues to
- Position Encoding → Attention: continues to
- Attention → Transformer Block: continues to
- Transformer Block → GPT Model: continues to
- GPT Model → Training Pipeline: continues to
- Training Pipeline → Inference: continues to
- Inference → Full Script Guide: continues to
- Learner → Topic Guides: explores
- Learner → Chapter Notebooks: runs examples
- BPE Tokenizer → Text Dataset: encodes text
- Text Dataset → Training Runner: provides batches
- Training Runner → GPT Model Code: trains
- Training Runner → AdamW Optimizer: updates with
- Training Runner → Cosine Warmup: sets learning rate
- GPT Model Code → Transformer Components: calls layers
- GPT Model Code → Token Generation: generates tokens