How datamonk15/llms-from-scratch works
This is an educational codebase for building, pretraining, and fine-tuning GPT-like language models from scratch, with examples for loading larger pretrained weights. Its main learning path runs from text/token processing and attention through GPT construction, pretraining, and task-specific fine-tuning; instruction tuning and interactive demos extend that path. The repository also contains alternative architectures and training experiments. Links between chapter stages are documented at a high level in the README; unsampled implementations are represented by their exact tree paths, without inferring internal calls.
Main entry points: ch04.py GPT model · ch05.py Pretraining · gpt_class_finetune.py Classification tuning · app.py Instruction chat UI
Components 19
Text and models
- Text processing
pkg/llms_from_scratch/ch02.py - Attention layers
pkg/llms_from_scratch/ch03.py - GPT model
pkg/llms_from_scratch/ch04.py - GPT-2 BPE
ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py - Pretrained weights
ch05/01_main-chapter-code/gpt_download.py
Training and tuning
- Pretraining
pkg/llms_from_scratch/ch05.py - Gutenberg training
ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py - Hyperparameter search
ch05/05_bonus_hparam_tuning/hparam_search.py - Muon training
ch05/18_muon/gpt_train_muon.py - Classification tuning
ch06/01_main-chapter-code/gpt_class_finetune.py - Fine-tuning experiments
ch06/02_bonus_additional-experiments/additional_experiments.py - Instruction tuning
ch07/01_main-chapter-code/gpt_instruction_finetuning.py - Preference tuning
ch07/04_preference-tuning-with-dpo/previous_chapters.py
Applications and variants
- Instruction chat UI
ch07/06_user_interface/app.py - Classification UI
ch06/04_user_interface/app.py - Qwen models
pkg/llms_from_scratch/qwen3.py - Cached generation
pkg/llms_from_scratch/kv_cache/generate.py - Llama models
pkg/llms_from_scratch/llama3.py
Other components
- Learner
Connections 22
- Learner → Text processing: learns tokenization
README.md - Text processing → Attention layers: builds toward
README.md - Attention layers → GPT model: builds toward
README.md - GPT model → Pretraining: pretrains model
README.md - Pretrained weights → Classification tuning: supplies weights
ch06/01_main-chapter-code/gpt_class_finetune.py - Pretraining → Classification tuning: fine-tunes model
README.md - Pretraining → Gutenberg training: specializes training
ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py - Pretraining → Hyperparameter search: tunes training
- Pretraining → Muon training: uses variant optimizer
- Classification tuning → Fine-tuning experiments: extends experiments
- Classification tuning → Instruction tuning: fine-tunes further
README.md - Instruction tuning → Preference tuning: extends tuning
- Learner → Instruction chat UI: submits instruction
ch07/06_user_interface/app.py - Instruction chat UI → GPT model: instantiates model
ch07/06_user_interface/app.py - Instruction chat UI → Pretraining: generates response
ch07/06_user_interface/app.py - Learner → Classification UI: submits review
ch06/04_user_interface/app.py - Classification UI → GPT model: instantiates model
ch06/04_user_interface/app.py - Classification UI → Classification tuning: classifies review
ch06/04_user_interface/app.py - GPT model → Qwen models: model alternative
- GPT model → Llama models: model alternative
- GPT model → Cached generation: generation alternative
- GPT-2 BPE → Text processing: encodes text