How rasbt/llms-from-scratch works
This repository is educational code for building, training, and adapting GPT-like language models from scratch. A principal workflow moves from text preparation and model construction to pretraining, then fine-tuning and generation or evaluation; classification is a separate fine-tuning outcome. The code also covers model variants, distributed training, and LoRA. The README establishes these broad stages, while sampled files show particular implementations and interfaces. Many chapter modules were not sampled, so edges between those subsystems are limited to documented relationships.
Main entry points: gpt.py GPT architecture · gpt_train.py GPT pretraining · gpt_instruction_finetuning.py Instruction fine-tuning · ch02 Text preparation
Components 20
Text and data
- Text preparation
ch02 - Instruction datasets
ch07/01_main-chapter-code/exercise_experiments.py
Model implementations
- GPT architecture
ch04/01_main-chapter-code/gpt.py - Model package modules
pkg/llms_from_scratch - Qwen3 model
pkg/llms_from_scratch/qwen3.py - Attention variants
ch04
Training and adaptation
- GPT pretraining
ch05/01_main-chapter-code/gpt_train.py - Muon training
ch05/18_muon/gpt_train_muon.py - Hyperparameter search
ch05/05_bonus_hparam_tuning/hparam_search.py - Classification fine-tuning
ch06/01_main-chapter-code/gpt_class_finetune.py - Instruction fine-tuning
ch07/01_main-chapter-code/gpt_instruction_finetuning.py - LoRA fine-tuning
appendix-E/01_main-chapter-code/previous_chapters.py - Distributed training
ch05/10_llm-training-speed/02_opt_multi_gpu_ddp.py
Inference and evaluation
- Pretrained weight loading
ch05/01_main-chapter-code/gpt_download.py - Text generation
ch05/01_main-chapter-code/gpt_generate.py - Model evaluation
ch07/01_main-chapter-code/ollama_evaluate.py
User interfaces
- Review classifier UI
ch06/04_user_interface/app.py - Instruction chat UI
ch07/06_user_interface/app.py - Qwen chat interfaces
ch05/11_qwen3/qwen3-chat-interface
Other components
- Engineer / learner
Connections 20
- Engineer / learner → Text preparation: builds text inputs
README.md - Text preparation → GPT architecture: prepares model inputs
- Engineer / learner → GPT pretraining: develops and trains
README.md - GPT architecture → GPT pretraining: trains model
- GPT pretraining → Text generation: enables generation
- Engineer / learner → Classification fine-tuning: fine-tunes model
README.md - Engineer / learner → Instruction fine-tuning: fine-tunes model
README.md - Instruction datasets → Instruction fine-tuning: formats examples
ch07/01_main-chapter-code/exercise_experiments.py - Pretrained weight loading → GPT architecture: loads weights into
- GPT architecture → Text generation: generates tokens
ch05/01_main-chapter-code/gpt_generate.py - Hyperparameter search → Model package modules: imports model components
ch05/05_bonus_hparam_tuning/hparam_search.py - Muon training → GPT pretraining: trains GPT variant
- Distributed training → GPT pretraining: trains across GPUs
- LoRA fine-tuning → Instruction fine-tuning: adapts model
- Attention variants → GPT architecture: extends attention designs
- Qwen3 model → Qwen chat interfaces: powers chat
- Classification fine-tuning → Review classifier UI: serves predictions to
ch06/04_user_interface/app.py - Instruction fine-tuning → Instruction chat UI: produces fine-tuned weights
ch07/06_user_interface/app.py - Instruction chat UI → Model package modules: uses generation functions
ch07/06_user_interface/app.py - Engineer / learner → Model evaluation: evaluates models
README.md