How ietamaher/llms-from-scratch works
This repository is an educational, PyTorch-based implementation of GPT-like language models. The principal workflow is user text through tokenization, transformer inference, and generated or classified results. Its broader lifecycle covers corpus preparation, pretraining, supervised classification, instruction tuning, preference tuning, and parameter-efficient LoRA adaptation. Sampled files show the Chainlit application, model internals, datasets, distributed training, Qwen3, and tuning utilities; several chapter capabilities are represented by their exact enclosing paths where source was not sampled.
Components 19
Data and Tokenization
- GPT-2 Tokenizer
ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py - Dataset Utilities
pkg/llms_from_scratch/ch02.py - Training Corpus
ch05/03_bonus_pretraining_on_gutenberg/prepare_dataset.py
Model Runtime
- GPT Transformer
ch04/01_main-chapter-code/gpt.py - Qwen3 Model
pkg/llms_from_scratch/qwen3.py - Text Generation
pkg/llms_from_scratch/generate.py - Cached Generation
pkg/llms_from_scratch/kv_cache/generate.py
Training and Adaptation
- Pretraining
ch05/01_main-chapter-code/gpt_train.py - Distributed Training
ch05/10_llm-training-speed/02_opt_multi_gpu_ddp.py - Classification Tuning
ch06/01_main-chapter-code/gpt_class_finetune.py - Instruction Tuning
ch07/01_main-chapter-code/gpt_instruction_finetuning.py - Preference Tuning
ch07/04_preference-tuning-with-dpo - LoRA Adaptation
appendix-E/01_main-chapter-code/previous_chapters.py
Applications
- Review Classifier UI
ch06/04_user_interface/app.py - Instruction Chat UI
ch07/06_user_interface/app.py
Model Extensions
- Model Weights
ch05/01_main-chapter-code/gpt_download.py - Attention Variants
ch04/09_dsa/gpt_with_kv_dsa.py
Other components
- User
- Weight Provider
Connections 28
- User → Instruction Chat UI: sends prompt
- Instruction Chat UI → GPT-2 Tokenizer: encodes text
- Instruction Chat UI → Text Generation: requests generation
- Text Generation → GPT Transformer: runs model
- Instruction Chat UI → User: returns response
- User → Review Classifier UI: submits review
- Review Classifier UI → GPT-2 Tokenizer: encodes review
- Review Classifier UI → GPT Transformer: classifies text
- Review Classifier UI → User: returns label
- Training Corpus → Dataset Utilities: provides text
- Dataset Utilities → Pretraining: loads batches
- Pretraining → GPT Transformer: trains model
- Distributed Training → Dataset Utilities: loads batches
- Distributed Training → GPT Transformer: trains replicas
- Instruction Tuning → Dataset Utilities: loads examples
- Instruction Tuning → GPT Transformer: fine tunes
- Classification Tuning → Dataset Utilities: loads labels
- Classification Tuning → GPT Transformer: fine tunes
- Preference Tuning → GPT Transformer: optimizes preferences
- LoRA Adaptation → GPT Transformer: adds adapters
- GPT Transformer → Text Generation: produces logits
- Text Generation → GPT-2 Tokenizer: decodes tokens
- Instruction Chat UI → GPT-2 Tokenizer: decodes output
- Cached Generation → GPT Transformer: uses cached model
- Qwen3 Model → Cached Generation: supports inference
- Attention Variants → Text Generation: supports inference
- Weight Provider → Model Weights: supplies weights
- Model Weights → GPT Transformer: loads weights