How salmanmik/tiny-llm-from-scratch works
This repository is a hands-on NumPy course that builds a tiny language model, progressing from text tokenization through model training and generated text. The README describes the chapters and textbook as product capabilities; sampled code confirms the standalone demos and the final training flow. The graph treats chapter files as learning stages, not as imported modules: the README says to run each file, while sampled files do not import one another. Step 8’s excerpt confirms its internal model, gradient-check, optimizer, and generation responsibilities; the textbook’s contents were not sampled.
Main entry points: step8_train_tiny_gpt.py GPT training · step1_tokenizer.py Tokenization · step2_embeddings.py Embeddings · step3_bigram_train.py Bigram training
Components 14
Model-building course
- Tokenization
code/step1_tokenizer.py - Embeddings
code/step2_embeddings.py - Bigram training
code/step3_bigram_train.py - Self attention
code/step4_attention.py - Heads and position
code/step5_multihead_positional.py - Transformer block
code/step6_transformer_block.py - Tiny GPT
code/step7_tiny_gpt.py - GPT training
code/step8_train_tiny_gpt.py - Gradient check
code/step8_train_tiny_gpt.py - Adam optimizer
code/step8_train_tiny_gpt.py - Text generation
code/step8_train_tiny_gpt.py
Interactive textbook
- Course textbook
textbook/index.html
Other components
- Learner
- Learner browser
Connections 14
- Learner → Tokenization: runs chapter
README.md - Learner → Embeddings: runs chapter
README.md - Learner → Bigram training: runs chapter
README.md - Learner → Self attention: runs chapter
README.md - Learner → Heads and position: runs chapter
README.md - Learner → Transformer block: runs chapter
README.md - Learner → Tiny GPT: runs chapter
README.md - Learner → GPT training: runs chapter
README.md - Learner browser → Course textbook: opens
README.md - GPT training → Gradient check: checks gradients
code/step8_train_tiny_gpt.py - GPT training → Adam optimizer: updates parameters
code/step8_train_tiny_gpt.py - GPT training → Text generation: generates samples
code/step8_train_tiny_gpt.py - Gradient check → GPT training: validates gradients
code/step8_train_tiny_gpt.py - Adam optimizer → GPT training: adjusts parameters
code/step8_train_tiny_gpt.py