How bartdesmet/tinyllm works
TinyLLM is an educational, scalar-autograd GPT language-model demo. Its main workflow reads a corpus, builds a tokenizer and decoder-only Transformer, trains next-token predictions (or loads an inference checkpoint), then autoregressively continues a user prompt. The graph preserves both tokenizer choices and the model’s principal layers. Relationships among the model components are supported by the README and source-index references; unsampled implementations are represented only at their documented subsystem level.
Main entry points: GptModel.cs GPT model · Program.cs Demo entry point · TransformerBlock.cs Transformer blocks · Trainer.cs Training loop
Components 21
Application flow
- Demo entry point
TinyLLM/Program.cs - Training corpus
TinyLLM/Data/corpus.txt - Inference checkpoint
TinyLLM/Model/ModelCheckpoint.cs
Tokenization
- Tokenizer contract
TinyLLM/Tokenization/ITokenizer.cs - Character tokenizer
TinyLLM/Tokenization/CharTokenizer.cs - Byte-level BPE
TinyLLM/Tokenization/BytePairTokenizer.cs - Token debug view
TinyLLM/Tokenization/TokenizerDebugFormatter.cs
GPT model
- GPT model
TinyLLM/Model/GptModel.cs - Token and position embeddings
TinyLLM/Model/Embedding.cs - Transformer blocks
TinyLLM/Model/TransformerBlock.cs - Causal attention
TinyLLM/Model/CausalSelfAttention.cs - Feed-forward layer
TinyLLM/Model/FeedForward.cs - Layer normalization
TinyLLM/Model/LayerNorm.cs - Linear projection
TinyLLM/Model/Linear.cs - Parameter enumeration
TinyLLM/Model/Module.cs - Scalar autograd
TinyLLM/Autograd/Value.cs
Training
- Training loop
TinyLLM/Training/Trainer.cs - Cross-entropy loss
TinyLLM/Training/CrossEntropyLoss.cs - AdamW optimizer
TinyLLM/Training/AdamW.cs
Inference
- Autoregressive sampler
TinyLLM/Sampling/Sampler.cs
Other components
- User
Connections 33
- User → Demo entry point: runs and prompts
TinyLLM/Program.cs - Demo entry point → Training corpus: loads corpus
TinyLLM/Program.cs - Demo entry point → Character tokenizer: builds or selects
TinyLLM/Program.cs - Demo entry point → Byte-level BPE: trains or selects
TinyLLM/Program.cs - Character tokenizer → Tokenizer contract: implements
TinyLLM/Tokenization/CharTokenizer.cs - Byte-level BPE → Tokenizer contract: implements
TinyLLM/Tokenization/BytePairTokenizer.cs - Character tokenizer → Token debug view: formats mappings with
TinyLLM/Tokenization/CharTokenizer.cs - Byte-level BPE → Token debug view: formats mappings with
TinyLLM/Tokenization/BytePairTokenizer.cs - Demo entry point → GPT model: constructs
TinyLLM/Program.cs - GPT model → Token and position embeddings: looks up embeddings
TinyLLM/Model/GptModel.cs - GPT model → Transformer blocks: forwards through
TinyLLM/Model/GptModel.cs - Transformer blocks → Causal attention: uses
TinyLLM/Model/TransformerBlock.cs - Transformer blocks → Feed-forward layer: uses
TinyLLM/Model/TransformerBlock.cs - Transformer blocks → Layer normalization: normalizes with
TinyLLM/Model/TransformerBlock.cs - GPT model → Layer normalization: applies final normalization
TinyLLM/Model/GptModel.cs - GPT model → Linear projection: projects logits with
TinyLLM/Model/GptModel.cs - Causal attention → Linear projection: projects with
TinyLLM/Model/CausalSelfAttention.cs - Feed-forward layer → Linear projection: projects with
TinyLLM/Model/FeedForward.cs - GPT model → Parameter enumeration: enumerates parameters through
TinyLLM/Model/GptModel.cs - Parameter enumeration → Scalar autograd: enumerates scalar parameters
TinyLLM/Model/Module.cs - Demo entry point → Training loop: starts training
TinyLLM/Program.cs - Training loop → GPT model: forwards token windows through
TinyLLM/Training/Trainer.cs - Training loop → Cross-entropy loss: computes
TinyLLM/Training/Trainer.cs - Training loop → AdamW optimizer: updates parameters with
TinyLLM/Training/Trainer.cs - Cross-entropy loss → Scalar autograd: builds differentiable loss
TinyLLM/Training/CrossEntropyLoss.cs - AdamW optimizer → Scalar autograd: updates scalar values
TinyLLM/Training/AdamW.cs - Demo entry point → Inference checkpoint: loads or saves
TinyLLM/Program.cs - Inference checkpoint → GPT model: rebuilds model
TinyLLM/Model/ModelCheckpoint.cs - Inference checkpoint → Tokenizer contract: restores tokenizer
TinyLLM/Model/ModelCheckpoint.cs - Demo entry point → Autoregressive sampler: requests generation
TinyLLM/Program.cs - Autoregressive sampler → GPT model: runs context through
TinyLLM/Sampling/Sampler.cs - Autoregressive sampler → Tokenizer contract: encodes prompt and decodes output
TinyLLM/Sampling/Sampler.cs - Autoregressive sampler → User: streams generated text
TinyLLM/Sampling/Sampler.cs