How shreyasgaikwad-04/minigpt works
MiniGPT is an educational decoder-only language model, trained on a small TinyStories subset and used to generate text from saved checkpoints. The training workflow fetches examples, fits a tokenizer, builds batches, optimizes the model with next-token loss, and saves checkpoints; generation reloads a checkpoint and tokenizer, then autoregressively samples tokens for a prompt. The model’s documented internal stages are represented within its implementation module. Training and generation source were sampled; the README documents the broader workflow, while details beyond those excerpts are not inferred.
Main entry points: train.py Training loop · generate.py Generation loop · model.py MiniGPT · data.py Fetch and batch data
Components 18
Data preparation
- Fetch and batch data
src/data.py - Tokenizer
src/tokenizer.py - Saved vocabulary
src/train.py
Transformer model
- MiniGPT
src/model.py - Token and position embeddings
src/model.py - Transformer blocks
src/model.py - Causal attention
src/model.py - Feed-forward layers
src/model.py - Token logits
src/model.py
Training
- Training config
configs/tiny.yaml - Training loop
src/train.py - Next-token loss
src/train.py - Model checkpoint
src/train.py - Training metrics
src/utils.py
Generation
- Generation loop
src/generate.py - Generated text
src/generate.py
Other components
- User
- TinyStories
Connections 25
- User → Training loop: starts training
- User → Generation loop: provides prompt
- Training loop → Training config: reads config
- Training loop → Fetch and batch data: requests examples
- Fetch and batch data → TinyStories: loads subset
- Training loop → Tokenizer: fits vocabulary
- Training loop → Saved vocabulary: saves vocabulary
- Training loop → Fetch and batch data: builds batches
- Training loop → MiniGPT: trains model
- MiniGPT → Token and position embeddings: embeds IDs
- MiniGPT → Transformer blocks: applies blocks
- Transformer blocks → Causal attention: computes attention
- Transformer blocks → Feed-forward layers: applies feed-forward
- MiniGPT → Token logits: produces logits
- Training loop → Next-token loss: computes loss
- Next-token loss → Token logits: scores predictions
- Training loop → Model checkpoint: saves checkpoint
- Training loop → Training metrics: saves metrics
- Generation loop → Model checkpoint: loads weights
- Generation loop → Saved vocabulary: loads tokenizer
- Generation loop → Tokenizer: encodes prompt
- Generation loop → MiniGPT: queries next token
- Generation loop → Tokenizer: decodes tokens
- Generation loop → Generated text: returns text
- Generated text → User: displays result