How 4nath0s/gpt2-attention-variants works
This repository trains GPT-2-small variants with compact attention caches, then compares their validation perplexity, generation speed, and cache size. The workflow is TinyStories preprocessing → token-block datasets → model training and checkpointing → cached generation and evaluation. Attention variants support causal or sliding-window masks; MHA and GQA also support cross-layer K/V sharing. Edges reflect the shown calls and README workflow; generated artifacts are represented by the source files that write or read them.
Main entry points: generation.py Comparison run · training.py Training loop · model.py GPT model · preprocess.py Preprocessing
Components 16
Data preparation
- Preprocessing
preprocess.py - Token bins
preprocess.py - Token datasets
dataset.py
Model and attention
- Model config
config.py - GPT model
model.py - MHA attention
mha.py - GQA / MQA
gqa.py - MLA attention
mla.py - Attention mask
mask.py
Training
- Training loop
training.py - Model checkpoint
training.py
Generation and evaluation
- Comparison run
generation.py - Metrics and plots
generation.py
Other components
- User
- TinyStories
- GPT-2 tokenizer
Connections 23
- User → Preprocessing: runs
README.md - TinyStories → Preprocessing: streams data
preprocess.py - GPT-2 tokenizer → Preprocessing: encodes text
preprocess.py - Preprocessing → Token bins: writes tokens
preprocess.py - User → Training loop: runs
README.md - Token bins → Token datasets: read by
dataset.py - Training loop → Token datasets: loads batches
training.py - Training loop → Model config: creates
training.py - Training loop → GPT model: trains
training.py - GPT model → MHA attention: selects
model.py - GPT model → GQA / MQA: selects
model.py - GPT model → MLA attention: selects
model.py - MHA attention → Attention mask: builds mask
mha.py - GQA / MQA → Attention mask: builds mask
gqa.py - MLA attention → Attention mask: builds mask
mla.py - Training loop → Model checkpoint: saves
training.py - User → Comparison run: runs
README.md - Comparison run → Model checkpoint: loads
generation.py - Comparison run → GPT model: evaluates
generation.py - Comparison run → GPT-2 tokenizer: tokenizes prompt
generation.py - Comparison run → Training loop: uses token helpers
generation.py - Comparison run → Token bins: reads validation tokens
generation.py - Comparison run → Metrics and plots: reports results
generation.py