# 4nath0s/gpt2-attention-variants architecture

> Architecture diagram of the GitHub repository 4nath0s/gpt2-attention-variants, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/4nath0s/gpt2-attention-variants
- Repository: https://github.com/4nath0s/gpt2-attention-variants

## Overview

This repository trains GPT-2-small variants with compact attention caches, then compares their validation perplexity, generation speed, and cache size. The workflow is TinyStories preprocessing → token-block datasets → model training and checkpointing → cached generation and evaluation. Attention variants support causal or sliding-window masks; MHA and GQA also support cross-layer K/V sharing. Edges reflect the shown calls and README workflow; generated artifacts are represented by the source files that write or read them.

Main entry points: [`generation.py`](https://github.com/4nath0s/gpt2-attention-variants/blob/main/generation.py) (Comparison run), [`training.py`](https://github.com/4nath0s/gpt2-attention-variants/blob/main/training.py) (Training loop), [`model.py`](https://github.com/4nath0s/gpt2-attention-variants/blob/main/model.py) (GPT model), [`preprocess.py`](https://github.com/4nath0s/gpt2-attention-variants/blob/main/preprocess.py) (Preprocessing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Data preparation"]
  node_preprocess["Preprocessing<br/>[preprocess.py]"]
  node_token_bins[("Token bins<br/>[preprocess.py]")]
  node_dataset["Token datasets<br/>[dataset.py]"]
end

subgraph group_model["Model and attention"]
  node_config["Model config<br/>[config.py]"]
  node_gpt["GPT model<br/>[model.py]"]
  node_mha["MHA attention<br/>[mha.py]"]
  node_gqa["GQA / MQA<br/>[gqa.py]"]
  node_mla["MLA attention<br/>[mla.py]"]
  node_mask["Attention mask<br/>[mask.py]"]
end

subgraph group_train["Training"]
  node_training["Training loop<br/>[training.py]"]
  node_checkpoint[("Model checkpoint<br/>[training.py]")]
end

subgraph group_eval["Generation and evaluation"]
  node_comparison["Comparison run<br/>[generation.py]"]
  node_metrics["Metrics and plots<br/>[generation.py]"]
end

node_user(("User"))
node_stories[("TinyStories")]
node_tokenizer["GPT-2 tokenizer"]

node_user -->|"runs"| node_preprocess
node_stories -->|"streams data"| node_preprocess
node_tokenizer -->|"encodes text"| node_preprocess
node_preprocess -->|"writes tokens"| node_token_bins
node_user -->|"runs"| node_training
node_token_bins -->|"read by"| node_dataset
node_training -->|"loads batches"| node_dataset
node_training -->|"creates"| node_config
node_training -->|"trains"| node_gpt
node_gpt -->|"selects"| node_mha
node_gpt -->|"selects"| node_gqa
node_gpt -->|"selects"| node_mla
node_mha -->|"builds mask"| node_mask
node_gqa -->|"builds mask"| node_mask
node_mla -->|"builds mask"| node_mask
node_training -->|"saves"| node_checkpoint
node_user -->|"runs"| node_comparison
node_comparison -->|"loads"| node_checkpoint
node_comparison -->|"evaluates"| node_gpt
node_comparison -->|"tokenizes prompt"| node_tokenizer
node_comparison -->|"uses token helpers"| node_training
node_comparison -->|"reads validation tokens"| node_token_bins
node_comparison -->|"reports results"| node_metrics

click node_preprocess "https://github.com/4nath0s/gpt2-attention-variants/blob/main/preprocess.py"
click node_token_bins "https://github.com/4nath0s/gpt2-attention-variants/blob/main/preprocess.py"
click node_dataset "https://github.com/4nath0s/gpt2-attention-variants/blob/main/dataset.py"
click node_config "https://github.com/4nath0s/gpt2-attention-variants/blob/main/config.py"
click node_gpt "https://github.com/4nath0s/gpt2-attention-variants/blob/main/model.py"
click node_mha "https://github.com/4nath0s/gpt2-attention-variants/blob/main/mha.py"
click node_gqa "https://github.com/4nath0s/gpt2-attention-variants/blob/main/gqa.py"
click node_mla "https://github.com/4nath0s/gpt2-attention-variants/blob/main/mla.py"
click node_mask "https://github.com/4nath0s/gpt2-attention-variants/blob/main/mask.py"
click node_training "https://github.com/4nath0s/gpt2-attention-variants/blob/main/training.py"
click node_checkpoint "https://github.com/4nath0s/gpt2-attention-variants/blob/main/training.py"
click node_comparison "https://github.com/4nath0s/gpt2-attention-variants/blob/main/generation.py"
click node_metrics "https://github.com/4nath0s/gpt2-attention-variants/blob/main/generation.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_preprocess,node_token_bins,node_dataset,node_user toneBlue
class node_config,node_gpt,node_mha,node_gqa,node_mla,node_mask,node_stories toneAmber
class node_training,node_checkpoint toneMint
class node_comparison,node_metrics toneRose
class node_tokenizer toneIndigo
```

## Components

### Data preparation

- [Preprocessing](https://github.com/4nath0s/gpt2-attention-variants/blob/main/preprocess.py): `preprocess.py`
- [Token bins](https://github.com/4nath0s/gpt2-attention-variants/blob/main/preprocess.py): `preprocess.py`
- [Token datasets](https://github.com/4nath0s/gpt2-attention-variants/blob/main/dataset.py): `dataset.py`

### Model and attention

- [Model config](https://github.com/4nath0s/gpt2-attention-variants/blob/main/config.py): `config.py`
- [GPT model](https://github.com/4nath0s/gpt2-attention-variants/blob/main/model.py): `model.py`
- [MHA attention](https://github.com/4nath0s/gpt2-attention-variants/blob/main/mha.py): `mha.py`
- [GQA / MQA](https://github.com/4nath0s/gpt2-attention-variants/blob/main/gqa.py): `gqa.py`
- [MLA attention](https://github.com/4nath0s/gpt2-attention-variants/blob/main/mla.py): `mla.py`
- [Attention mask](https://github.com/4nath0s/gpt2-attention-variants/blob/main/mask.py): `mask.py`

### Training

- [Training loop](https://github.com/4nath0s/gpt2-attention-variants/blob/main/training.py): `training.py`
- [Model checkpoint](https://github.com/4nath0s/gpt2-attention-variants/blob/main/training.py): `training.py`

### Generation and evaluation

- [Comparison run](https://github.com/4nath0s/gpt2-attention-variants/blob/main/generation.py): `generation.py`
- [Metrics and plots](https://github.com/4nath0s/gpt2-attention-variants/blob/main/generation.py): `generation.py`

### Other components

- **User**
- **TinyStories**
- **GPT-2 tokenizer**

## Connections

- User → Preprocessing: runs (evidence: `README.md`)
- TinyStories → Preprocessing: streams data (evidence: `preprocess.py`)
- GPT-2 tokenizer → Preprocessing: encodes text (evidence: `preprocess.py`)
- Preprocessing → Token bins: writes tokens (evidence: `preprocess.py`)
- User → Training loop: runs (evidence: `README.md`)
- Token bins → Token datasets: read by (evidence: `dataset.py`)
- Training loop → Token datasets: loads batches (evidence: `training.py`)
- Training loop → Model config: creates (evidence: `training.py`)
- Training loop → GPT model: trains (evidence: `training.py`)
- GPT model → MHA attention: selects (evidence: `model.py`)
- GPT model → GQA / MQA: selects (evidence: `model.py`)
- GPT model → MLA attention: selects (evidence: `model.py`)
- MHA attention → Attention mask: builds mask (evidence: `mha.py`)
- GQA / MQA → Attention mask: builds mask (evidence: `gqa.py`)
- MLA attention → Attention mask: builds mask (evidence: `mla.py`)
- Training loop → Model checkpoint: saves (evidence: `training.py`)
- User → Comparison run: runs (evidence: `README.md`)
- Comparison run → Model checkpoint: loads (evidence: `generation.py`)
- Comparison run → GPT model: evaluates (evidence: `generation.py`)
- Comparison run → GPT-2 tokenizer: tokenizes prompt (evidence: `generation.py`)
- Comparison run → Training loop: uses token helpers (evidence: `generation.py`)
- Comparison run → Token bins: reads validation tokens (evidence: `generation.py`)
- Comparison run → Metrics and plots: reports results (evidence: `generation.py`)
