# raiyanyahya/how-to-train-your-gpt architecture

> Architecture diagram of the GitHub repository raiyanyahya/how-to-train-your-gpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raiyanyahya/how-to-train-your-gpt
- Repository: https://github.com/raiyanyahya/how-to-train-your-gpt

## Overview

How to Train Your GPT is an interactive textbook with a runnable, single-file GPT training and inference example. The principal workflow is to follow the chapters from tokenization through model construction, training, and inference, then explore the concepts in companion notebooks or run `main.py`. The graph includes documented standalone explainers and sampled runtime components. Runtime wiring beyond the shown excerpts is omitted where it cannot be confirmed.

Main entry points: [`main.py`](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py) (Training Runner), [`chapters/00_overview.md`](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/00_overview.md) (GPT Overview), [`chapters/02_tokenization.md`](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/02_tokenization.md) (Tokenization), [`chapters/03_embeddings.md`](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/03_embeddings.md) (Embeddings)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_curriculum["Core Curriculum"]
  node_overview["GPT Overview<br/>[00_overview.md]"]
  node_tokenization["Tokenization<br/>[02_tokenization.md]"]
  node_embeddings["Embeddings<br/>[03_embeddings.md]"]
  node_position["Position Encoding"]
  node_attention["Attention<br/>[05_attention.md]"]
  node_block["Transformer Block"]
  node_model["GPT Model<br/>[07_gpt_model.md]"]
  node_training["Training Pipeline<br/>[08_training.md]"]
  node_inference["Inference<br/>[09_inference.md]"]
  node_full_script["Full Script Guide<br/>[10_full_script.md]"]
end

subgraph group_explainers["Topic Explainers"]
  node_topic_guides["Topic Guides"]
end

subgraph group_runtime["Runnable GPT"]
  node_tokenizer["BPE Tokenizer<br/>[main.py]"]
  node_dataset["Text Dataset<br/>[main.py]"]
  node_gpt["GPT Model Code<br/>[main.py]"]
  node_transformer["Transformer Components<br/>[main.py]"]
  node_optimizer["AdamW Optimizer<br/>[main.py]"]
  node_scheduler["Cosine Warmup<br/>[main.py]"]
  node_runner["Training Runner<br/>[main.py]"]
  node_generator["Token Generation<br/>[main.py]"]
end

subgraph group_notebooks["Interactive Notebooks"]
  node_notebooks["Chapter Notebooks"]
end

node_reader(("Learner"))

node_reader -->|"starts with"| node_overview
node_overview -->|"continues to"| node_tokenization
node_tokenization -->|"continues to"| node_embeddings
node_embeddings -->|"continues to"| node_position
node_position -->|"continues to"| node_attention
node_attention -->|"continues to"| node_block
node_block -->|"continues to"| node_model
node_model -->|"continues to"| node_training
node_training -->|"continues to"| node_inference
node_inference -->|"continues to"| node_full_script
node_reader -->|"explores"| node_topic_guides
node_reader -->|"runs examples"| node_notebooks
node_tokenizer -->|"encodes text"| node_dataset
node_dataset -->|"provides batches"| node_runner
node_runner -->|"trains"| node_gpt
node_runner -->|"updates with"| node_optimizer
node_runner -->|"sets learning rate"| node_scheduler
node_gpt -->|"calls layers"| node_transformer
node_gpt -->|"generates tokens"| node_generator

click node_overview "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/00_overview.md"
click node_tokenization "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/02_tokenization.md"
click node_embeddings "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/03_embeddings.md"
click node_position "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/04_positional_encoding.md"
click node_attention "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/05_attention.md"
click node_block "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/06_transformer_block.md"
click node_model "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/07_gpt_model.md"
click node_training "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/08_training.md"
click node_inference "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/09_inference.md"
click node_full_script "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/10_full_script.md"
click node_topic_guides "https://github.com/raiyanyahya/how-to-train-your-gpt/tree/master/explanations%20and%20examples%20WIP"
click node_tokenizer "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_dataset "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_gpt "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_transformer "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_optimizer "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_scheduler "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_runner "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_generator "https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py"
click node_notebooks "https://github.com/raiyanyahya/how-to-train-your-gpt/tree/master/notebooks"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_overview,node_tokenization,node_embeddings,node_position,node_attention,node_block,node_model,node_training,node_inference,node_full_script toneBlue
class node_topic_guides toneAmber
class node_tokenizer,node_dataset,node_gpt,node_transformer,node_optimizer,node_scheduler,node_runner,node_generator toneMint
class node_notebooks toneRose
class node_reader toneIndigo
```

## Components

### Core Curriculum

- [GPT Overview](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/00_overview.md): `chapters/00_overview.md`
- [Tokenization](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/02_tokenization.md): `chapters/02_tokenization.md`
- [Embeddings](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/03_embeddings.md): `chapters/03_embeddings.md`
- [Position Encoding](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/04_positional_encoding.md): `chapters/04_positional_encoding.md`
- [Attention](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/05_attention.md): `chapters/05_attention.md`
- [Transformer Block](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/06_transformer_block.md): `chapters/06_transformer_block.md`
- [GPT Model](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/07_gpt_model.md): `chapters/07_gpt_model.md`
- [Training Pipeline](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/08_training.md): `chapters/08_training.md`
- [Inference](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/09_inference.md): `chapters/09_inference.md`
- [Full Script Guide](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/chapters/10_full_script.md): `chapters/10_full_script.md`

### Topic Explainers

- [Topic Guides](https://github.com/raiyanyahya/how-to-train-your-gpt/tree/master/explanations%20and%20examples%20WIP): `explanations and examples WIP`

### Runnable GPT

- [BPE Tokenizer](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [Text Dataset](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [GPT Model Code](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [Transformer Components](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [AdamW Optimizer](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [Cosine Warmup](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [Training Runner](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`
- [Token Generation](https://github.com/raiyanyahya/how-to-train-your-gpt/blob/master/main.py): `main.py`

### Interactive Notebooks

- [Chapter Notebooks](https://github.com/raiyanyahya/how-to-train-your-gpt/tree/master/notebooks): `notebooks`

### Other components

- **Learner**

## Connections

- Learner → GPT Overview: starts with
- GPT Overview → Tokenization: continues to
- Tokenization → Embeddings: continues to
- Embeddings → Position Encoding: continues to
- Position Encoding → Attention: continues to
- Attention → Transformer Block: continues to
- Transformer Block → GPT Model: continues to
- GPT Model → Training Pipeline: continues to
- Training Pipeline → Inference: continues to
- Inference → Full Script Guide: continues to
- Learner → Topic Guides: explores
- Learner → Chapter Notebooks: runs examples
- BPE Tokenizer → Text Dataset: encodes text
- Text Dataset → Training Runner: provides batches
- Training Runner → GPT Model Code: trains
- Training Runner → AdamW Optimizer: updates with
- Training Runner → Cosine Warmup: sets learning rate
- GPT Model Code → Transformer Components: calls layers
- GPT Model Code → Token Generation: generates tokens