# datamonk15/llms-from-scratch architecture

> Architecture diagram of the GitHub repository datamonk15/llms-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datamonk15/llms-from-scratch
- Repository: https://github.com/datamonk15/llms-from-scratch

## Overview

This is an educational codebase for building, pretraining, and fine-tuning GPT-like language models from scratch, with examples for loading larger pretrained weights. Its main learning path runs from text/token processing and attention through GPT construction, pretraining, and task-specific fine-tuning; instruction tuning and interactive demos extend that path. The repository also contains alternative architectures and training experiments. Links between chapter stages are documented at a high level in the README; unsampled implementations are represented by their exact tree paths, without inferring internal calls.

Main entry points: [`pkg/llms_from_scratch/ch04.py`](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch04.py) (GPT model), [`pkg/llms_from_scratch/ch05.py`](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch05.py) (Pretraining), [`ch06/01_main-chapter-code/gpt_class_finetune.py`](https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/01_main-chapter-code/gpt_class_finetune.py) (Classification tuning), [`ch07/06_user_interface/app.py`](https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/06_user_interface/app.py) (Instruction chat UI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_modeling["Text and models"]
  node_text["Text processing<br/>[ch02.py]"]
  node_attention["Attention layers<br/>[ch03.py]"]
  node_gpt["GPT model<br/>[ch04.py]"]
  node_tokenizer["GPT-2 BPE<br/>[bpe_openai_gpt2.py]"]
  node_weights["Pretrained weights<br/>[gpt_download.py]"]
end

subgraph group_training["Training and tuning"]
  node_pretraining["Pretraining<br/>[ch05.py]"]
  node_gutenberg["Gutenberg training"]
  node_hparam["Hyperparameter search<br/>[hparam_search.py]"]
  node_muon["Muon training<br/>[gpt_train_muon.py]"]
  node_classification["Classification tuning"]
  node_experiments["Fine-tuning experiments"]
  node_instruction["Instruction tuning"]
  node_dpo["Preference tuning"]
end

subgraph group_serving["Applications and variants"]
  node_gpt_demo["Instruction chat UI<br/>[app.py]"]
  node_class_demo["Classification UI<br/>[app.py]"]
  node_qwen["Qwen models<br/>[qwen3.py]"]
  node_kv_cache["Cached generation<br/>[generate.py]"]
  node_llama["Llama models<br/>[llama3.py]"]
end

node_learner(("Learner"))

node_learner -->|"learns tokenization"| node_text
node_text -->|"builds toward"| node_attention
node_attention -->|"builds toward"| node_gpt
node_gpt -->|"pretrains model"| node_pretraining
node_weights -.->|"supplies weights"| node_classification
node_pretraining -->|"fine-tunes model"| node_classification
node_pretraining -->|"specializes training"| node_gutenberg
node_pretraining -.->|"tunes training"| node_hparam
node_pretraining -.->|"uses variant optimizer"| node_muon
node_classification -.->|"extends experiments"| node_experiments
node_classification -->|"fine-tunes further"| node_instruction
node_instruction -.->|"extends tuning"| node_dpo
node_learner -->|"submits instruction"| node_gpt_demo
node_gpt_demo -->|"instantiates model"| node_gpt
node_gpt_demo -->|"generates response"| node_pretraining
node_learner -->|"submits review"| node_class_demo
node_class_demo -->|"instantiates model"| node_gpt
node_class_demo -->|"classifies review"| node_classification
node_gpt -.->|"model alternative"| node_qwen
node_gpt -.->|"model alternative"| node_llama
node_gpt -.->|"generation alternative"| node_kv_cache
node_tokenizer -.->|"encodes text"| node_text

click node_text "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch02.py"
click node_attention "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch03.py"
click node_gpt "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch04.py"
click node_tokenizer "https://github.com/datamonk15/llms-from-scratch/blob/main/ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py"
click node_weights "https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_download.py"
click node_pretraining "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch05.py"
click node_gutenberg "https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py"
click node_hparam "https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/05_bonus_hparam_tuning/hparam_search.py"
click node_muon "https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/18_muon/gpt_train_muon.py"
click node_classification "https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/01_main-chapter-code/gpt_class_finetune.py"
click node_experiments "https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/02_bonus_additional-experiments/additional_experiments.py"
click node_instruction "https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/01_main-chapter-code/gpt_instruction_finetuning.py"
click node_dpo "https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/04_preference-tuning-with-dpo/previous_chapters.py"
click node_gpt_demo "https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/06_user_interface/app.py"
click node_class_demo "https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/04_user_interface/app.py"
click node_qwen "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py"
click node_kv_cache "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/kv_cache/generate.py"
click node_llama "https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/llama3.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_text,node_attention,node_gpt,node_tokenizer,node_weights toneBlue
class node_pretraining,node_gutenberg,node_hparam,node_muon,node_classification,node_experiments,node_instruction,node_dpo toneAmber
class node_gpt_demo,node_class_demo,node_qwen,node_kv_cache,node_llama toneMint
class node_learner toneIndigo
```

## Components

### Text and models

- [Text processing](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch02.py): `pkg/llms_from_scratch/ch02.py`
- [Attention layers](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch03.py): `pkg/llms_from_scratch/ch03.py`
- [GPT model](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch04.py): `pkg/llms_from_scratch/ch04.py`
- [GPT-2 BPE](https://github.com/datamonk15/llms-from-scratch/blob/main/ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py): `ch02/02_bonus_bytepair-encoder/bpe_openai_gpt2.py`
- [Pretrained weights](https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_download.py): `ch05/01_main-chapter-code/gpt_download.py`

### Training and tuning

- [Pretraining](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/ch05.py): `pkg/llms_from_scratch/ch05.py`
- [Gutenberg training](https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py): `ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py`
- [Hyperparameter search](https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/05_bonus_hparam_tuning/hparam_search.py): `ch05/05_bonus_hparam_tuning/hparam_search.py`
- [Muon training](https://github.com/datamonk15/llms-from-scratch/blob/main/ch05/18_muon/gpt_train_muon.py): `ch05/18_muon/gpt_train_muon.py`
- [Classification tuning](https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/01_main-chapter-code/gpt_class_finetune.py): `ch06/01_main-chapter-code/gpt_class_finetune.py`
- [Fine-tuning experiments](https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/02_bonus_additional-experiments/additional_experiments.py): `ch06/02_bonus_additional-experiments/additional_experiments.py`
- [Instruction tuning](https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/01_main-chapter-code/gpt_instruction_finetuning.py): `ch07/01_main-chapter-code/gpt_instruction_finetuning.py`
- [Preference tuning](https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/04_preference-tuning-with-dpo/previous_chapters.py): `ch07/04_preference-tuning-with-dpo/previous_chapters.py`

### Applications and variants

- [Instruction chat UI](https://github.com/datamonk15/llms-from-scratch/blob/main/ch07/06_user_interface/app.py): `ch07/06_user_interface/app.py`
- [Classification UI](https://github.com/datamonk15/llms-from-scratch/blob/main/ch06/04_user_interface/app.py): `ch06/04_user_interface/app.py`
- [Qwen models](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py): `pkg/llms_from_scratch/qwen3.py`
- [Cached generation](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/kv_cache/generate.py): `pkg/llms_from_scratch/kv_cache/generate.py`
- [Llama models](https://github.com/datamonk15/llms-from-scratch/blob/main/pkg/llms_from_scratch/llama3.py): `pkg/llms_from_scratch/llama3.py`

### Other components

- **Learner**

## Connections

- Learner → Text processing: learns tokenization (evidence: `README.md`)
- Text processing → Attention layers: builds toward (evidence: `README.md`)
- Attention layers → GPT model: builds toward (evidence: `README.md`)
- GPT model → Pretraining: pretrains model (evidence: `README.md`)
- Pretrained weights → Classification tuning: supplies weights (evidence: `ch06/01_main-chapter-code/gpt_class_finetune.py`)
- Pretraining → Classification tuning: fine-tunes model (evidence: `README.md`)
- Pretraining → Gutenberg training: specializes training (evidence: `ch05/03_bonus_pretraining_on_gutenberg/pretraining_simple.py`)
- Pretraining → Hyperparameter search: tunes training
- Pretraining → Muon training: uses variant optimizer
- Classification tuning → Fine-tuning experiments: extends experiments
- Classification tuning → Instruction tuning: fine-tunes further (evidence: `README.md`)
- Instruction tuning → Preference tuning: extends tuning
- Learner → Instruction chat UI: submits instruction (evidence: `ch07/06_user_interface/app.py`)
- Instruction chat UI → GPT model: instantiates model (evidence: `ch07/06_user_interface/app.py`)
- Instruction chat UI → Pretraining: generates response (evidence: `ch07/06_user_interface/app.py`)
- Learner → Classification UI: submits review (evidence: `ch06/04_user_interface/app.py`)
- Classification UI → GPT model: instantiates model (evidence: `ch06/04_user_interface/app.py`)
- Classification UI → Classification tuning: classifies review (evidence: `ch06/04_user_interface/app.py`)
- GPT model → Qwen models: model alternative
- GPT model → Llama models: model alternative
- GPT model → Cached generation: generation alternative
- GPT-2 BPE → Text processing: encodes text
