# rasbt/llms-from-scratch architecture

> Architecture diagram of the GitHub repository rasbt/llms-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rasbt/llms-from-scratch
- Repository: https://github.com/rasbt/llms-from-scratch
- Video tour (about a minute): https://gitdiagram.com/rasbt/llms-from-scratch/video

## Overview

This repository is educational code for building, training, and adapting GPT-like language models from scratch. A principal workflow moves from text preparation and model construction to pretraining, then fine-tuning and generation or evaluation; classification is a separate fine-tuning outcome. The code also covers model variants, distributed training, and LoRA. The README establishes these broad stages, while sampled files show particular implementations and interfaces. Many chapter modules were not sampled, so edges between those subsystems are limited to documented relationships.

Main entry points: [`ch04/01_main-chapter-code/gpt.py`](https://github.com/rasbt/llms-from-scratch/blob/main/ch04/01_main-chapter-code/gpt.py) (GPT architecture), [`ch05/01_main-chapter-code/gpt_train.py`](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_train.py) (GPT pretraining), [`ch07/01_main-chapter-code/gpt_instruction_finetuning.py`](https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/gpt_instruction_finetuning.py) (Instruction fine-tuning), [`ch02`](https://github.com/rasbt/llms-from-scratch/tree/main/ch02) (Text preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Text and data"]
  node_textprep["Text preparation"]
  node_instructiondata["Instruction datasets"]
end

subgraph group_models["Model implementations"]
  node_gpt["GPT architecture<br/>[gpt.py]"]
  node_modelpkg["Model package modules"]
  node_qwen["Qwen3 model<br/>[qwen3.py]"]
  node_attentionvariants["Attention variants"]
end

subgraph group_training["Training and adaptation"]
  node_pretrain["GPT pretraining<br/>[gpt_train.py]"]
  node_muontrain["Muon training<br/>[gpt_train_muon.py]"]
  node_hparam["Hyperparameter search<br/>[hparam_search.py]"]
  node_classifier["Classification fine-tuning"]
  node_instructiontune["Instruction fine-tuning"]
  node_lora["LoRA fine-tuning"]
  node_ddp["Distributed training"]
end

subgraph group_inference["Inference and evaluation"]
  node_weightload["Pretrained weight loading<br/>[gpt_download.py]"]
  node_generation["Text generation<br/>[gpt_generate.py]"]
  node_evaluation["Model evaluation<br/>[ollama_evaluate.py]"]
end

subgraph group_interfaces["User interfaces"]
  node_classificationui["Review classifier UI<br/>[app.py]"]
  node_chatui["Instruction chat UI<br/>[app.py]"]
  node_qwenui["Qwen chat interfaces"]
end

node_learner(("Engineer / learner"))

node_learner -->|"builds text inputs"| node_textprep
node_textprep -.->|"prepares model inputs"| node_gpt
node_learner -->|"develops and trains"| node_pretrain
node_gpt -.->|"trains model"| node_pretrain
node_pretrain -.->|"enables generation"| node_generation
node_learner -->|"fine-tunes model"| node_classifier
node_learner -->|"fine-tunes model"| node_instructiontune
node_instructiondata -->|"formats examples"| node_instructiontune
node_weightload -->|"loads weights into"| node_gpt
node_gpt -->|"generates tokens"| node_generation
node_hparam -->|"imports model components"| node_modelpkg
node_muontrain -.->|"trains GPT variant"| node_pretrain
node_ddp -.->|"trains across GPUs"| node_pretrain
node_lora -.->|"adapts model"| node_instructiontune
node_attentionvariants -.->|"extends attention designs"| node_gpt
node_qwen -.->|"powers chat"| node_qwenui
node_classifier -->|"serves predictions to"| node_classificationui
node_instructiontune -->|"produces fine-tuned weights"| node_chatui
node_chatui -->|"uses generation functions"| node_modelpkg
node_learner -->|"evaluates models"| node_evaluation

click node_textprep "https://github.com/rasbt/llms-from-scratch/tree/main/ch02"
click node_instructiondata "https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/exercise_experiments.py"
click node_gpt "https://github.com/rasbt/llms-from-scratch/blob/main/ch04/01_main-chapter-code/gpt.py"
click node_modelpkg "https://github.com/rasbt/llms-from-scratch/tree/main/pkg/llms_from_scratch"
click node_qwen "https://github.com/rasbt/llms-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py"
click node_attentionvariants "https://github.com/rasbt/llms-from-scratch/tree/main/ch04"
click node_pretrain "https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_train.py"
click node_muontrain "https://github.com/rasbt/llms-from-scratch/blob/main/ch05/18_muon/gpt_train_muon.py"
click node_hparam "https://github.com/rasbt/llms-from-scratch/blob/main/ch05/05_bonus_hparam_tuning/hparam_search.py"
click node_classifier "https://github.com/rasbt/llms-from-scratch/blob/main/ch06/01_main-chapter-code/gpt_class_finetune.py"
click node_instructiontune "https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/gpt_instruction_finetuning.py"
click node_lora "https://github.com/rasbt/llms-from-scratch/blob/main/appendix-E/01_main-chapter-code/previous_chapters.py"
click node_ddp "https://github.com/rasbt/llms-from-scratch/blob/main/ch05/10_llm-training-speed/02_opt_multi_gpu_ddp.py"
click node_weightload "https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_download.py"
click node_generation "https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_generate.py"
click node_evaluation "https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/ollama_evaluate.py"
click node_classificationui "https://github.com/rasbt/llms-from-scratch/blob/main/ch06/04_user_interface/app.py"
click node_chatui "https://github.com/rasbt/llms-from-scratch/blob/main/ch07/06_user_interface/app.py"
click node_qwenui "https://github.com/rasbt/llms-from-scratch/tree/main/ch05/11_qwen3/qwen3-chat-interface"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_textprep,node_instructiondata toneBlue
class node_gpt,node_modelpkg,node_qwen,node_attentionvariants toneAmber
class node_pretrain,node_muontrain,node_hparam,node_classifier,node_instructiontune,node_lora,node_ddp toneMint
class node_weightload,node_generation,node_evaluation toneRose
class node_classificationui,node_chatui,node_qwenui,node_learner toneIndigo
```

## Components

### Text and data

- [Text preparation](https://github.com/rasbt/llms-from-scratch/tree/main/ch02): `ch02`
- [Instruction datasets](https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/exercise_experiments.py): `ch07/01_main-chapter-code/exercise_experiments.py`

### Model implementations

- [GPT architecture](https://github.com/rasbt/llms-from-scratch/blob/main/ch04/01_main-chapter-code/gpt.py): `ch04/01_main-chapter-code/gpt.py`
- [Model package modules](https://github.com/rasbt/llms-from-scratch/tree/main/pkg/llms_from_scratch): `pkg/llms_from_scratch`
- [Qwen3 model](https://github.com/rasbt/llms-from-scratch/blob/main/pkg/llms_from_scratch/qwen3.py): `pkg/llms_from_scratch/qwen3.py`
- [Attention variants](https://github.com/rasbt/llms-from-scratch/tree/main/ch04): `ch04`

### Training and adaptation

- [GPT pretraining](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_train.py): `ch05/01_main-chapter-code/gpt_train.py`
- [Muon training](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/18_muon/gpt_train_muon.py): `ch05/18_muon/gpt_train_muon.py`
- [Hyperparameter search](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/05_bonus_hparam_tuning/hparam_search.py): `ch05/05_bonus_hparam_tuning/hparam_search.py`
- [Classification fine-tuning](https://github.com/rasbt/llms-from-scratch/blob/main/ch06/01_main-chapter-code/gpt_class_finetune.py): `ch06/01_main-chapter-code/gpt_class_finetune.py`
- [Instruction fine-tuning](https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/gpt_instruction_finetuning.py): `ch07/01_main-chapter-code/gpt_instruction_finetuning.py`
- [LoRA fine-tuning](https://github.com/rasbt/llms-from-scratch/blob/main/appendix-E/01_main-chapter-code/previous_chapters.py): `appendix-E/01_main-chapter-code/previous_chapters.py`
- [Distributed training](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/10_llm-training-speed/02_opt_multi_gpu_ddp.py): `ch05/10_llm-training-speed/02_opt_multi_gpu_ddp.py`

### Inference and evaluation

- [Pretrained weight loading](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_download.py): `ch05/01_main-chapter-code/gpt_download.py`
- [Text generation](https://github.com/rasbt/llms-from-scratch/blob/main/ch05/01_main-chapter-code/gpt_generate.py): `ch05/01_main-chapter-code/gpt_generate.py`
- [Model evaluation](https://github.com/rasbt/llms-from-scratch/blob/main/ch07/01_main-chapter-code/ollama_evaluate.py): `ch07/01_main-chapter-code/ollama_evaluate.py`

### User interfaces

- [Review classifier UI](https://github.com/rasbt/llms-from-scratch/blob/main/ch06/04_user_interface/app.py): `ch06/04_user_interface/app.py`
- [Instruction chat UI](https://github.com/rasbt/llms-from-scratch/blob/main/ch07/06_user_interface/app.py): `ch07/06_user_interface/app.py`
- [Qwen chat interfaces](https://github.com/rasbt/llms-from-scratch/tree/main/ch05/11_qwen3/qwen3-chat-interface): `ch05/11_qwen3/qwen3-chat-interface`

### Other components

- **Engineer / learner**

## Connections

- Engineer / learner → Text preparation: builds text inputs (evidence: `README.md`)
- Text preparation → GPT architecture: prepares model inputs
- Engineer / learner → GPT pretraining: develops and trains (evidence: `README.md`)
- GPT architecture → GPT pretraining: trains model
- GPT pretraining → Text generation: enables generation
- Engineer / learner → Classification fine-tuning: fine-tunes model (evidence: `README.md`)
- Engineer / learner → Instruction fine-tuning: fine-tunes model (evidence: `README.md`)
- Instruction datasets → Instruction fine-tuning: formats examples (evidence: `ch07/01_main-chapter-code/exercise_experiments.py`)
- Pretrained weight loading → GPT architecture: loads weights into
- GPT architecture → Text generation: generates tokens (evidence: `ch05/01_main-chapter-code/gpt_generate.py`)
- Hyperparameter search → Model package modules: imports model components (evidence: `ch05/05_bonus_hparam_tuning/hparam_search.py`)
- Muon training → GPT pretraining: trains GPT variant
- Distributed training → GPT pretraining: trains across GPUs
- LoRA fine-tuning → Instruction fine-tuning: adapts model
- Attention variants → GPT architecture: extends attention designs
- Qwen3 model → Qwen chat interfaces: powers chat
- Classification fine-tuning → Review classifier UI: serves predictions to (evidence: `ch06/04_user_interface/app.py`)
- Instruction fine-tuning → Instruction chat UI: produces fine-tuned weights (evidence: `ch07/06_user_interface/app.py`)
- Instruction chat UI → Model package modules: uses generation functions (evidence: `ch07/06_user_interface/app.py`)
- Engineer / learner → Model evaluation: evaluates models (evidence: `README.md`)
