# salmanmik/tiny-llm-from-scratch architecture

> Architecture diagram of the GitHub repository salmanmik/tiny-llm-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salmanmik/tiny-llm-from-scratch
- Repository: https://github.com/salmanmik/tiny-llm-from-scratch

## Overview

This repository is a hands-on NumPy course that builds a tiny language model, progressing from text tokenization through model training and generated text. The README describes the chapters and textbook as product capabilities; sampled code confirms the standalone demos and the final training flow. The graph treats chapter files as learning stages, not as imported modules: the README says to run each file, while sampled files do not import one another. Step 8’s excerpt confirms its internal model, gradient-check, optimizer, and generation responsibilities; the textbook’s contents were not sampled.

Main entry points: [`code/step8_train_tiny_gpt.py`](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py) (GPT training), [`code/step1_tokenizer.py`](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step1_tokenizer.py) (Tokenization), [`code/step2_embeddings.py`](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step2_embeddings.py) (Embeddings), [`code/step3_bigram_train.py`](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step3_bigram_train.py) (Bigram training)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_course["Model-building course"]
  node_tokenize["Tokenization<br/>[step1_tokenizer.py]"]
  node_embeddings["Embeddings"]
  node_bigram["Bigram training"]
  node_attention["Self attention<br/>[step4_attention.py]"]
  node_multihead["Heads and position"]
  node_block["Transformer block"]
  node_gpt["Tiny GPT<br/>[step7_tiny_gpt.py]"]
  node_training["GPT training"]
  node_gradient["Gradient check"]
  node_adam["Adam optimizer"]
  node_generation["Text generation"]
end

subgraph group_learning["Interactive textbook"]
  node_textbook["Course textbook<br/>[index.html]"]
end

node_learner(("Learner"))
node_browser(("Learner browser"))

node_learner -->|"runs chapter"| node_tokenize
node_learner -->|"runs chapter"| node_embeddings
node_learner -->|"runs chapter"| node_bigram
node_learner -->|"runs chapter"| node_attention
node_learner -->|"runs chapter"| node_multihead
node_learner -->|"runs chapter"| node_block
node_learner -->|"runs chapter"| node_gpt
node_learner -->|"runs chapter"| node_training
node_browser -->|"opens"| node_textbook
node_training -->|"checks gradients"| node_gradient
node_training -->|"updates parameters"| node_adam
node_training -->|"generates samples"| node_generation
node_gradient -->|"validates gradients"| node_training
node_adam -->|"adjusts parameters"| node_training

click node_tokenize "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step1_tokenizer.py"
click node_embeddings "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step2_embeddings.py"
click node_bigram "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step3_bigram_train.py"
click node_attention "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step4_attention.py"
click node_multihead "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step5_multihead_positional.py"
click node_block "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step6_transformer_block.py"
click node_gpt "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step7_tiny_gpt.py"
click node_training "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py"
click node_gradient "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py"
click node_adam "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py"
click node_generation "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py"
click node_textbook "https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/textbook/index.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_tokenize,node_embeddings,node_bigram,node_attention,node_multihead,node_block,node_gpt,node_training,node_gradient,node_adam,node_generation,node_browser toneBlue
class node_textbook toneAmber
class node_learner toneIndigo
```

## Components

### Model-building course

- [Tokenization](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step1_tokenizer.py): `code/step1_tokenizer.py`
- [Embeddings](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step2_embeddings.py): `code/step2_embeddings.py`
- [Bigram training](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step3_bigram_train.py): `code/step3_bigram_train.py`
- [Self attention](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step4_attention.py): `code/step4_attention.py`
- [Heads and position](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step5_multihead_positional.py): `code/step5_multihead_positional.py`
- [Transformer block](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step6_transformer_block.py): `code/step6_transformer_block.py`
- [Tiny GPT](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step7_tiny_gpt.py): `code/step7_tiny_gpt.py`
- [GPT training](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py): `code/step8_train_tiny_gpt.py`
- [Gradient check](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py): `code/step8_train_tiny_gpt.py`
- [Adam optimizer](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py): `code/step8_train_tiny_gpt.py`
- [Text generation](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/code/step8_train_tiny_gpt.py): `code/step8_train_tiny_gpt.py`

### Interactive textbook

- [Course textbook](https://github.com/salmanmik/tiny-llm-from-scratch/blob/main/textbook/index.html): `textbook/index.html`

### Other components

- **Learner**
- **Learner browser**

## Connections

- Learner → Tokenization: runs chapter (evidence: `README.md`)
- Learner → Embeddings: runs chapter (evidence: `README.md`)
- Learner → Bigram training: runs chapter (evidence: `README.md`)
- Learner → Self attention: runs chapter (evidence: `README.md`)
- Learner → Heads and position: runs chapter (evidence: `README.md`)
- Learner → Transformer block: runs chapter (evidence: `README.md`)
- Learner → Tiny GPT: runs chapter (evidence: `README.md`)
- Learner → GPT training: runs chapter (evidence: `README.md`)
- Learner browser → Course textbook: opens (evidence: `README.md`)
- GPT training → Gradient check: checks gradients (evidence: `code/step8_train_tiny_gpt.py`)
- GPT training → Adam optimizer: updates parameters (evidence: `code/step8_train_tiny_gpt.py`)
- GPT training → Text generation: generates samples (evidence: `code/step8_train_tiny_gpt.py`)
- Gradient check → GPT training: validates gradients (evidence: `code/step8_train_tiny_gpt.py`)
- Adam optimizer → GPT training: adjusts parameters (evidence: `code/step8_train_tiny_gpt.py`)
