# joker383627/a-llm-from-scratch architecture

> Architecture diagram of the GitHub repository joker383627/a-llm-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/joker383627/a-llm-from-scratch
- Repository: https://github.com/joker383627/a-llm-from-scratch
- Video tour (about a minute): https://gitdiagram.com/joker383627/a-llm-from-scratch/video

## Overview

Tiny LLM is an experimental from-scratch autoregressive language-model project. Its main workflow trains a decoder-only Transformer on pretokenized FineWeb data, validates it, and saves resumable checkpoints; a CLI loads model parameters and generates text from prompts. The repository also includes experimental BPE training and encoding. The README documents the FineWeb-to-binary preparation stage, but no preparation implementation appears in the sampled source or tree, so that link reflects documentation rather than inspected code.

Main entry points: [`Brain/main.py`](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py) (Training Entry), [`Brain/train.py`](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/train.py) (Train and Evaluate), [`chat.py`](https://github.com/joker383627/a-llm-from-scratch/blob/main/chat.py) (Chat CLI), [`Tokenizer/tokenizer_modified.py`](https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/tokenizer_modified.py) (Python BPE Tokenizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_data["Training Data"]
  node_fineweb[("FineWeb")]
  node_gpt2tok["GPT-2 Tokenizer"]
  node_binary[("Token Binaries")]
  node_dataset["Mapped Dataset<br/>[preprocess.py]"]
  node_loader["DataLoaders<br/>[main.py]"]
end

subgraph group_model["Model Training"]
  node_trainer["Training Entry<br/>[main.py]"]
  node_trainloop["Train and Evaluate<br/>[train.py]"]
  node_transformer["Decoder Transformer<br/>[transformer.py]"]
  node_optimizer["AdamW Optimizer<br/>[main.py]"]
  node_checkpointcode["Checkpoint Manager<br/>[checkpoints.py]"]
  node_checkpointstore[("Checkpoint Files")]
end

subgraph group_inference["Text Generation"]
  node_chat["Chat CLI<br/>[chat.py]"]
  node_generate["Generation<br/>[generate.py]"]
  node_response["Generated Text"]
end

subgraph group_bpe["Experimental BPE"]
  node_bpetokenizer["Python BPE Tokenizer"]
  node_bpetrainee["BPE Trainer<br/>[bpe_trainer.cpp]"]
  node_bpeencoder["BPE Encoder<br/>[bpe_encoder.cpp]"]
  node_vocabfile["Vocabulary JSON<br/>[new_tok.json]"]
end

node_user(("Prompt User"))
node_bpeuser(("Tokenizer Developer"))

node_fineweb -->|"tokenize per README"| node_gpt2tok
node_gpt2tok -->|"produce IDs per README"| node_binary
node_binary -->|"memory-map"| node_dataset
node_dataset -->|"supply samples"| node_loader
node_trainer -->|"construct"| node_dataset
node_trainer -->|"construct"| node_loader
node_trainer -->|"construct"| node_transformer
node_trainer -->|"construct"| node_optimizer
node_trainer -->|"load and save"| node_checkpointcode
node_checkpointcode -->|"read and write"| node_checkpointstore
node_trainer -->|"run training and validation"| node_trainloop
node_trainloop -->|"iterate batches"| node_loader
node_trainloop -->|"forward and backpropagate"| node_transformer
node_trainloop -->|"update parameters"| node_optimizer
node_trainloop -->|"save progress"| node_checkpointcode
node_user -->|"enter prompts"| node_chat
node_chat -->|"load parameters"| node_transformer
node_chat -->|"request completion"| node_generate
node_generate -->|"score next tokens"| node_transformer
node_generate -->|"decode tokens"| node_response
node_chat -->|"display"| node_response
node_bpeuser -->|"train or encode"| node_bpetokenizer
node_bpetokenizer -->|"train merges"| node_bpetrainee
node_bpetokenizer -->|"encode text"| node_bpeencoder
node_bpetokenizer -->|"save vocabulary"| node_vocabfile

click node_dataset "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/preprocess.py"
click node_loader "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py"
click node_trainer "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py"
click node_trainloop "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/train.py"
click node_transformer "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/transformer.py"
click node_optimizer "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py"
click node_checkpointcode "https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/checkpoints.py"
click node_chat "https://github.com/joker383627/a-llm-from-scratch/blob/main/chat.py"
click node_generate "https://github.com/joker383627/a-llm-from-scratch/blob/main/GPTapp/generate.py"
click node_bpetokenizer "https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/tokenizer_modified.py"
click node_bpetrainee "https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/bpe_trainer.cpp"
click node_bpeencoder "https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/bpe_encoder.cpp"
click node_vocabfile "https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/new_tok.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fineweb,node_gpt2tok,node_binary,node_dataset,node_loader,node_user toneBlue
class node_trainer,node_trainloop,node_transformer,node_optimizer,node_checkpointcode,node_checkpointstore toneAmber
class node_chat,node_generate,node_response toneMint
class node_bpetokenizer,node_bpetrainee,node_bpeencoder,node_vocabfile toneRose
class node_bpeuser toneIndigo
```

## Components

### Training Data

- **FineWeb**
- **GPT-2 Tokenizer**
- **Token Binaries**
- [Mapped Dataset](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/preprocess.py): `Brain/preprocess.py`
- [DataLoaders](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py): `Brain/main.py`

### Model Training

- [Training Entry](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py): `Brain/main.py`
- [Train and Evaluate](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/train.py): `Brain/train.py`
- [Decoder Transformer](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/transformer.py): `Brain/transformer.py`
- [AdamW Optimizer](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/main.py): `Brain/main.py`
- [Checkpoint Manager](https://github.com/joker383627/a-llm-from-scratch/blob/main/Brain/checkpoints.py): `Brain/checkpoints.py`
- **Checkpoint Files**

### Text Generation

- [Chat CLI](https://github.com/joker383627/a-llm-from-scratch/blob/main/chat.py): `chat.py`
- [Generation](https://github.com/joker383627/a-llm-from-scratch/blob/main/GPTapp/generate.py): `GPTapp/generate.py`
- **Generated Text**

### Experimental BPE

- [Python BPE Tokenizer](https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/tokenizer_modified.py): `Tokenizer/tokenizer_modified.py`
- [BPE Trainer](https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/bpe_trainer.cpp): `Tokenizer/bpe_trainer.cpp`
- [BPE Encoder](https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/bpe_encoder.cpp): `Tokenizer/bpe_encoder.cpp`
- [Vocabulary JSON](https://github.com/joker383627/a-llm-from-scratch/blob/main/Tokenizer/new_tok.json): `Tokenizer/new_tok.json`

### Other components

- **Prompt User**
- **Tokenizer Developer**

## Connections

- FineWeb → GPT-2 Tokenizer: tokenize per README
- GPT-2 Tokenizer → Token Binaries: produce IDs per README
- Token Binaries → Mapped Dataset: memory-map
- Mapped Dataset → DataLoaders: supply samples
- Training Entry → Mapped Dataset: construct
- Training Entry → DataLoaders: construct
- Training Entry → Decoder Transformer: construct
- Training Entry → AdamW Optimizer: construct
- Training Entry → Checkpoint Manager: load and save
- Checkpoint Manager → Checkpoint Files: read and write
- Training Entry → Train and Evaluate: run training and validation
- Train and Evaluate → DataLoaders: iterate batches
- Train and Evaluate → Decoder Transformer: forward and backpropagate
- Train and Evaluate → AdamW Optimizer: update parameters
- Train and Evaluate → Checkpoint Manager: save progress
- Prompt User → Chat CLI: enter prompts
- Chat CLI → Decoder Transformer: load parameters
- Chat CLI → Generation: request completion
- Generation → Decoder Transformer: score next tokens
- Generation → Generated Text: decode tokens
- Chat CLI → Generated Text: display
- Tokenizer Developer → Python BPE Tokenizer: train or encode
- Python BPE Tokenizer → BPE Trainer: train merges
- Python BPE Tokenizer → BPE Encoder: encode text
- Python BPE Tokenizer → Vocabulary JSON: save vocabulary