# bartdesmet/tinyllm architecture

> Architecture diagram of the GitHub repository bartdesmet/tinyllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bartdesmet/tinyllm
- Repository: https://github.com/bartdesmet/tinyllm

## Overview

TinyLLM is an educational, scalar-autograd GPT language-model demo. Its main workflow reads a corpus, builds a tokenizer and decoder-only Transformer, trains next-token predictions (or loads an inference checkpoint), then autoregressively continues a user prompt. The graph preserves both tokenizer choices and the model’s principal layers. Relationships among the model components are supported by the README and source-index references; unsampled implementations are represented only at their documented subsystem level.

Main entry points: [`TinyLLM/Model/GptModel.cs`](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/GptModel.cs) (GPT model), [`TinyLLM/Program.cs`](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Program.cs) (Demo entry point), [`TinyLLM/Model/TransformerBlock.cs`](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/TransformerBlock.cs) (Transformer blocks), [`TinyLLM/Training/Trainer.cs`](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/Trainer.cs) (Training loop)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_app["Application flow"]
  node_program["Demo entry point<br/>[Program.cs]"]
  node_corpus["Training corpus<br/>[corpus.txt]"]
  node_checkpoint["Inference checkpoint<br/>[ModelCheckpoint.cs]"]
end

subgraph group_tokens["Tokenization"]
  node_tokenizer_contract["Tokenizer contract<br/>[ITokenizer.cs]"]
  node_char_tokenizer["Character tokenizer<br/>[CharTokenizer.cs]"]
  node_bpe_tokenizer["Byte-level BPE"]
  node_token_debug["Token debug view"]
end

subgraph group_network["GPT model"]
  node_model["GPT model<br/>[GptModel.cs]"]
  node_embedding["Token and position embeddings<br/>[Embedding.cs]"]
  node_block["Transformer blocks"]
  node_attention["Causal attention"]
  node_feedforward["Feed-forward layer<br/>[FeedForward.cs]"]
  node_layernorm["Layer normalization<br/>[LayerNorm.cs]"]
  node_linear["Linear projection<br/>[Linear.cs]"]
  node_module["Parameter enumeration<br/>[Module.cs]"]
  node_autograd["Scalar autograd<br/>[Value.cs]"]
end

subgraph group_learning["Training"]
  node_trainer["Training loop<br/>[Trainer.cs]"]
  node_loss["Cross-entropy loss"]
  node_optimizer["AdamW optimizer<br/>[AdamW.cs]"]
end

subgraph group_generation["Inference"]
  node_sampler["Autoregressive sampler<br/>[Sampler.cs]"]
end

node_user(("User"))

node_user -->|"runs and prompts"| node_program
node_program -->|"loads corpus"| node_corpus
node_program -->|"builds or selects"| node_char_tokenizer
node_program -->|"trains or selects"| node_bpe_tokenizer
node_char_tokenizer -->|"implements"| node_tokenizer_contract
node_bpe_tokenizer -->|"implements"| node_tokenizer_contract
node_char_tokenizer -->|"formats mappings with"| node_token_debug
node_bpe_tokenizer -->|"formats mappings with"| node_token_debug
node_program -->|"constructs"| node_model
node_model -->|"looks up embeddings"| node_embedding
node_model -->|"forwards through"| node_block
node_block -->|"uses"| node_attention
node_block -->|"uses"| node_feedforward
node_block -->|"normalizes with"| node_layernorm
node_model -->|"applies final normalization"| node_layernorm
node_model -->|"projects logits with"| node_linear
node_attention -->|"projects with"| node_linear
node_feedforward -->|"projects with"| node_linear
node_model -->|"enumerates parameters through"| node_module
node_module -->|"enumerates scalar parameters"| node_autograd
node_program -->|"starts training"| node_trainer
node_trainer -->|"forwards token windows through"| node_model
node_trainer -->|"computes"| node_loss
node_trainer -->|"updates parameters with"| node_optimizer
node_loss -->|"builds differentiable loss"| node_autograd
node_optimizer -->|"updates scalar values"| node_autograd
node_program -->|"loads or saves"| node_checkpoint
node_checkpoint -->|"rebuilds model"| node_model
node_checkpoint -->|"restores tokenizer"| node_tokenizer_contract
node_program -->|"requests generation"| node_sampler
node_sampler -->|"runs context through"| node_model
node_sampler -->|"encodes prompt and decodes output"| node_tokenizer_contract
node_sampler -->|"streams generated text"| node_user

click node_program "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Program.cs"
click node_corpus "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Data/corpus.txt"
click node_checkpoint "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/ModelCheckpoint.cs"
click node_tokenizer_contract "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/ITokenizer.cs"
click node_char_tokenizer "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/CharTokenizer.cs"
click node_bpe_tokenizer "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/BytePairTokenizer.cs"
click node_token_debug "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/TokenizerDebugFormatter.cs"
click node_model "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/GptModel.cs"
click node_embedding "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/Embedding.cs"
click node_block "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/TransformerBlock.cs"
click node_attention "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/CausalSelfAttention.cs"
click node_feedforward "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/FeedForward.cs"
click node_layernorm "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/LayerNorm.cs"
click node_linear "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/Linear.cs"
click node_module "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/Module.cs"
click node_autograd "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Autograd/Value.cs"
click node_trainer "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/Trainer.cs"
click node_loss "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/CrossEntropyLoss.cs"
click node_optimizer "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/AdamW.cs"
click node_sampler "https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Sampling/Sampler.cs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_program,node_corpus,node_checkpoint,node_user toneBlue
class node_tokenizer_contract,node_char_tokenizer,node_bpe_tokenizer,node_token_debug toneAmber
class node_model,node_embedding,node_block,node_attention,node_feedforward,node_layernorm,node_linear,node_module,node_autograd toneMint
class node_trainer,node_loss,node_optimizer toneRose
class node_sampler toneIndigo
```

## Components

### Application flow

- [Demo entry point](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Program.cs): `TinyLLM/Program.cs`
- [Training corpus](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Data/corpus.txt): `TinyLLM/Data/corpus.txt`
- [Inference checkpoint](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/ModelCheckpoint.cs): `TinyLLM/Model/ModelCheckpoint.cs`

### Tokenization

- [Tokenizer contract](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/ITokenizer.cs): `TinyLLM/Tokenization/ITokenizer.cs`
- [Character tokenizer](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/CharTokenizer.cs): `TinyLLM/Tokenization/CharTokenizer.cs`
- [Byte-level BPE](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/BytePairTokenizer.cs): `TinyLLM/Tokenization/BytePairTokenizer.cs`
- [Token debug view](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Tokenization/TokenizerDebugFormatter.cs): `TinyLLM/Tokenization/TokenizerDebugFormatter.cs`

### GPT model

- [GPT model](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/GptModel.cs): `TinyLLM/Model/GptModel.cs`
- [Token and position embeddings](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/Embedding.cs): `TinyLLM/Model/Embedding.cs`
- [Transformer blocks](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/TransformerBlock.cs): `TinyLLM/Model/TransformerBlock.cs`
- [Causal attention](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/CausalSelfAttention.cs): `TinyLLM/Model/CausalSelfAttention.cs`
- [Feed-forward layer](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/FeedForward.cs): `TinyLLM/Model/FeedForward.cs`
- [Layer normalization](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/LayerNorm.cs): `TinyLLM/Model/LayerNorm.cs`
- [Linear projection](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/Linear.cs): `TinyLLM/Model/Linear.cs`
- [Parameter enumeration](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Model/Module.cs): `TinyLLM/Model/Module.cs`
- [Scalar autograd](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Autograd/Value.cs): `TinyLLM/Autograd/Value.cs`

### Training

- [Training loop](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/Trainer.cs): `TinyLLM/Training/Trainer.cs`
- [Cross-entropy loss](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/CrossEntropyLoss.cs): `TinyLLM/Training/CrossEntropyLoss.cs`
- [AdamW optimizer](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Training/AdamW.cs): `TinyLLM/Training/AdamW.cs`

### Inference

- [Autoregressive sampler](https://github.com/bartdesmet/tinyllm/blob/master/TinyLLM/Sampling/Sampler.cs): `TinyLLM/Sampling/Sampler.cs`

### Other components

- **User**

## Connections

- User → Demo entry point: runs and prompts (evidence: `TinyLLM/Program.cs`)
- Demo entry point → Training corpus: loads corpus (evidence: `TinyLLM/Program.cs`)
- Demo entry point → Character tokenizer: builds or selects (evidence: `TinyLLM/Program.cs`)
- Demo entry point → Byte-level BPE: trains or selects (evidence: `TinyLLM/Program.cs`)
- Character tokenizer → Tokenizer contract: implements (evidence: `TinyLLM/Tokenization/CharTokenizer.cs`)
- Byte-level BPE → Tokenizer contract: implements (evidence: `TinyLLM/Tokenization/BytePairTokenizer.cs`)
- Character tokenizer → Token debug view: formats mappings with (evidence: `TinyLLM/Tokenization/CharTokenizer.cs`)
- Byte-level BPE → Token debug view: formats mappings with (evidence: `TinyLLM/Tokenization/BytePairTokenizer.cs`)
- Demo entry point → GPT model: constructs (evidence: `TinyLLM/Program.cs`)
- GPT model → Token and position embeddings: looks up embeddings (evidence: `TinyLLM/Model/GptModel.cs`)
- GPT model → Transformer blocks: forwards through (evidence: `TinyLLM/Model/GptModel.cs`)
- Transformer blocks → Causal attention: uses (evidence: `TinyLLM/Model/TransformerBlock.cs`)
- Transformer blocks → Feed-forward layer: uses (evidence: `TinyLLM/Model/TransformerBlock.cs`)
- Transformer blocks → Layer normalization: normalizes with (evidence: `TinyLLM/Model/TransformerBlock.cs`)
- GPT model → Layer normalization: applies final normalization (evidence: `TinyLLM/Model/GptModel.cs`)
- GPT model → Linear projection: projects logits with (evidence: `TinyLLM/Model/GptModel.cs`)
- Causal attention → Linear projection: projects with (evidence: `TinyLLM/Model/CausalSelfAttention.cs`)
- Feed-forward layer → Linear projection: projects with (evidence: `TinyLLM/Model/FeedForward.cs`)
- GPT model → Parameter enumeration: enumerates parameters through (evidence: `TinyLLM/Model/GptModel.cs`)
- Parameter enumeration → Scalar autograd: enumerates scalar parameters (evidence: `TinyLLM/Model/Module.cs`)
- Demo entry point → Training loop: starts training (evidence: `TinyLLM/Program.cs`)
- Training loop → GPT model: forwards token windows through (evidence: `TinyLLM/Training/Trainer.cs`)
- Training loop → Cross-entropy loss: computes (evidence: `TinyLLM/Training/Trainer.cs`)
- Training loop → AdamW optimizer: updates parameters with (evidence: `TinyLLM/Training/Trainer.cs`)
- Cross-entropy loss → Scalar autograd: builds differentiable loss (evidence: `TinyLLM/Training/CrossEntropyLoss.cs`)
- AdamW optimizer → Scalar autograd: updates scalar values (evidence: `TinyLLM/Training/AdamW.cs`)
- Demo entry point → Inference checkpoint: loads or saves (evidence: `TinyLLM/Program.cs`)
- Inference checkpoint → GPT model: rebuilds model (evidence: `TinyLLM/Model/ModelCheckpoint.cs`)
- Inference checkpoint → Tokenizer contract: restores tokenizer (evidence: `TinyLLM/Model/ModelCheckpoint.cs`)
- Demo entry point → Autoregressive sampler: requests generation (evidence: `TinyLLM/Program.cs`)
- Autoregressive sampler → GPT model: runs context through (evidence: `TinyLLM/Sampling/Sampler.cs`)
- Autoregressive sampler → Tokenizer contract: encodes prompt and decodes output (evidence: `TinyLLM/Sampling/Sampler.cs`)
- Autoregressive sampler → User: streams generated text (evidence: `TinyLLM/Sampling/Sampler.cs`)
