# multix0/chargpt architecture

> Architecture diagram of the GitHub repository multix0/chargpt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/multix0/chargpt
- Repository: https://github.com/multix0/chargpt

## Overview

This repository is a from-scratch, character-level GPT training and interactive inference application using C++ and CUDA. Its main workflow trains on Shakespeare text, serializes the model and tokenizer, then lets a user prompt the trained model in a terminal. The graph preserves the documented training, Transformer, GPU, and chat capabilities. Some internal relationships are based on README descriptions because their implementation excerpts were not provided.

Main entry points: [`src/model.hpp`](https://github.com/multix0/chargpt/blob/main/src/model.hpp) (GPT Model), [`src/main.cpp`](https://github.com/multix0/chargpt/blob/main/src/main.cpp) (Training Driver), [`src/chat.cpp`](https://github.com/multix0/chargpt/blob/main/src/chat.cpp) (Interactive Chat), [`src/tokenizer.hpp`](https://github.com/multix0/chargpt/blob/main/src/tokenizer.hpp) (Character Tokenizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training"]
  node_trainer["Training Driver<br/>[main.cpp]"]
  node_dataset["Shakespeare Text<br/>[input.txt]"]
  node_tokenizer["Character Tokenizer<br/>[tokenizer.hpp]"]
  node_batching["Sequence Sampling<br/>[main.cpp]"]
  node_loss["Loss and Gradients<br/>[main.cpp]"]
  node_accumulation["Gradient Accumulation<br/>[main.cpp]"]
  node_optimizer["Adam Optimizer<br/>[optimizer.hpp]"]
end

subgraph group_model["GPT Model"]
  node_gpt["GPT Model<br/>[model.hpp]"]
  node_embeddings["Token and Position Embeddings<br/>[model.hpp]"]
  node_transformer["Transformer Blocks<br/>[model.hpp]"]
  node_attention["Causal Multi-Head Attention<br/>[model.hpp]"]
  node_feedforward["GELU Feed-Forward<br/>[model.hpp]"]
  node_layernorm["Layer Normalization<br/>[model.hpp]"]
end

subgraph group_gpu["GPU Runtime"]
  node_matrix["GPU Matrix Operations<br/>[matrix.hpp]"]
  node_kernels["CUDA Kernels<br/>[cuda_kernels.cu]"]
end

subgraph group_chat["Chat Inference"]
  node_artifact["Model Binary"]
  node_chat["Interactive Chat<br/>[chat.cpp]"]
end

node_user(("Terminal User"))

node_trainer -->|"reads text"| node_dataset
node_trainer -->|"builds and encodes"| node_tokenizer
node_trainer -->|"samples sequences"| node_batching
node_batching -->|"supplies targets"| node_loss
node_trainer -->|"runs forward and backward"| node_gpt
node_gpt -->|"uses token IDs"| node_tokenizer
node_gpt -->|"embeds tokens"| node_embeddings
node_gpt -->|"applies blocks"| node_transformer
node_transformer -->|"applies attention"| node_attention
node_transformer -->|"applies feed-forward"| node_feedforward
node_transformer -->|"normalizes activations"| node_layernorm
node_loss -->|"supplies gradients"| node_gpt
node_trainer -->|"accumulates gradients"| node_accumulation
node_trainer -->|"updates parameters"| node_optimizer
node_gpt -->|"uses matrix operations"| node_matrix
node_matrix -->|"dispatches operations"| node_kernels
node_optimizer -->|"dispatches Adam update"| node_kernels
node_trainer -->|"saves model and vocabulary"| node_artifact
node_user -->|"enters prompt"| node_chat
node_chat -->|"loads model"| node_artifact
node_chat -->|"encodes and decodes"| node_tokenizer
node_chat -->|"requests generation"| node_gpt
node_gpt -->|"returns generated tokens"| node_chat

click node_trainer "https://github.com/multix0/chargpt/blob/main/src/main.cpp"
click node_dataset "https://github.com/multix0/chargpt/blob/main/data/input.txt"
click node_tokenizer "https://github.com/multix0/chargpt/blob/main/src/tokenizer.hpp"
click node_batching "https://github.com/multix0/chargpt/blob/main/src/main.cpp"
click node_loss "https://github.com/multix0/chargpt/blob/main/src/main.cpp"
click node_accumulation "https://github.com/multix0/chargpt/blob/main/src/main.cpp"
click node_optimizer "https://github.com/multix0/chargpt/blob/main/src/optimizer.hpp"
click node_gpt "https://github.com/multix0/chargpt/blob/main/src/model.hpp"
click node_embeddings "https://github.com/multix0/chargpt/blob/main/src/model.hpp"
click node_transformer "https://github.com/multix0/chargpt/blob/main/src/model.hpp"
click node_attention "https://github.com/multix0/chargpt/blob/main/src/model.hpp"
click node_feedforward "https://github.com/multix0/chargpt/blob/main/src/model.hpp"
click node_layernorm "https://github.com/multix0/chargpt/blob/main/src/model.hpp"
click node_matrix "https://github.com/multix0/chargpt/blob/main/src/matrix.hpp"
click node_kernels "https://github.com/multix0/chargpt/blob/main/src/cuda_kernels.cu"
click node_chat "https://github.com/multix0/chargpt/blob/main/src/chat.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_trainer,node_dataset,node_tokenizer,node_batching,node_loss,node_accumulation,node_optimizer,node_user toneBlue
class node_gpt,node_embeddings,node_transformer,node_attention,node_feedforward,node_layernorm toneAmber
class node_matrix,node_kernels toneMint
class node_artifact,node_chat toneRose
```

## Components

### Training

- [Training Driver](https://github.com/multix0/chargpt/blob/main/src/main.cpp): `src/main.cpp`
- [Shakespeare Text](https://github.com/multix0/chargpt/blob/main/data/input.txt): `data/input.txt`
- [Character Tokenizer](https://github.com/multix0/chargpt/blob/main/src/tokenizer.hpp): `src/tokenizer.hpp`
- [Sequence Sampling](https://github.com/multix0/chargpt/blob/main/src/main.cpp): `src/main.cpp`
- [Loss and Gradients](https://github.com/multix0/chargpt/blob/main/src/main.cpp): `src/main.cpp`
- [Gradient Accumulation](https://github.com/multix0/chargpt/blob/main/src/main.cpp): `src/main.cpp`
- [Adam Optimizer](https://github.com/multix0/chargpt/blob/main/src/optimizer.hpp): `src/optimizer.hpp`

### GPT Model

- [GPT Model](https://github.com/multix0/chargpt/blob/main/src/model.hpp): `src/model.hpp`
- [Token and Position Embeddings](https://github.com/multix0/chargpt/blob/main/src/model.hpp): `src/model.hpp`
- [Transformer Blocks](https://github.com/multix0/chargpt/blob/main/src/model.hpp): `src/model.hpp`
- [Causal Multi-Head Attention](https://github.com/multix0/chargpt/blob/main/src/model.hpp): `src/model.hpp`
- [GELU Feed-Forward](https://github.com/multix0/chargpt/blob/main/src/model.hpp): `src/model.hpp`
- [Layer Normalization](https://github.com/multix0/chargpt/blob/main/src/model.hpp): `src/model.hpp`

### GPU Runtime

- [GPU Matrix Operations](https://github.com/multix0/chargpt/blob/main/src/matrix.hpp): `src/matrix.hpp`
- [CUDA Kernels](https://github.com/multix0/chargpt/blob/main/src/cuda_kernels.cu): `src/cuda_kernels.cu`

### Chat Inference

- **Model Binary**
- [Interactive Chat](https://github.com/multix0/chargpt/blob/main/src/chat.cpp): `src/chat.cpp`

### Other components

- **Terminal User**

## Connections

- Training Driver → Shakespeare Text: reads text (evidence: `README.md`)
- Training Driver → Character Tokenizer: builds and encodes (evidence: `src/main.cpp`)
- Training Driver → Sequence Sampling: samples sequences (evidence: `src/main.cpp`)
- Sequence Sampling → Loss and Gradients: supplies targets (evidence: `src/main.cpp`)
- Training Driver → GPT Model: runs forward and backward (evidence: `src/main.cpp`)
- GPT Model → Character Tokenizer: uses token IDs (evidence: `src/model.hpp`)
- GPT Model → Token and Position Embeddings: embeds tokens (evidence: `README.md`)
- GPT Model → Transformer Blocks: applies blocks (evidence: `README.md`)
- Transformer Blocks → Causal Multi-Head Attention: applies attention (evidence: `README.md`)
- Transformer Blocks → GELU Feed-Forward: applies feed-forward (evidence: `README.md`)
- Transformer Blocks → Layer Normalization: normalizes activations (evidence: `README.md`)
- Loss and Gradients → GPT Model: supplies gradients (evidence: `src/main.cpp`)
- Training Driver → Gradient Accumulation: accumulates gradients (evidence: `src/main.cpp`)
- Training Driver → Adam Optimizer: updates parameters (evidence: `src/main.cpp`)
- GPT Model → GPU Matrix Operations: uses matrix operations (evidence: `src/model.hpp`)
- GPU Matrix Operations → CUDA Kernels: dispatches operations (evidence: `src/matrix.hpp`)
- Adam Optimizer → CUDA Kernels: dispatches Adam update (evidence: `src/optimizer.hpp`)
- Training Driver → Model Binary: saves model and vocabulary (evidence: `README.md`)
- Terminal User → Interactive Chat: enters prompt (evidence: `src/chat.cpp`)
- Interactive Chat → Model Binary: loads model (evidence: `src/chat.cpp`)
- Interactive Chat → Character Tokenizer: encodes and decodes (evidence: `src/chat.cpp`)
- Interactive Chat → GPT Model: requests generation (evidence: `src/chat.cpp`)
- GPT Model → Interactive Chat: returns generated tokens (evidence: `README.md`)
