How multix0/chargpt works
This repository is a from-scratch, character-level GPT training and interactive inference application using C++ and CUDA. Its main workflow trains on Shakespeare text, serializes the model and tokenizer, then lets a user prompt the trained model in a terminal. The graph preserves the documented training, Transformer, GPU, and chat capabilities. Some internal relationships are based on README descriptions because their implementation excerpts were not provided.
Main entry points: model.hpp GPT Model · main.cpp Training Driver · chat.cpp Interactive Chat · tokenizer.hpp Character Tokenizer
Components 18
Training
- Training Driver
src/main.cpp - Shakespeare Text
data/input.txt - Character Tokenizer
src/tokenizer.hpp - Sequence Sampling
src/main.cpp - Loss and Gradients
src/main.cpp - Gradient Accumulation
src/main.cpp - Adam Optimizer
src/optimizer.hpp
GPT Model
- GPT Model
src/model.hpp - Token and Position Embeddings
src/model.hpp - Transformer Blocks
src/model.hpp - Causal Multi-Head Attention
src/model.hpp - GELU Feed-Forward
src/model.hpp - Layer Normalization
src/model.hpp
GPU Runtime
- GPU Matrix Operations
src/matrix.hpp - CUDA Kernels
src/cuda_kernels.cu
Chat Inference
- Model Binary
- Interactive Chat
src/chat.cpp
Other components
- Terminal User
Connections 23
- Training Driver → Shakespeare Text: reads text
README.md - Training Driver → Character Tokenizer: builds and encodes
src/main.cpp - Training Driver → Sequence Sampling: samples sequences
src/main.cpp - Sequence Sampling → Loss and Gradients: supplies targets
src/main.cpp - Training Driver → GPT Model: runs forward and backward
src/main.cpp - GPT Model → Character Tokenizer: uses token IDs
src/model.hpp - GPT Model → Token and Position Embeddings: embeds tokens
README.md - GPT Model → Transformer Blocks: applies blocks
README.md - Transformer Blocks → Causal Multi-Head Attention: applies attention
README.md - Transformer Blocks → GELU Feed-Forward: applies feed-forward
README.md - Transformer Blocks → Layer Normalization: normalizes activations
README.md - Loss and Gradients → GPT Model: supplies gradients
src/main.cpp - Training Driver → Gradient Accumulation: accumulates gradients
src/main.cpp - Training Driver → Adam Optimizer: updates parameters
src/main.cpp - GPT Model → GPU Matrix Operations: uses matrix operations
src/model.hpp - GPU Matrix Operations → CUDA Kernels: dispatches operations
src/matrix.hpp - Adam Optimizer → CUDA Kernels: dispatches Adam update
src/optimizer.hpp - Training Driver → Model Binary: saves model and vocabulary
README.md - Terminal User → Interactive Chat: enters prompt
src/chat.cpp - Interactive Chat → Model Binary: loads model
src/chat.cpp - Interactive Chat → Character Tokenizer: encodes and decodes
src/chat.cpp - Interactive Chat → GPT Model: requests generation
src/chat.cpp - GPT Model → Interactive Chat: returns generated tokens
README.md