How karpathy/nanochat works
nanochat is a compact experimental harness for training and using GPT language models on a single GPU node. The main workflow prepares text data and a tokenizer, trains a depth-configured GPT, saves checkpoints, then loads models for supervised or reinforcement fine-tuning, task evaluation, and interactive inference. The README also highlights compute-optimal model scaling and tokenizer evaluation. Edges cite README claims or SOURCE INDEX references; unsampled subsystem details are represented only where documented, so some relationships are omitted.
Main entry points: base_train.py Pretraining · checkpoint_manager.py Checkpoint manager · common.py Task datasets · engine.py Generation engine
Components 24
Data and tokenization
- Pretraining corpus
nanochat/dataset.py - Training data loader
nanochat/dataloader.py - Tokenizer training
scripts/tok_train.py - Tokenizer
nanochat/tokenizer.py - Tokenizer evaluation
scripts/tok_eval.py
Model and training
- Pretraining
scripts/base_train.py - GPT transformer
nanochat/gpt.py - Attention kernels
nanochat/flash_attention.py - Optimizer
nanochat/optim.py - FP8 training
nanochat/fp8.py - Generation engine
nanochat/engine.py - Checkpoint manager
nanochat/checkpoint_manager.py
Chat adaptation
- Supervised fine-tuning
scripts/chat_sft.py - Reinforcement fine-tuning
scripts/chat_rl.py
Evaluation and inference
- Task datasets
tasks/common.py - GSM8K
tasks/gsm8k.py - MMLU
tasks/mmlu.py - Other task suites
tasks - Pretraining evaluation
scripts/base_eval.py - CORE metric
nanochat/core_eval.py - Chat evaluation
scripts/chat_eval.py - Chat CLI
scripts/chat_cli.py
Other components
- Researcher
- Dataset hub
Connections 29
- Researcher → Pretraining: configures depth
README.md - Researcher → Tokenizer training: trains tokenizer
- Researcher → Chat CLI: chats with model
README.md - Pretraining corpus → Training data loader: provides documents
nanochat/dataloader.py - Tokenizer training → Tokenizer: trains and saves
scripts/tok_train.py - Tokenizer evaluation → Tokenizer: evaluates
scripts/tok_eval.py - Dataset hub → Task datasets: loads datasets
- GSM8K → Task datasets: uses task base
tasks/gsm8k.py - MMLU → Task datasets: uses task base
tasks/mmlu.py - Other task suites → Task datasets: uses task base
tasks/gsm8k.py - Training data loader → Pretraining: supplies batches
scripts/base_train.py - Pretraining → GPT transformer: trains model
scripts/base_train.py - Pretraining → Optimizer: updates weights
- Pretraining → FP8 training: enables FP8
scripts/base_train.py - GPT transformer → Attention kernels: calls attention
nanochat/gpt.py - Pretraining → Checkpoint manager: saves state
scripts/base_train.py - Supervised fine-tuning → Checkpoint manager: loads model
scripts/chat_sft.py - Supervised fine-tuning → Task datasets: uses conversations
scripts/chat_sft.py - Supervised fine-tuning → Generation engine: evaluates samples
scripts/chat_sft.py - Reinforcement fine-tuning → Checkpoint manager: loads model
scripts/chat_rl.py - Reinforcement fine-tuning → GSM8K: uses math tasks
scripts/chat_rl.py - Pretraining evaluation → Checkpoint manager: loads model
scripts/base_eval.py - Pretraining evaluation → CORE metric: computes metric
scripts/base_eval.py - Pretraining evaluation → Generation engine: evaluates model
scripts/base_eval.py - Chat evaluation → Checkpoint manager: loads model
scripts/chat_eval.py - Chat evaluation → Generation engine: generates responses
scripts/chat_eval.py - Chat evaluation → Other task suites: evaluates suites
scripts/chat_eval.py - Chat CLI → Checkpoint manager: loads model
scripts/chat_cli.py - Chat CLI → Generation engine: generates replies
scripts/chat_cli.py