How joker383627/a-llm-from-scratch works
Tiny LLM is an experimental from-scratch autoregressive language-model project. Its main workflow trains a decoder-only Transformer on pretokenized FineWeb data, validates it, and saves resumable checkpoints; a CLI loads model parameters and generates text from prompts. The repository also includes experimental BPE training and encoding. The README documents the FineWeb-to-binary preparation stage, but no preparation implementation appears in the sampled source or tree, so that link reflects documentation rather than inspected code.
Main entry points: main.py Training Entry · train.py Train and Evaluate · chat.py Chat CLI · tokenizer_modified.py Python BPE Tokenizer
Components 20
Training Data
- FineWeb
- GPT-2 Tokenizer
- Token Binaries
- Mapped Dataset
Brain/preprocess.py - DataLoaders
Brain/main.py
Model Training
- Training Entry
Brain/main.py - Train and Evaluate
Brain/train.py - Decoder Transformer
Brain/transformer.py - AdamW Optimizer
Brain/main.py - Checkpoint Manager
Brain/checkpoints.py - Checkpoint Files
Text Generation
- Chat CLI
chat.py - Generation
GPTapp/generate.py - Generated Text
Experimental BPE
- Python BPE Tokenizer
Tokenizer/tokenizer_modified.py - BPE Trainer
Tokenizer/bpe_trainer.cpp - BPE Encoder
Tokenizer/bpe_encoder.cpp - Vocabulary JSON
Tokenizer/new_tok.json
Other components
- Prompt User
- Tokenizer Developer
Connections 25
- FineWeb → GPT-2 Tokenizer: tokenize per README
- GPT-2 Tokenizer → Token Binaries: produce IDs per README
- Token Binaries → Mapped Dataset: memory-map
- Mapped Dataset → DataLoaders: supply samples
- Training Entry → Mapped Dataset: construct
- Training Entry → DataLoaders: construct
- Training Entry → Decoder Transformer: construct
- Training Entry → AdamW Optimizer: construct
- Training Entry → Checkpoint Manager: load and save
- Checkpoint Manager → Checkpoint Files: read and write
- Training Entry → Train and Evaluate: run training and validation
- Train and Evaluate → DataLoaders: iterate batches
- Train and Evaluate → Decoder Transformer: forward and backpropagate
- Train and Evaluate → AdamW Optimizer: update parameters
- Train and Evaluate → Checkpoint Manager: save progress
- Prompt User → Chat CLI: enter prompts
- Chat CLI → Decoder Transformer: load parameters
- Chat CLI → Generation: request completion
- Generation → Decoder Transformer: score next tokens
- Generation → Generated Text: decode tokens
- Chat CLI → Generated Text: display
- Tokenizer Developer → Python BPE Tokenizer: train or encode
- Python BPE Tokenizer → BPE Trainer: train merges
- Python BPE Tokenizer → BPE Encoder: encode text
- Python BPE Tokenizer → Vocabulary JSON: save vocabulary