How fareedkhan-dev/train-llm-from-scratch works
Train LLM From Scratch is a PyTorch toolkit and control panel for taking text from corpus preparation through pretraining, alignment, evaluation, and chat. The README documents the principal progression; source excerpts additionally show checkpoint-based inference, chat formatting, and the UI job manager. The model internals and several training scripts are not fully sampled, so those links rely on the README’s documented workflow rather than inferred calls.
Main entry points: pretrain_base.py Base pretraining · inference.py Checkpoint inference · app.py Streamlit control panel · sft.py Supervised fine-tuning
Components 23
Data preparation
- Corpus preparation
scripts - Dataset loading
data_loader/data_loader.py - BPE tokenizer
src/tokenizer/bpe.py
Models and runtime
- Classic Transformer
src/models/transformer.py - Modern Transformer
src/models/modern/model.py - Modern blocks
src/models/modern/block.py - Modern attention
src/models/modern/attention.py - Model factory
src/models/factory.py - Checkpoint handling
src/checkpoint.py
Training and alignment
- Base pretraining
scripts/pretrain_base.py - Supervised fine-tuning
src/post_training/sft.py - Reward modeling
src/post_training/reward_train.py - Preference optimization
src/post_training/dpo.py - PPO training
src/post_training/ppo.py - GRPO training
src/post_training/grpo.py
Evaluation and inference
- Chat formatting
src/post_training/chat_template.py - Token sampling
src/inference/sampling.py - Checkpoint inference
src/post_training/inference.py - Evaluation
src/post_training/evaluation.py
Control panel
- Streamlit control panel
ui/app.py - Stage registry
ui/stages.py - Background jobs
ui/jobs.py
Other components
- Engineer
Connections 28
- Engineer → Corpus preparation: prepares data
README.md - Corpus preparation → Dataset loading: produces datasets
README.md - Dataset loading → BPE tokenizer: encodes text
- BPE tokenizer → Base pretraining: provides tokens
README.md - Base pretraining → Classic Transformer: trains model
README.md - Base pretraining → Modern Transformer: trains model
README.md - Modern Transformer → Modern blocks: runs blocks
src/models/modern/model.py - Modern blocks → Modern attention: uses attention
src/models/modern/block.py - Model factory → Classic Transformer: builds model
src/models/factory.py - Model factory → Modern Transformer: builds model
src/models/factory.py - Base pretraining → Checkpoint handling: saves checkpoints
README.md - Checkpoint handling → Checkpoint inference: loads weights
src/post_training/inference.py - Base pretraining → Supervised fine-tuning: fine-tunes base
README.md - Supervised fine-tuning → Reward modeling: trains reward model
README.md - Reward modeling → PPO training: supports PPO
README.md - Supervised fine-tuning → Preference optimization: optimizes preferences
README.md - Preference optimization → GRPO training: continues alignment
README.md - Checkpoint inference → Model factory: builds model
src/post_training/inference.py - Checkpoint inference → Checkpoint handling: loads checkpoint
src/post_training/inference.py - Checkpoint inference → Chat formatting: formats prompts
src/post_training/inference.py - Checkpoint inference → Evaluation: generates replies
src/post_training/inference.py - Evaluation → Token sampling: samples tokens
- Engineer → Streamlit control panel: launches workflow
README.md - Streamlit control panel → Background jobs: manages jobs
ui/app.py - Background jobs → Stage registry: reads stages
ui/jobs.py - Stage registry → Base pretraining: configures stage
ui/stages.py - Streamlit control panel → Evaluation: offers evaluation
README.md - Streamlit control panel → Checkpoint inference: offers chat