How ayushsyntax/gpt-from-scratch-in-pytorch works
Project Type and Purpose
• GPT-Mini is an open-source, research-style machine learning project: a compact, decoder-only Transformer language model implemented in PyTorch, trained on Shakespeare’s complete works for character-level text generation. It also includes scripts for training, evaluation, and a Gradio-based deployment.
File Structure Analysis
Top-level:
• configs/ – YAML files for model/training hyperparameters
• data/ – raw text corpus (tinyshakespeare.txt)
• deploy/ – inference app (Gradio via app.py)
• experiments/ – experiment-tracking notes or scripts
• src/ – core code
• tests/ – unit tests
• train.py – entry point for training
• evaluate.py – evaluation script
• requirements.txt, LICENSE, .gitignore
src/ subdirectories:
• data/ – dataloader, tokenizer
• model/ – Transformer blocks (attention, embeddings, transformer)
• train/ – config loader, Trainer class
• utils/ – debug helpers, model export
README Insights
• Describes end-to-end workflow: Prompt → Tokenize → Embed → Decoder×6 → LM head → Softmax → Generation
• Highlights core components: Tokenizer, Embeddings, Decoder Blocks, Generation loop
• Lists tech stack: Python, PyTorch, Gradio for deployment
• Provides model specs and evaluation metrics
a. Main Components
1. Data Layer
• raw corpus (data/tinyshakespeare.txt)
• configs (configs/*.yaml)
2. Preprocessing
• Tokenizer (src/data/tokenizer.py)
• Dataloader (src/data/dataloader.py)
3. Model
• Input Embeddings + Positional Embeddings (src/model/embeddings.py)
• Decoder Block ×6 (src/model/attention.py & transformer.py)
• LM Head (linear projection in transformer.py)
4. Training Pipeline
• Config Loader (src/train/config.py)
• Trainer (src/train/trainer.py)
• Training loop orchestration (train.py)
• Checkpointing & Metrics output (evaluation_metrics.json)
5. Evaluation
• evaluate.py
• test suite (tests/*.py)
6. Deployment
• Gradio App (deploy/app.py)
• Export logic (src/utils/export.py)
b. Relationships & Data Flow
1. Raw text → Tokenizer → Token IDs
2. Token IDs + Positions → Embedding layer → Tensor [seq_len×embed_dim]
3. Embedded tensor → Decoder stack → Final hidden states
4. Hidden states → LM head → Probability distribution → Next-token sampling
5. Trainer orchestrates 1–4 in batches, updates model via PyTorch optimizers
6. After training, model + tokenizer exported → Gradio app loads them → serves inference requests
c. Architectural Patterns & Principles
• Modular layering: clear separation of data ingestion, modeling, training, and deployment
• Encoder-only style (decoder-only Transformer)
• Configuration-driven experiments (YAML under configs/)
• Test-driven: unit tests for each core component
• Reusable utilities: debug and export helpers
d. Key Technologies
• PyTorch (model, training loop)
• YAML (experiment/config management)
• Gradio (deployment UI)
• Python packaging & scripts (train.py, evaluate.py)