How ghelanikirtan/transformer-using-pytorch-from-scratch works
Project type and purpose
• This is a research-style, end-to-end Transformer language model implementation in PyTorch (“decoder-only”, GPT-style).
• It provides scripts for data preprocessing, vocabulary building, training, validation (perplexity), and single-prompt or interactive text generation.
File structure
• Top-level files:
• config.yaml (hyperparameters, file paths)
• requirements.txt (Python dependencies)
• training.log (runtime logging)
• Data directories (mentioned in README):
• data/processed/ (preprocessed token files, vocab.pt)
• checkpoints/ (model checkpoints)
• src/ directory:
• preprocess_data.py (reads raw text → tokenizes → builds vocab.pt)
• dataset.py (loads processed tokens + vocab → PyTorch Dataset/ DataLoader)
• network.py (defines Transformer model: embeddings, multi-head attention, feed-forward, layernorm)
• train.py (training loop: forward, loss, backward, optimizer, checkpointing, log)
• validate.py (evaluates on held-out data to compute perplexity)
• predict.py (generates text from a single prompt)
• run_lm.py (interactive console loop for generation)
• utils.py (configuration loader, logger setup, any other helpers)
• constants.py (device selection, seed setting)
README for additional insights
• Confirms architecture (scaled dot-product attention, multi-head, positional encodings, etc.)
• Mentions goals (train on larger corpora, add sampling strategies).
Detailed steps for the principal software engineer
• Left lane: “Data & Config” (config.yaml, raw data, preprocess)
• Middle lane: “Core Pipeline” (dataset.py → network.py → train.py/validate.py)
• Right lane: “Persistence & Inference” (checkpoints/, predict.py, run_lm.py)
4. Flow: config → preprocessing → data store → dataset → model → training → checkpoints + logs → validation → inference modules.