rajcoder123-noob/transformer

How rajcoder123-noob/transformer works

MiniGPT is an educational language-modeling project with two runnable paths: a character-level Bigram baseline and a GPT-2-BPE decoder-only Transformer. The GPT path prepares a corpus, tokenizes and batches it, trains and evaluates the model, saves checkpoints, then generates text from a prompt. The README also documents OpenWebText ingestion and checkpointing. Source excerpts show the implementations and their calls; the documented corpus artifact is not present in the supplied tree, so it has no repository path.

Main entry points: gpt.py Training Loop · bigram.py Character Bigram · download_data.py Corpus Downloader · generated.txt Generated Text

Components 18

Corpus Preparation

Corpus Downloader download_data.py
Training Corpus

Bigram Baseline

Character Bigram bigram.py
Character Codec bigram.py

GPT Transformer

GPT-2 BPE gpt.py
Train Validation Split gpt.py
Batch Loader gpt.py
Masked Attention gpt.py
Feed Forward gpt.py
GPT Language Model gpt.py
Loss Evaluation gpt.py
Training Loop gpt.py

Model Outputs

Generation Prompt
Model Checkpoints gpt.py
Text Generation gpt.py
Generated Text generated.txt

Other components

User
OpenWebText

Connections 22

OpenWebText → Corpus Downloader: streams documents README.md
Corpus Downloader → Training Corpus: writes corpus README.md
Training Corpus → Character Bigram: loads text bigram.py
Training Corpus → GPT-2 BPE: loads text gpt.py
Character Bigram → Character Codec: encodes characters bigram.py
Character Bigram → Character Bigram: trains and samples bigram.py
User → Corpus Downloader: runs preparation README.md
User → Character Bigram: runs baseline README.md
User → Training Loop: starts training README.md
GPT-2 BPE → Train Validation Split: encodes tokens gpt.py
Train Validation Split → Batch Loader: provides splits gpt.py
Batch Loader → Training Loop: provides batches gpt.py
Training Loop → Loss Evaluation: evaluates loss gpt.py
Training Loop → GPT Language Model: trains model gpt.py
GPT Language Model → Masked Attention: uses attention gpt.py
GPT Language Model → Feed Forward: uses feed forward gpt.py
Training Loop → Model Checkpoints: saves weights gpt.py
User → Generation Prompt: supplies prompt README.md
Generation Prompt → Text Generation: starts generation README.md
Text Generation → GPT Language Model: samples next tokens gpt.py
Text Generation → GPT-2 BPE: decodes tokens gpt.py
Text Generation → Generated Text: writes output gpt.py

Drawn by GitDiagram on November 15, 2026 from the repository's file tree, README and source files. Read as Markdown ·