How syed-muhammadtaha/gpt-2 works
This repository contains a GPT-2-style language-model pretraining implementation: FineWeb-Edu text is tokenized into binary shards, batches are loaded, and a transformer is trained with checkpointing and W&B logging. The sampled source does not show inference or evaluation. There is an entrypoint mismatch: the configured gpt-2 command calls src/gpt_2/__init__.py, which only prints a greeting; it does not launch main.py. The README is empty, so capabilities and wiring here are based on the sampled source.
Main entry points: main.py Training loop · preproc_fineweb.py FineWeb preprocessor · loader.py Batch loader · model.py GPT transformer
Components 11
Data preparation
- FineWeb preprocessor
src/gpt_2/scripts/preproc_fineweb.py
Training runtime
- Training loop
src/gpt_2/main.py - Batch loader
src/gpt_2/loader.py - Training checkpoint
Model
- GPT transformer
src/gpt_2/model.py
Other components
- Command user
- Package entrypoint
src/gpt_2/__init__.py - FineWeb-Edu
- GPT-2 tokenizer
- Token shards
- W&B tracking
Connections 10
- Command user → Package entrypoint: runs command
- FineWeb preprocessor → FineWeb-Edu: loads dataset
- FineWeb preprocessor → GPT-2 tokenizer: encodes text
- FineWeb preprocessor → Token shards: writes tokens
- Batch loader → Token shards: reads batches
- Training loop → Batch loader: requests batches
- Training loop → GPT transformer: computes logits
- Training loop → Training checkpoint: saves state
- Training checkpoint → Training loop: restores state
- Training loop → W&B tracking: logs metrics