How nabin2004/cs224_final_project_gpt2 works
Project type and purpose
• This is a research/learning codebase implementing a GPT-2 language model from scratch (Part 1) and then applying it to two downstream tasks—paraphrase detection and sonnet generation (Part 2).
• It’s structured as a modular ML system: you build the model core, then drive it via training/evaluation scripts on datasets.
File structure
Top‐level components:
• data/ + datasets.py: raw text and CSV datasets, plus data‐loading utilities
• modules/: implementation of Transformer building blocks (attention, GPT2 layer)
• models/: high-level wrappers (BaseGPT, GPT2) that stack modules into a full network
• optimizer.py: custom optimization step
• classifier.py, paraphrase_detection.py, sonnet_generation.py: entry-point scripts for training/evaluating downstream tasks
• evaluation.py, sanity_check.py, optimizer_test.py: test and evaluation harnesses
• utils.py, config.py: shared helper functions and configuration
• predictions/: storage for output predictions
• prepare_submit.py, setup.sh, env.yml: deployment/setup artifacts
Key insights from the README
• Two phases: implementing missing GPT-2 components (modules, model, optimizer) and then using the trained model for paraphrase detection & sonnet generation.
• The project uses a pipeline style: code → sanity checks → training on tasks → generate/evaluate outputs.
Main components & their relationships
• Data Layer
– Raw inputs live under data/; datasets.py transforms them into batch iterators.
– The Data Layer feeds batched token indices to the Model Core.
• Model Core
– modules/attention.py + modules/gpt2_layer.py define the Transformer sub‐layers.
– models/gpt2.py (and base_gpt.py) assemble these sub‐layers into a full GPT-2 architecture.
• Training Engine
– optimizer.py applies gradient updates; optimizer_test.py validates it.
– classifier.py trains/fine-tunes for sentiment (Part 1 usage), paraphrase_detection.py and sonnet_generation.py drive Part 2 training and inference.
• Evaluation & Predictions
– evaluation.py computes metrics; paraphrase_detection.py and sonnet_generation.py write outputs into predictions/.
– sanity_check.py verifies model implementation before large runs.
• Utilities
– utils.py and config.py provide shared functions (e.g. tokenization, logging).
• Deployment/Setup
– setup.sh and env.yml configure the conda environment. prepare_submit.py bundles results.
Architectural patterns & design principles
• Modular layering: low-level modules → model assembly → training/inference scripts
• Pipeline flow: Data Loader → Model → Optimizer → Evaluation → Predictions
• Separation of concerns: implementations (modules, models) are decoupled from orchestration (scripts).