# nabin2004/cs224_final_project_gpt2 architecture

> Architecture diagram of the GitHub repository nabin2004/cs224_final_project_gpt2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nabin2004/cs224_final_project_gpt2
- Repository: https://github.com/nabin2004/cs224_final_project_gpt2

## Overview

##### Project type and purpose
- This is a research/learning codebase implementing a GPT-2 language model from scratch (Part 1) and then applying it to two downstream tasks—paraphrase detection and sonnet generation (Part 2).
- It’s structured as a modular ML system: you build the model core, then drive it via training/evaluation scripts on datasets.

##### File structure
Top‐level components:
- data/ + datasets.py: raw text and CSV datasets, plus data‐loading utilities
- modules/: implementation of Transformer building blocks (attention, GPT2 layer)
- models/: high-level wrappers (BaseGPT, GPT2) that stack modules into a full network
- optimizer.py: custom optimization step
- classifier.py, paraphrase_detection.py, sonnet_generation.py: entry-point scripts for training/evaluating downstream tasks
- evaluation.py, sanity_check.py, optimizer_test.py: test and evaluation harnesses
- utils.py, config.py: shared helper functions and configuration
- predictions/: storage for output predictions
- prepare_submit.py, setup.sh, env.yml: deployment/setup artifacts

##### Key insights from the README
- Two phases: implementing missing GPT-2 components (modules, model, optimizer) and then using the trained model for paraphrase detection & sonnet generation.
- The project uses a pipeline style: code → sanity checks → training on tasks → generate/evaluate outputs.

##### Main components & their relationships
- Data Layer
– Raw inputs live under data/; datasets.py transforms them into batch iterators.
– The Data Layer feeds batched token indices to the Model Core.
- Model Core
– modules/attention.py + modules/gpt2_layer.py define the Transformer sub‐layers.
– models/gpt2.py (and base_gpt.py) assemble these sub‐layers into a full GPT-2 architecture.
- Training Engine
– optimizer.py applies gradient updates; optimizer_test.py validates it.
– classifier.py trains/fine-tunes for sentiment (Part 1 usage), paraphrase_detection.py and sonnet_generation.py drive Part 2 training and inference.
- Evaluation & Predictions
– evaluation.py computes metrics; paraphrase_detection.py and sonnet_generation.py write outputs into predictions/.
– sanity_check.py verifies model implementation before large runs.
- Utilities
– utils.py and config.py provide shared functions (e.g. tokenization, logging).
- Deployment/Setup
– setup.sh and env.yml configure the conda environment. prepare_submit.py bundles results.

##### Architectural patterns & design principles
- Modular layering: low-level modules → model assembly → training/inference scripts
- Pipeline flow: Data Loader → Model → Optimizer → Evaluation → Predictions
- Separation of concerns: implementations (modules, models) are decoupled from orchestration (scripts).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        DataLayer["Data Layer\ndata/"]:::data
        DataUtils["Data Loading Utilities\ndatasets.py"]:::data
    end

    %% Model Core
    subgraph "Model Core"
        direction TB
        Attention["Attention Module\n(modules/attention.py)"]:::model
        GPT2Layer["GPT2 Layer\n(modules/gpt2_layer.py)"]:::model
        BaseGPT["BaseGPT\n(models/base_gpt.py)"]:::model
        GPT2Model["GPT2 Model\n(models/gpt2.py)"]:::model
    end

    %% Training Engine
    subgraph "Training Engine"
        direction TB
        Optimizer["Optimizer Implementation\n(optimizer.py)"]:::train
        SanityCheck["Sanity Check\n(sanity_check.py)"]:::train
        OptimizerTest["Optimizer Test\n(optimizer_test.py)"]:::train
    end

    %% Training & Inference Scripts
    subgraph "Training & Inference Scripts"
        direction TB
        Classifier["Classifier Script\n(classifier.py)"]:::train
        Paraphrase["Paraphrase Detection\n(paraphrase_detection.py)"]:::train
        Sonnet["Sonnet Generation\n(sonnet_generation.py)"]:::train
    end

    %% Evaluation & Prediction
    subgraph "Evaluation & Prediction"
        direction TB
        Evaluation["Evaluation\n(evaluation.py)"]:::eval
        Predictions["Predictions Output\n(predictions/)"]:::eval
    end

    %% Utilities & Configuration
    subgraph "Utilities & Configuration"
        direction TB
        Utils["Utilities\n(utils.py)"]:::util
        Config["Configuration\n(config.py)"]:::util
    end

    %% Setup & Deployment
    subgraph "Setup & Deployment"
        direction TB
        SetupSh["Setup Script\n(setup.sh)"]:::setup
        Env["Environment File\n(env.yml)"]:::setup
        Prepare["Prepare Submit\n(prepare_submit.py)"]:::setup
    end

    %% Connections
    DataLayer --> DataUtils
    DataUtils -->|"batches of token IDs"| Attention

    Attention --> GPT2Layer
    GPT2Layer --> BaseGPT
    BaseGPT --> GPT2Model

    GPT2Model -->|"loss computation & grads"| Optimizer
    SanityCheck -->|"validates model implementation"| GPT2Model
    OptimizerTest -->|"validates optimizer"| Optimizer

    GPT2Model -->|"fine-tune/inference"| Classifier
    GPT2Model -->|"fine-tune/inference"| Paraphrase
    GPT2Model -->|"sequence generation"| Sonnet

    Classifier -->|"uses optimizer"| Optimizer
    Paraphrase -->|"uses optimizer"| Optimizer
    Sonnet -->|"uses optimizer"| Optimizer

    Classifier --> Evaluation
    Paraphrase --> Evaluation
    Sonnet --> Evaluation
    Evaluation -->|"writes outputs"| Predictions

    %% Utilities connections (dashed)
    Config -.-> DataLayer
    Config -.-> GPT2Model
    Config -.-> Classifier
    Config -.-> Evaluation
    Utils -.-> DataUtils
    Utils -.-> GPT2Model
    Utils -.-> Optimizer
    Utils -.-> Evaluation

    %% Click Events
    click DataLayer "https://github.com/nabin2004/cs224_final_project_gpt2/tree/main/data/"
    click DataUtils "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/datasets.py"
    click Attention "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/modules/attention.py"
    click GPT2Layer "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/modules/gpt2_layer.py"
    click BaseGPT "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/models/base_gpt.py"
    click GPT2Model "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/models/gpt2.py"
    click Optimizer "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/optimizer.py"
    click SanityCheck "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/sanity_check.py"
    click OptimizerTest "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/optimizer_test.py"
    click Classifier "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/classifier.py"
    click Paraphrase "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/paraphrase_detection.py"
    click Sonnet "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/sonnet_generation.py"
    click Evaluation "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/evaluation.py"
    click Predictions "https://github.com/nabin2004/cs224_final_project_gpt2/tree/main/predictions/"
    click Utils "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/utils.py"
    click Config "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/config.py"
    click SetupSh "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/setup.sh"
    click Env "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/env.yml"
    click Prepare "https://github.com/nabin2004/cs224_final_project_gpt2/blob/main/prepare_submit.py"

    %% Styles
    classDef data fill:#add8e6,stroke:#333,stroke-width:1px
    classDef model fill:#90ee90,stroke:#333,stroke-width:1px
    classDef train fill:#ffa500,stroke:#333,stroke-width:1px
    classDef eval fill:#dda0dd,stroke:#333,stroke-width:1px
    classDef util fill:#d3d3d3,stroke:#333,stroke-width:1px
    classDef setup fill:#d3d3d3,stroke:#333,stroke-width:1px
```
