How julian-reed/cs224n_final works
Main Components
• Data Management:
• Data Directory (data/): Contains training, development, and test datasets including sonnets and CSV files.
• datasets.py: Likely responsible for data loading, preprocessing, and possibly splitting data for training and evaluation.
• Model Definition & Architecture:
• models/ directory: Contains base_gpt.py and gpt2.py for setting up the GPT-2 model.
• modules/ directory: Contains components like attention.py and gpt2_layer.py that implement key architectural elements (e.g., attention mechanism and transformer layers) of GPT-2.
• Configuration and Utility:
• config.py: Holds configuration parameters that likely control aspects like model hyperparameters or training-setup.
• utils.py: Provides helper functions used throughout the codebase.
• Optimization Techniques:
• optimizer.py and smart_optimizer.py: Implement standard and “smart” optimizer variants.
• SMART, LoRA, and SOAP adjustments are integrated through these optimization files, helping to regulate the training with different regularization techniques.
• Task-Specific Modules:
• classifier.py and smart_classifier.py: Code for classification tasks built on top of the model.
• paraphrase_detection.py and smart_paraphrase_detection.py: Modules devoted to detecting paraphrases.
• sonnet_generation.py, smart_sonnet_generation.py: Responsible for generating sonnets and related output.
• Evaluation & Prediction:
• evaluation.py: Script to assess model performance.
• predictions/ directory: Contains output files from different experiments including generated sonnets and classification/prediction results.
• prepare_submit.py: Likely packages the results for submission.
• Environment & Setup:
• env.yml, setup.sh: Define the project environment and setup instructions.
• README.md and Final Report.pdf: Provide project context, documentation, and high-level overviews that are useful for understanding overall design.
Relationships and Interactions
• Data flows from the “data” directory into the data management module (datasets.py) to prepare inputs for training.
• The model components (models/ and modules/) use configurations from config.py and utilities from utils.py to initialize and compute the GPT-2 architecture.
• Optimization modules (optimizer.py, smart_optimizer.py, etc.) inject techniques (LoRA, SMART, SOAP) into the training process managed by the main model scripts.
• Task-specific scripts (such as classifier.py, paraphrase_detection.py, and sonnet_generation.py) build upon the optimized model to perform tasks like classification, paraphrase detection, or sonnet generation.
• Outputs flow to the predictions/ directory and are further used by evaluation.py to assess the system's overall performance.
Architectural Patterns
• Modularity: The separation between models, modules, optimization, and task-specific code indicates a modular design. Each component (data processing, model building, optimization, evaluation) operates independently but communicates through defined interfaces.
• Separation of Concerns: There is a clear division between the core model architecture (in models/ and modules/), the optimization techniques, and the task-specific implementations.
• Reusability: Utility scripts and configuration files are shared across different parts of the system ensuring consistency.