# bharaniabhishek123/final_project_cs229 architecture

> Architecture diagram of the GitHub repository bharaniabhishek123/final_project_cs229, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bharaniabhishek123/final_project_cs229
- Repository: https://github.com/bharaniabhishek123/final_project_cs229

## Overview

##### File structure
- Top-level directories:
- data/ – raw and cleaned CSV datasets for US and GB videos and comments
- youtube_sentiment/ – Python modules for each pipeline stage (preprocess.py, sentiment.py, precision_recall.py, etc.)
- proposal/, progress/, final/, poster/ – LaTeX and PPT artifacts documenting the research progress and results
- Configuration files: LICENSE, README.md, .idea/
- No explicit microservice or web layer—this is a batch data pipeline.

##### README for additional insights
- The README only confirms that it’s the Fall 2017 CS229 project. There is no pre-drawn architecture, so we must infer the pipeline structure from folder names and script names.

a. Main components
1. Raw Data Storage
- Raw CSVs and ZIPs under data/youtube/ (GBComments-train.csv, USvideos_clean_train.csv, etc.)
2. Data Ingestion & Cleaning
- youtube_sentiment/preprocess.py reads raw CSVs, handles missing values, text normalization, splits train/dev/test, writes cleaned CSVs back to data/youtube/*.
3. Feature Extraction & Tagging
- youtube_sentiment/tags.py and category.py extract metadata features (video category, tags)
- youtube_sentiment/generateWordCloud.py creates visual word clouds for exploratory analysis
4. Model Training & Sentiment Classification
- youtube_sentiment/sentiment.py encapsulates model training (e.g., TF-IDF or bag-of-words → classifier), cross-validation
5. Evaluation & Metrics
- youtube_sentiment/precision_recall.py calculates accuracy, precision/recall, generates performance tables
6. Reporting & Visualization
- Final LaTeX pipeline under final/ generates figures, tables, writes final.tex
- poster/ holds PPT slides
- proposal/ and progress/ folders track earlier drafts
b. Relationships and interactions
- Raw Data → Data Ingestion & Cleaning → Clean Data
- Clean Data → Feature Extraction → Feature Store
- Feature Store → Model Training → Trained Model
- Trained Model + Test Data → Evaluation → Metrics & Plots
- Metrics & Plots + Narrative → Reporting artifacts (LaTeX, PPT)
c. Architectural patterns and design principles
- Linear batch pipeline (ETL → ML training → evaluation → reporting)
- Modular script organization: each stage in a dedicated Python module
- Data persisted at each stage as CSV artifacts, enabling reproducibility
d. Key technologies/frameworks
- Python (pandas, scikit-learn, matplotlib/wordcloud)
- LaTeX (LLNCS template) for final write-up
- PPT (PowerPoint) for poster

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Storage
    RawData["Raw Data Storage\n(data/youtube/)"]:::storage
    %% Processing Modules
    Preprocessing["Data Ingestion & Cleaning Module\n(preprocess.py)"]:::process
    Tagging["Feature Extraction – Tagging Module\n(tags.py)"]:::process
    Category["Feature Extraction – Category Metadata Module\n(category.py)"]:::process
    WordCloud["Exploratory Visualization – Word Cloud Generator\n(generateWordCloud.py)"]:::process
    Sentiment["Model Training & Sentiment Classification Module\n(sentiment.py)"]:::process
    PrecisionRecall["Evaluation & Metrics Module\n(precision_recall.py)"]:::process
    %% Output Artifacts
    FinalReport["Final Report (LaTeX) Artifacts\n(final/)"]:::output
    Poster["Poster (Presentation) Artifacts\n(poster/)"]:::output
    Progress["Progress Report Drafts\n(progress/)"]:::output
    Proposal["Initial Proposal Artifacts\n(proposal/)"]:::output

    %% Data Flow
    RawData -->|"raw CSVs"| Preprocessing
    Preprocessing -->|"cleaned_CSV"| Tagging
    Preprocessing -->|"cleaned_CSV"| Category
    Preprocessing -->|"cleaned_CSV"| WordCloud
    Tagging -->|"tag_features"| Sentiment
    Category -->|"category_features"| Sentiment
    WordCloud -.->|exploratory_plots| Progress
    WordCloud -.->|exploratory_plots| Proposal
    Sentiment -->|"model.pkl"| PrecisionRecall
    PrecisionRecall -->|"metrics.csv"| FinalReport
    PrecisionRecall -->|"metrics.csv"| Poster
    %% Iterative Feedback
    Progress -.-> Preprocessing
    Proposal -.-> Preprocessing

    %% Click Events
    click RawData "https://github.com/bharaniabhishek123/final_project_cs229/tree/master/data/youtube/"
    click Preprocessing "https://github.com/bharaniabhishek123/final_project_cs229/blob/master/youtube_sentiment/preprocess.py"
    click Tagging "https://github.com/bharaniabhishek123/final_project_cs229/blob/master/youtube_sentiment/tags.py"
    click Category "https://github.com/bharaniabhishek123/final_project_cs229/blob/master/youtube_sentiment/category.py"
    click WordCloud "https://github.com/bharaniabhishek123/final_project_cs229/blob/master/youtube_sentiment/generateWordCloud.py"
    click Sentiment "https://github.com/bharaniabhishek123/final_project_cs229/blob/master/youtube_sentiment/sentiment.py"
    click PrecisionRecall "https://github.com/bharaniabhishek123/final_project_cs229/blob/master/youtube_sentiment/precision_recall.py"
    click FinalReport "https://github.com/bharaniabhishek123/final_project_cs229/tree/master/final/"
    click Poster "https://github.com/bharaniabhishek123/final_project_cs229/tree/master/poster/"
    click Progress "https://github.com/bharaniabhishek123/final_project_cs229/tree/master/progress/"
    click Proposal "https://github.com/bharaniabhishek123/final_project_cs229/tree/master/proposal/"

    %% Styles
    classDef storage fill:#AED6F1,stroke:#1B4F72,color:#1B2631;
    classDef process fill:#ABEBC6,stroke:#1D8348,color:#145A32;
    classDef output fill:#F9E79F,stroke:#B9770E,color:#7E5109;
```
