# shopnobanerjee/youtube-sentiment-analyser architecture

> Architecture diagram of the GitHub repository shopnobanerjee/youtube-sentiment-analyser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shopnobanerjee/youtube-sentiment-analyser
- Repository: https://github.com/shopnobanerjee/youtube-sentiment-analyser

## Overview

##### Project type and purpose
- This is a full-stack, real-time sentiment analysis application for YouTube Live Chats.
- It consists of two main subsystems:
- A Python FastAPI backend that loads a custom-trained ML model (with DVC/MLflow for experiment tracking and model versioning).
- A TypeScript Chrome extension front end that injects scripts into YouTube Live pages, captures chat messages, and displays sentiment results in a popup/dashboard.

##### File structure
- Top-level:
- .dvc, dvc.yaml, params.yaml → DVC pipeline configuration for data ingestion, preprocessing, and model training.
- data/ → raw and processed datasets (managed by DVC).
- experiments/ → Jupyter notebooks, saved model artifacts (tokenizers, label encoders, best_model.pth) and logs.
- preprocessing_eda/ → exploratory data analysis and preprocessing notebooks.
- src/ → Python modules for data ingestion, preprocessing, model building/evaluation/registration.
- server/ → FastAPI app (main.py), tokenizer, dependencies (pyproject.toml, requirements.txt).
- chrome_extension/ → TypeScript code, manifest.json, HTML views for popup and welcome pages.

##### README for additional insights
- Confirms FastAPI backend, MLflow tracking server for model lifecycle, and TypeScript Chrome extension for the front end.
- Highlights real-time chat capture, API communication, interactive dashboard, and model confidence scores.

a. Main components to include:
1. Data Ingestion & Versioning Layer (DVC)
2. Preprocessing & EDA (preprocessing_eda notebooks)
3. Model Training & Experiment Tracking (src/model + experiments + MLflow Tracking Server)
4. Model Registry/Artifact Store (MLflow model registry, S3 or file store)
5. Backend API Service (server: FastAPI, tokenizer, model loader)
6. External Data Sources (YouTube Live Chat API)
7. Chrome Extension Front End (content script, background script, popup/dashboard)
8. End User Browser (YouTube live stream page with injected extension UI)

b. Relationships and interactions:
1. **Data Flow**: Raw Chat Data → DVC pipeline → Processed Data → Model Training → Model Artifacts → MLflow Registry
2. **Training Cycle**: experiments/*.ipynb invoke src/model modules → log metrics/artifacts to MLflow → DVC commits updated models/tokenizers.
3. **Serving**: FastAPI server on →8000 loads latest model from MLflow/DVC output → Exposes sentiment analysis REST API.
4. **Runtime**:
- Chrome extension content script listens for new chat messages on the YouTube Live page.
- For each message, extension sends an HTTP POST to FastAPI `/predict` endpoint.
- Server tokenizes & preprocesses message → runs inference → returns sentiment label + confidence.
- Extension receives response → updates popup/dashboard charts in real time.

c. Architectural patterns and design principles:
- **Layered Architecture**: Separate data pipeline, training, serving, and presentation layers.
- **Modularization**: src/data, src/model, server/ and chrome_extension/ are clearly separated.
- **CI for ML (MLOps)**: DVC + MLflow for reproducible data and model versioning.
- **Event-Driven UI**: Extension content script triggers API calls on chat events.
- **API-First Design**: FastAPI provides REST endpoints that are consumed by the front end.

d. Key technologies/frameworks:
- DVC (data/version control), MLflow (experiment & model registry)
- FastAPI, Uvicorn (server)
- Python (pandas, scikit-learn/lightGBM) for data and model code
- TypeScript, Chrome Extension APIs for front end
- DVC pipeline YAML, params.yaml for reproducibility

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data & ML Pipeline Layer
    subgraph "Data & ML Pipeline" 
        direction TB
        DVC["DVC Pipeline (dvc.yaml, params.yaml)"]:::pipeline
        DVCLock["DVC Lock (dvc.lock, .dvc/)"]:::pipeline
        RawData["Raw Data (data/raw)"]:::pipeline
        PreprocessingEDA["Preprocessing & EDA Notebooks"]:::pipeline
        ModelTraining["Model Training & Experiments"]:::pipeline
        ModelRegistry["Model Registry / Artifacts"]:::pipeline
        Logs["Logs (model_building_errors.log, model_evaluation_errors.log, model_registration_errors.log)"]:::pipeline
    end

    %% Backend API Service
    subgraph "Backend API Service" 
        direction TB
        API["FastAPI Service (/predict)"]:::backend
        subgraph "Modules"
            direction TB
            Tokenizer["Tokenizer Module"]:::backend
            Preprocessor["Preprocessor Module"]:::backend
            Inference["Inference Module"]:::backend
        end
    end

    %% Chrome Extension Frontend
    subgraph "Chrome Extension Front End"
        direction TB
        Content["Content Script"]:::frontend
        Background["Background Script"]:::frontend
        Popup["Popup UI"]:::frontend
        Welcome["Welcome Page"]:::frontend
    end

    %% External Services
    YouTube["YouTube Live Chat API"]:::external
    MLflowUI["MLflow Tracking UI"]:::external
    Browser["End User Browser"]:::external

    %% Data & MLflow interactions
    RawData -->|"DVC ingest"| DVC
    DVC -->|"DVC preprocess"| PreprocessingEDA
    PreprocessingEDA -->|"Processed Data"| ModelTraining
    ModelTraining -->|"register"| ModelRegistry
    ModelTraining -->|"logs metrics"| MLflowUI
    ModelTraining --> Logs

    %% Backend loads model
    ModelRegistry -->|"load latest"| API
    MLflowUI -.->|visualize experiments| ModelTraining

    %% Runtime flow
    Browser -->|"runs extension"| Content
    Content -->|"injects UI"| Browser
    Browser -->|"chat events"| Content
    Content -->|"POST /predict"| API
    API -->|tokenize| Tokenizer
    Tokenizer -->|preprocess| Preprocessor
    Preprocessor -->|infer| Inference
    Inference -->|"sentiment+confidence"| API
    API -->|"response"| Content
    Content -->|"update UI"| Popup

    %% External data source
    YouTube -->|"live chat stream"| Browser

    %% Click Events
    click DVC "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/dvc.yaml"
    click DVCLock "https://github.com/shopnobanerjee/youtube-sentiment-analyser/tree/main/.dvc/"
    click PreprocessingEDA "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/preprocessing_eda/preprocessing.ipynb"
    click ModelTraining "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/src/model/model_building.py"
    click ModelTraining "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/src/model/model_evaluation.py"
    click ModelTraining "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/src/model/register_model.py"
    click ModelRegistry "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/experiments/best_model.pth"
    click ModelRegistry "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/experiments/tokenizer.pkl"
    click ModelRegistry "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/experiments/label_encoder.pkl"
    click API "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/server/main.py"
    click Tokenizer "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/server/tokenizer.py"
    click API "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/server/pyproject.toml"
    click API "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/server/requirements.txt"
    click Content "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/content.ts"
    click Background "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/background.ts"
    click Popup "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/popup.html"
    click Popup "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/popup.ts"
    click Welcome "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/welcome.html"
    click Welcome "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/welcome.ts"
    click API "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/server/.python-version"
    click Content "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/manifest.json"
    click Content "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/tsconfig.json"
    click Content "https://github.com/shopnobanerjee/youtube-sentiment-analyser/blob/main/chrome_extension/package.json"

    %% Styles
    classDef pipeline fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef backend fill:#d4edda,stroke:#333,stroke-width:1px
    classDef frontend fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef external fill:#f0f0f0,stroke:#666,stroke-width:1px,stroke-dasharray: 3 3
```
