How shopnobanerjee/youtube-sentiment-analyser works
Project type and purpose
• This is a full-stack, real-time sentiment analysis application for YouTube Live Chats.
• It consists of two main subsystems:
• A Python FastAPI backend that loads a custom-trained ML model (with DVC/MLflow for experiment tracking and model versioning).
• A TypeScript Chrome extension front end that injects scripts into YouTube Live pages, captures chat messages, and displays sentiment results in a popup/dashboard.
File structure
• Top-level:
• .dvc, dvc.yaml, params.yaml → DVC pipeline configuration for data ingestion, preprocessing, and model training.
• data/ → raw and processed datasets (managed by DVC).
• experiments/ → Jupyter notebooks, saved model artifacts (tokenizers, label encoders, best_model.pth) and logs.
• preprocessing_eda/ → exploratory data analysis and preprocessing notebooks.
• src/ → Python modules for data ingestion, preprocessing, model building/evaluation/registration.
• server/ → FastAPI app (main.py), tokenizer, dependencies (pyproject.toml, requirements.txt).
• chrome_extension/ → TypeScript code, manifest.json, HTML views for popup and welcome pages.
README for additional insights
• Confirms FastAPI backend, MLflow tracking server for model lifecycle, and TypeScript Chrome extension for the front end.
• Highlights real-time chat capture, API communication, interactive dashboard, and model confidence scores.
a. Main components to include:
1. Data Ingestion & Versioning Layer (DVC)
2. Preprocessing & EDA (preprocessing_eda notebooks)
3. Model Training & Experiment Tracking (src/model + experiments + MLflow Tracking Server)
4. Model Registry/Artifact Store (MLflow model registry, S3 or file store)
5. Backend API Service (server: FastAPI, tokenizer, model loader)
6. External Data Sources (YouTube Live Chat API)
7. Chrome Extension Front End (content script, background script, popup/dashboard)
8. End User Browser (YouTube live stream page with injected extension UI)
b. Relationships and interactions:
1. Data Flow: Raw Chat Data → DVC pipeline → Processed Data → Model Training → Model Artifacts → MLflow Registry
2. Training Cycle: experiments/*.ipynb invoke src/model modules → log metrics/artifacts to MLflow → DVC commits updated models/tokenizers.
3. Serving: FastAPI server on →8000 loads latest model from MLflow/DVC output → Exposes sentiment analysis REST API.
4. Runtime:
• Chrome extension content script listens for new chat messages on the YouTube Live page.
• For each message, extension sends an HTTP POST to FastAPI /predict endpoint.
• Server tokenizes & preprocesses message → runs inference → returns sentiment label + confidence.
• Extension receives response → updates popup/dashboard charts in real time.
c. Architectural patterns and design principles:
• Layered Architecture: Separate data pipeline, training, serving, and presentation layers.
• Modularization: src/data, src/model, server/ and chrome_extension/ are clearly separated.
• CI for ML (MLOps): DVC + MLflow for reproducible data and model versioning.
• Event-Driven UI: Extension content script triggers API calls on chat events.
• API-First Design: FastAPI provides REST endpoints that are consumed by the front end.
d. Key technologies/frameworks:
• DVC (data/version control), MLflow (experiment & model registry)
• FastAPI, Uvicorn (server)
• Python (pandas, scikit-learn/lightGBM) for data and model code
• TypeScript, Chrome Extension APIs for front end
• DVC pipeline YAML, params.yaml for reproducibility