prabanjan-ux/pyspark-youtube-sentiment

Purpose and Entry Points

This is a local end-to-end YouTube comment analytics pipeline: ingest comments, enrich them with NLP and engagement features, persist CSV datasets, and explore results in Streamlit. Batch and scheduled execution start at main.py; interactive exploration starts with dashboard.py.

Orchestration and Scheduling

main.py coordinates the operational sequence: fetch comments for a supplied YouTube video URL, invoke Spark processing, and write analytics-ready output. It supports one-time execution and recurring polling via the schedule runtime. config.py centralizes fetch limits, intervals, update mode, and data locations.

Paths: main.py, config.py

External Ingestion Boundary

youtube_fetcher.py is the boundary to Google’s YouTube Data API v3. It authenticates using the API key supplied through .env, extracts comment identity, author, text, likes, and timestamps, then merges into the raw local dataset while removing duplicates. API quota/rate limits are an operational constraint.

Paths: youtube_fetcher.py, .env, data/raw_comments.csv

Raw-to-Processed Data Pipeline

data_processor.py runs the main transformation layer on PySpark. It loads the raw CSV and produces cleaned, enriched records, including timestamp dimensions, comment-length/word/emoji features, and engagement-derived measures. This is the primary distributed-compute boundary and requires a compatible Java runtime plus PySpark.

Paths: data_processor.py, data/raw_comments.csv, data/processed_sentiment.csv

Sentiment Analysis Layer

sentiment_analyzer.py encapsulates textual scoring. NLTK VADER provides social-text positive/neutral/negative and compound sentiment values; TextBlob supplements polarity and subjectivity. Sentiment labels use score thresholds and are intended to become columns consumed by downstream analytics. NLTK lexical resources are a runtime dependency.

Paths: sentiment_analyzer.py, data_processor.py

Storage and Dataset Contract

CSV files form the inter-stage contract rather than a database or message bus. Raw comments are append/merge-oriented ingestion state; processed_sentiment.csv is the dashboard-facing analytical dataset. models/ is reserved for future trained-model artifacts, not part of the current execution path.

Paths: data/raw_comments.csv, data/processed_sentiment.csv, models/.gitkeep

Visualization Consumer

dashboard.py is a separate Streamlit process that reads the processed CSV and exposes filters, charts, top-comment views, and CSV export. Plotly supplies visualizations. It consumes persisted data rather than calling the YouTube API or owning the Spark pipeline, so freshness depends on prior main.py runs.

Paths: dashboard.py, data/processed_sentiment.csv

Runtime and Infrastructure

The system is Python 3.11+ with local filesystem persistence. Material dependencies are PySpark 3.5.7/Java 8 or 11, NLTK VADER, TextBlob, Pandas, Streamlit, Plotly, Google API Client, dotenv, and Schedule. Logs have a designated local directory; no cloud deployment, database, Kafka stream, or container runtime is present.

Paths: requirements.txt, logs/.gitkeep, config.py