# prabanjan-ux/pyspark-youtube-sentiment architecture

> Architecture diagram of the GitHub repository prabanjan-ux/pyspark-youtube-sentiment, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prabanjan-ux/pyspark-youtube-sentiment
- Repository: https://github.com/prabanjan-ux/pyspark-youtube-sentiment

## Overview

#### Purpose and Entry Points
This is a local end-to-end YouTube comment analytics pipeline: ingest comments, enrich them with NLP and engagement features, persist CSV datasets, and explore results in Streamlit. Batch and scheduled execution start at `main.py`; interactive exploration starts with `dashboard.py`.

#### Orchestration and Scheduling
`main.py` coordinates the operational sequence: fetch comments for a supplied YouTube video URL, invoke Spark processing, and write analytics-ready output. It supports one-time execution and recurring polling via the `schedule` runtime. `config.py` centralizes fetch limits, intervals, update mode, and data locations.

Paths: `main.py`, `config.py`

#### External Ingestion Boundary
`youtube_fetcher.py` is the boundary to Google’s YouTube Data API v3. It authenticates using the API key supplied through `.env`, extracts comment identity, author, text, likes, and timestamps, then merges into the raw local dataset while removing duplicates. API quota/rate limits are an operational constraint.

Paths: `youtube_fetcher.py`, `.env`, `data/raw_comments.csv`

#### Raw-to-Processed Data Pipeline
`data_processor.py` runs the main transformation layer on PySpark. It loads the raw CSV and produces cleaned, enriched records, including timestamp dimensions, comment-length/word/emoji features, and engagement-derived measures. This is the primary distributed-compute boundary and requires a compatible Java runtime plus PySpark.

Paths: `data_processor.py`, `data/raw_comments.csv`, `data/processed_sentiment.csv`

#### Sentiment Analysis Layer
`sentiment_analyzer.py` encapsulates textual scoring. NLTK VADER provides social-text positive/neutral/negative and compound sentiment values; TextBlob supplements polarity and subjectivity. Sentiment labels use score thresholds and are intended to become columns consumed by downstream analytics. NLTK lexical resources are a runtime dependency.

Paths: `sentiment_analyzer.py`, `data_processor.py`

#### Storage and Dataset Contract
CSV files form the inter-stage contract rather than a database or message bus. Raw comments are append/merge-oriented ingestion state; `processed_sentiment.csv` is the dashboard-facing analytical dataset. `models/` is reserved for future trained-model artifacts, not part of the current execution path.

Paths: `data/raw_comments.csv`, `data/processed_sentiment.csv`, `models/.gitkeep`

#### Visualization Consumer
`dashboard.py` is a separate Streamlit process that reads the processed CSV and exposes filters, charts, top-comment views, and CSV export. Plotly supplies visualizations. It consumes persisted data rather than calling the YouTube API or owning the Spark pipeline, so freshness depends on prior `main.py` runs.

Paths: `dashboard.py`, `data/processed_sentiment.csv`

#### Runtime and Infrastructure
The system is Python 3.11+ with local filesystem persistence. Material dependencies are PySpark 3.5.7/Java 8 or 11, NLTK VADER, TextBlob, Pandas, Streamlit, Plotly, Google API Client, dotenv, and Schedule. Logs have a designated local directory; no cloud deployment, database, Kafka stream, or container runtime is present.

Paths: `requirements.txt`, `logs/.gitkeep`, `config.py`

Main entry points: [`main.py`](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/main.py) (Batch scheduler), [`data_processor.py`](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data_processor.py) (Spark processor), [`youtube_fetcher.py`](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/youtube_fetcher.py) (Comment fetcher), [`requirements.txt`](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/requirements.txt) (Dependencies)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ops["Operations"]
  node_main["Batch scheduler<br/>Python entry point<br/>[main.py]"]
  node_config["Runtime config<br/>Configuration module<br/>[config.py]"]
  node_requirements["Dependencies<br/>Python environment<br/>[requirements.txt]"]
  node_logs["Local logs<br/>Log directory<br/>[.gitkeep]"]
end

subgraph group_pipeline["Data Pipeline"]
  node_youtube_api{{"YouTube Data API v3<br/>External API"}}
  node_env["API credentials<br/>Environment file<br/>[.env]"]
  node_fetcher["Comment fetcher<br/>Ingestion boundary<br/>[youtube_fetcher.py]"]
  node_raw_csv[("Raw comments<br/>CSV dataset<br/>[raw_comments.csv]")]
  node_processor["Spark processor<br/>PySpark transformation<br/>[data_processor.py]"]
  node_sentiment["Sentiment analyzer<br/>NLP scoring module"]
  node_processed_csv[("Processed analytics<br/>CSV dataset")]
end

subgraph group_explore["Exploration"]
  node_dashboard["Streamlit dashboard<br/>Interactive entry point<br/>[dashboard.py]"]
  node_plotly{{"Plotly charts<br/>Visualization library"}}
end

node_config -->|"controls runs"| node_main
node_requirements -.->|"runtime packages"| node_main
node_main -->|"triggers fetch"| node_fetcher
node_main -->|"triggers processing"| node_processor
node_main -->|"writes logs"| node_logs
node_env -.->|"API key"| node_fetcher
node_fetcher -->|"requests comments"| node_youtube_api
node_fetcher -->|"merge and deduplicate"| node_raw_csv
node_raw_csv -->|"loads CSV"| node_processor
node_sentiment -->|"scores text"| node_processor
node_processor -->|"writes enriched CSV"| node_processed_csv
node_processed_csv -->|"reads persisted data"| node_dashboard
node_plotly -.->|"renders charts"| node_dashboard
node_requirements -.->|"PySpark and NLP runtime"| node_processor

click node_main "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/main.py"
click node_config "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/config.py"
click node_requirements "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/requirements.txt"
click node_logs "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/logs/.gitkeep"
click node_env "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/.env"
click node_fetcher "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/youtube_fetcher.py"
click node_raw_csv "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data/raw_comments.csv"
click node_processor "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data_processor.py"
click node_sentiment "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/sentiment_analyzer.py"
click node_processed_csv "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data/processed_sentiment.csv"
click node_dashboard "https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/dashboard.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_config,node_requirements,node_logs toneBlue
class node_youtube_api,node_env,node_fetcher,node_raw_csv,node_processor,node_sentiment,node_processed_csv toneAmber
class node_dashboard,node_plotly toneMint
```

## Components

### Operations

- [Batch scheduler](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/main.py): Python entry point · Runs one-time or recurring ingestion and processing. · `main.py`
- [Runtime config](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/config.py): Configuration module · Defines limits, intervals, modes, and data locations. · `config.py`
- [Dependencies](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/requirements.txt): Python environment · Provides PySpark, NLP, API, scheduling, and UI libraries. · `requirements.txt`
- [Local logs](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/logs/.gitkeep): Log directory · `logs/.gitkeep`

### Data Pipeline

- **YouTube Data API v3**: External API · Quota-limited source of video comments.
- [API credentials](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/.env): Environment file · Supplies the YouTube API key. · `.env`
- [Comment fetcher](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/youtube_fetcher.py): Ingestion boundary · Fetches, normalizes, merges, and deduplicates comments. · `youtube_fetcher.py`
- [Raw comments](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data/raw_comments.csv): CSV dataset · Append and merge-oriented ingestion state. · `data/raw_comments.csv`
- [Spark processor](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data_processor.py): PySpark transformation · Cleans comments and derives time, text, emoji, and engagement features. · `data_processor.py`
- [Sentiment analyzer](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/sentiment_analyzer.py): NLP scoring module · Uses VADER and TextBlob to produce sentiment measures and labels. · `sentiment_analyzer.py`
- [Processed analytics](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/data/processed_sentiment.csv): CSV dataset · Dashboard-facing enriched sentiment dataset. · `data/processed_sentiment.csv`

### Exploration

- [Streamlit dashboard](https://github.com/prabanjan-ux/pyspark-youtube-sentiment/blob/main/dashboard.py): Interactive entry point · Filters, charts, top comments, and CSV export. · `dashboard.py`
- **Plotly charts**: Visualization library

## Connections

- Runtime config → Batch scheduler: controls runs
- Dependencies → Batch scheduler: runtime packages
- Batch scheduler → Comment fetcher: triggers fetch
- Batch scheduler → Spark processor: triggers processing
- Batch scheduler → Local logs: writes logs
- API credentials → Comment fetcher: API key
- Comment fetcher → YouTube Data API v3: requests comments
- Comment fetcher → Raw comments: merge and deduplicate
- Raw comments → Spark processor: loads CSV
- Sentiment analyzer → Spark processor: scores text
- Spark processor → Processed analytics: writes enriched CSV
- Processed analytics → Streamlit dashboard: reads persisted data
- Plotly charts → Streamlit dashboard: renders charts
- Dependencies → Spark processor: PySpark and NLP runtime