# aabithgwc/karnataka-project- architecture

> Architecture diagram of the GitHub repository aabithgwc/karnataka-project-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aabithgwc/karnataka-project-
- Repository: https://github.com/aabithgwc/karnataka-project-

## Overview

#### Purpose and entry points
A Python batch pipeline analyzes public YouTube discussion across configured Karnataka regional channels: extract videos/comments, remove non-speech comments, then use Anthropic Claude for translation, sentiment, governance recommendations, and themes.

- Full orchestrator: `main.py`
- Recent-window extraction runner: `extract_last_24h.py`
- Channel scope/category input: `channels_config.json`
- Environment/API configuration: `.env.example`

#### Pipeline orchestration
`main.py` is the control boundary for end-to-end execution. It accepts extraction window, video cap, sampling size, API keys, and an extractor-skip option, then coordinates the three stage entry points. Each stage persists artifacts to its own module-local `output/` directory, allowing later stages or reruns to consume saved data.

Paths: `main.py`, `youtube_data_extractor/app.py`, `data_cleaning/app.py`

#### Stage 1: YouTube multi-channel ingestion
The extractor reads configured channels, resolves each channel’s uploads playlist and metadata, enumerates videos and statistics, and retrieves comment threads/replies. Its YouTube API wrapper centralizes API v3 access and retry behavior; services separate channel, video, and comment collection concerns.

Primary raw boundary: YouTube Data API v3 → structured JSON/CSV datasets.

Paths: `youtube_data_extractor/youtube_client.py`, `youtube_data_extractor/channel_service.py`, `youtube_data_extractor/comments_service.py`

#### Extraction datasets and time filtering
Extraction writes channel, video, and comment datasets, including recent-window variants. The standalone recent extractor and pipeline `--hours` flow support bounded collection rather than requiring a full historical run. Downstream cleaning is driven from persisted comment data, not direct API calls.

Paths: `extract_last_24h.py`, `youtube_data_extractor/output/comments.json`, `youtube_data_extractor/output/comments_last_24h.json`

#### Stage 2: comment quality filtering
Cleaning loads raw comments and filters emoji-only/non-speech text, preserving both accepted comments and rejected records as separate artifacts. This is the boundary that reduces noisy social content before expensive LLM processing.

Paths: `data_cleaning/emoji_filter.py`, `data_cleaning/output/cleaned_comments.json`, `data_cleaning/output/removed_emoji_only_comments.json`

#### Stage 3: Claude analysis and context enrichment
The sentiment engine samples videos/comments, optionally obtains video subtitles/transcripts for context, and sends Karnataka-tuned prompts with structured result expectations through an Anthropic client. It produces translated English text, sentiment and confidence, reasoning, recommended governance action, thematic tags, and video summaries.

Primary external boundary: cleaned local data/transcripts → Anthropic Claude API → structured analysis results.

Paths: `sentiment_analysis/sampler.py`, `sentiment_analysis/transcript_fetcher.py`, `sentiment_analysis/sentiment_service.py`

#### Cost, outputs, and additional services
Claude usage is tracked per batch with token/cost logging, making LLM spend an explicit operational concern. Final analytical artifacts and summaries remain filesystem-based. Root-level services and output files indicate an adjacent events/festivals integration and Supabase persistence capability, though they are outside the README’s three-stage core flow.

Paths: `sentiment_analysis/usage_tracker.py`, `sentiment_analysis/output/karnataka_sentiment_results.json`, `supabase_service.py`

Main entry points: [`sentiment_analysis/sentiment_service.py`](https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/sentiment_service.py) (Sentiment and governance analysis), [`youtube_data_extractor/youtube_client.py`](https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/youtube_client.py) (YouTube API client), [`data_cleaning/emoji_filter.py`](https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/emoji_filter.py) (Speech quality filter), [`main.py`](https://github.com/aabithgwc/karnataka-project-/blob/main/main.py) (Pipeline orchestrator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_main["Pipeline orchestrator<br/>Python CLI<br/>[main.py]"]
  node_recent_runner["Recent-window runner<br/>Python CLI"]
  node_channel_config["Channel scope<br/>JSON config"]
end

subgraph group_ingestion["YouTube ingestion"]
  node_youtube_api{{"YouTube Data API v3<br/>external API"}}
  node_youtube_client["YouTube API client<br/>API wrapper<br/>[youtube_client.py]"]
  node_channel_service["Channel and video collection<br/>collection service<br/>[channel_service.py]"]
  node_comments_service["Comment collection<br/>collection service"]
  node_raw_comments["Raw comments<br/>JSON dataset<br/>[comments.json]"]
  node_recent_comments["Recent comments<br/>JSON dataset"]
end

subgraph group_cleaning["Comment cleaning"]
  node_emoji_filter["Speech quality filter<br/>filter service<br/>[emoji_filter.py]"]
  node_cleaned_comments["Cleaned comments<br/>JSON dataset"]
  node_rejected_comments["Rejected comments<br/>JSON dataset"]
end

subgraph group_analysis["Claude analysis"]
  node_sampler["Video/comment sampler<br/>sampling service<br/>[sampler.py]"]
  node_transcript_fetcher["Transcript context<br/>context service"]
  node_claude_api{{"Anthropic Claude API<br/>external LLM API"}}
  node_sentiment_service["Sentiment and governance analysis<br/>LLM analysis service"]
  node_usage_tracker["LLM usage tracking<br/>cost tracker<br/>[usage_tracker.py]"]
  node_analysis_results["Analysis results<br/>JSON dataset"]
end

node_supabase[("Supabase persistence<br/>optional persistence service")]

node_main -->|"reads scope"| node_channel_config
node_main -->|"runs extraction"| node_youtube_client
node_main -->|"runs cleaning"| node_emoji_filter
node_main -->|"runs analysis"| node_sentiment_service
node_recent_runner -->|"runs bounded extraction"| node_youtube_client
node_youtube_client -->|"API requests"| node_youtube_api
node_youtube_client -->|"supports collection"| node_channel_service
node_youtube_client -->|"supports collection"| node_comments_service
node_channel_config -->|"configured channels"| node_channel_service
node_channel_service -->|"videos"| node_comments_service
node_comments_service -->|"writes"| node_raw_comments
node_comments_service -->|"writes bounded set"| node_recent_comments
node_raw_comments -->|"loads"| node_emoji_filter
node_recent_comments -->|"loads recent set"| node_emoji_filter
node_emoji_filter -->|"writes accepted"| node_cleaned_comments
node_emoji_filter -->|"writes rejected"| node_rejected_comments
node_cleaned_comments -->|"analysis input"| node_sampler
node_sampler -->|"sampled comments"| node_sentiment_service
node_transcript_fetcher -->|"optional context"| node_sentiment_service
node_sentiment_service -->|"structured prompts"| node_claude_api
node_sentiment_service -->|"token usage"| node_usage_tracker
node_sentiment_service -->|"writes results"| node_analysis_results
node_analysis_results -.->|"optional persistence"| node_supabase

click node_main "https://github.com/aabithgwc/karnataka-project-/blob/main/main.py"
click node_recent_runner "https://github.com/aabithgwc/karnataka-project-/blob/main/extract_last_24h.py"
click node_channel_config "https://github.com/aabithgwc/karnataka-project-/blob/main/channels_config.json"
click node_youtube_client "https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/youtube_client.py"
click node_channel_service "https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/channel_service.py"
click node_comments_service "https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/comments_service.py"
click node_raw_comments "https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/output/comments.json"
click node_recent_comments "https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/output/comments_last_24h.json"
click node_emoji_filter "https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/emoji_filter.py"
click node_cleaned_comments "https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/output/cleaned_comments.json"
click node_rejected_comments "https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/output/removed_emoji_only_comments.json"
click node_sampler "https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/sampler.py"
click node_transcript_fetcher "https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/transcript_fetcher.py"
click node_sentiment_service "https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/sentiment_service.py"
click node_usage_tracker "https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/usage_tracker.py"
click node_analysis_results "https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/output/karnataka_sentiment_results.json"
click node_supabase "https://github.com/aabithgwc/karnataka-project-/blob/main/supabase_service.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_recent_runner,node_channel_config toneBlue
class node_youtube_api,node_youtube_client,node_channel_service,node_comments_service,node_raw_comments,node_recent_comments toneAmber
class node_emoji_filter,node_cleaned_comments,node_rejected_comments toneMint
class node_sampler,node_transcript_fetcher,node_claude_api,node_sentiment_service,node_usage_tracker,node_analysis_results toneRose
class node_supabase toneNeutral
```

## Components

### Orchestration

- [Pipeline orchestrator](https://github.com/aabithgwc/karnataka-project-/blob/main/main.py): Python CLI · Coordinates extraction, cleaning, and analysis with run options. · `main.py`
- [Recent-window runner](https://github.com/aabithgwc/karnataka-project-/blob/main/extract_last_24h.py): Python CLI · Runs bounded recent comment extraction. · `extract_last_24h.py`
- [Channel scope](https://github.com/aabithgwc/karnataka-project-/blob/main/channels_config.json): JSON config · `channels_config.json`

### YouTube ingestion

- **YouTube Data API v3**: external API · Channel, video, statistics, and comment-thread source.
- [YouTube API client](https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/youtube_client.py): API wrapper · Centralizes API v3 access and retries. · `youtube_data_extractor/youtube_client.py`
- [Channel and video collection](https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/channel_service.py): collection service · Resolves uploads playlists, metadata, and videos. · `youtube_data_extractor/channel_service.py`
- [Comment collection](https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/comments_service.py): collection service · Retrieves comment threads and replies. · `youtube_data_extractor/comments_service.py`
- [Raw comments](https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/output/comments.json): JSON dataset · Persisted full extraction comments. · `youtube_data_extractor/output/comments.json`
- [Recent comments](https://github.com/aabithgwc/karnataka-project-/blob/main/youtube_data_extractor/output/comments_last_24h.json): JSON dataset · Persisted time-bounded comments. · `youtube_data_extractor/output/comments_last_24h.json`

### Comment cleaning

- [Speech quality filter](https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/emoji_filter.py): filter service · Separates speech comments from emoji-only or non-speech content. · `data_cleaning/emoji_filter.py`
- [Cleaned comments](https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/output/cleaned_comments.json): JSON dataset · Accepted input for LLM processing. · `data_cleaning/output/cleaned_comments.json`
- [Rejected comments](https://github.com/aabithgwc/karnataka-project-/blob/main/data_cleaning/output/removed_emoji_only_comments.json): JSON dataset · Audit artifact for removed emoji-only comments. · `data_cleaning/output/removed_emoji_only_comments.json`

### Claude analysis

- [Video/comment sampler](https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/sampler.py): sampling service · Selects analysis input from cleaned persisted data. · `sentiment_analysis/sampler.py`
- [Transcript context](https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/transcript_fetcher.py): context service · Optionally obtains video subtitles and transcripts. · `sentiment_analysis/transcript_fetcher.py`
- **Anthropic Claude API**: external LLM API · Returns structured translation and governance analysis.
- [Sentiment and governance analysis](https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/sentiment_service.py): LLM analysis service · Builds Karnataka-tuned prompts and structures results. · `sentiment_analysis/sentiment_service.py`
- [LLM usage tracking](https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/usage_tracker.py): cost tracker · Logs batch tokens and Claude costs. · `sentiment_analysis/usage_tracker.py`
- [Analysis results](https://github.com/aabithgwc/karnataka-project-/blob/main/sentiment_analysis/output/karnataka_sentiment_results.json): JSON dataset · Translations, sentiment, actions, themes, and summaries. · `sentiment_analysis/output/karnataka_sentiment_results.json`

### Other components

- [Supabase persistence](https://github.com/aabithgwc/karnataka-project-/blob/main/supabase_service.py): optional persistence service · Adjacent capability outside the core three-stage flow. · `supabase_service.py`

## Connections

- Pipeline orchestrator → Channel scope: reads scope
- Pipeline orchestrator → YouTube API client: runs extraction
- Pipeline orchestrator → Speech quality filter: runs cleaning
- Pipeline orchestrator → Sentiment and governance analysis: runs analysis
- Recent-window runner → YouTube API client: runs bounded extraction
- YouTube API client → YouTube Data API v3: API requests
- YouTube API client → Channel and video collection: supports collection
- YouTube API client → Comment collection: supports collection
- Channel scope → Channel and video collection: configured channels
- Channel and video collection → Comment collection: videos
- Comment collection → Raw comments: writes
- Comment collection → Recent comments: writes bounded set
- Raw comments → Speech quality filter: loads
- Recent comments → Speech quality filter: loads recent set
- Speech quality filter → Cleaned comments: writes accepted
- Speech quality filter → Rejected comments: writes rejected
- Cleaned comments → Video/comment sampler: analysis input
- Video/comment sampler → Sentiment and governance analysis: sampled comments
- Transcript context → Sentiment and governance analysis: optional context
- Sentiment and governance analysis → Anthropic Claude API: structured prompts
- Sentiment and governance analysis → LLM usage tracking: token usage
- Sentiment and governance analysis → Analysis results: writes results
- Analysis results → Supabase persistence: optional persistence