aabithgwc/karnataka-project-

Purpose and entry points

A Python batch pipeline analyzes public YouTube discussion across configured Karnataka regional channels: extract videos/comments, remove non-speech comments, then use Anthropic Claude for translation, sentiment, governance recommendations, and themes.

• Full orchestrator: main.py

• Recent-window extraction runner: extract_last_24h.py

• Channel scope/category input: channels_config.json

• Environment/API configuration: .env.example

Pipeline orchestration

main.py is the control boundary for end-to-end execution. It accepts extraction window, video cap, sampling size, API keys, and an extractor-skip option, then coordinates the three stage entry points. Each stage persists artifacts to its own module-local output/ directory, allowing later stages or reruns to consume saved data.

Paths: main.py, youtube_data_extractor/app.py, data_cleaning/app.py

Stage 1: YouTube multi-channel ingestion

The extractor reads configured channels, resolves each channel’s uploads playlist and metadata, enumerates videos and statistics, and retrieves comment threads/replies. Its YouTube API wrapper centralizes API v3 access and retry behavior; services separate channel, video, and comment collection concerns.

Primary raw boundary: YouTube Data API v3 → structured JSON/CSV datasets.

Paths: youtube_data_extractor/youtube_client.py, youtube_data_extractor/channel_service.py, youtube_data_extractor/comments_service.py

Extraction datasets and time filtering

Extraction writes channel, video, and comment datasets, including recent-window variants. The standalone recent extractor and pipeline --hours flow support bounded collection rather than requiring a full historical run. Downstream cleaning is driven from persisted comment data, not direct API calls.

Paths: extract_last_24h.py, youtube_data_extractor/output/comments.json, youtube_data_extractor/output/comments_last_24h.json

Stage 2: comment quality filtering

Cleaning loads raw comments and filters emoji-only/non-speech text, preserving both accepted comments and rejected records as separate artifacts. This is the boundary that reduces noisy social content before expensive LLM processing.

Paths: data_cleaning/emoji_filter.py, data_cleaning/output/cleaned_comments.json, data_cleaning/output/removed_emoji_only_comments.json

Stage 3: Claude analysis and context enrichment

The sentiment engine samples videos/comments, optionally obtains video subtitles/transcripts for context, and sends Karnataka-tuned prompts with structured result expectations through an Anthropic client. It produces translated English text, sentiment and confidence, reasoning, recommended governance action, thematic tags, and video summaries.

Primary external boundary: cleaned local data/transcripts → Anthropic Claude API → structured analysis results.

Paths: sentiment_analysis/sampler.py, sentiment_analysis/transcript_fetcher.py, sentiment_analysis/sentiment_service.py

Cost, outputs, and additional services

Claude usage is tracked per batch with token/cost logging, making LLM spend an explicit operational concern. Final analytical artifacts and summaries remain filesystem-based. Root-level services and output files indicate an adjacent events/festivals integration and Supabase persistence capability, though they are outside the README’s three-stage core flow.

Paths: sentiment_analysis/usage_tracker.py, sentiment_analysis/output/karnataka_sentiment_results.json, supabase_service.py