How nunoom/social-analytics-angola works
This repository implements a YouTube data-collection scraper: it searches topics, fetches video details and comments, tracks API quota, and stores collected records in PostgreSQL. Collection can run directly or on a six-hour schedule. The sampled source shows the scraper’s orchestration and database writes; the README is empty, so other product capabilities are not documented here.
Main entry points: youtube_scraper.py Collection orchestrator · quota_manager.py Quota manager · scheduler.py Scheduled collection · config.py Environment configuration
Components 12
YouTube collection
- Scheduled collection
scraper/scheduler.py - Collection orchestrator
scraper/youtube_scraper.py - Topic video search
scraper/youtube_scraper.py - Video details
scraper/youtube_scraper.py - Comment collection
scraper/youtube_scraper.py
Scheduling and quota
- Quota manager
scraper/quota_manager.py
Configuration and storage
- Environment configuration
scraper/config.py - Video persistence
scraper/youtube_scraper.py - Comment persistence
scraper/youtube_scraper.py
Other components
- Operator
- YouTube API
- PostgreSQL
Connections 21
- Operator → Collection orchestrator: starts directly
scraper/youtube_scraper.py - Operator → Scheduled collection: starts scheduler
scraper/scheduler.py - Scheduled collection → Collection orchestrator: runs collection
scraper/scheduler.py - Collection orchestrator → Topic video search: searches topics
scraper/youtube_scraper.py - Collection orchestrator → Video details: fetches details
scraper/youtube_scraper.py - Collection orchestrator → Comment collection: fetches comments
scraper/youtube_scraper.py - Topic video search → YouTube API: searches videos
scraper/youtube_scraper.py - Video details → YouTube API: requests details
scraper/youtube_scraper.py - Comment collection → YouTube API: requests comments
scraper/youtube_scraper.py - Collection orchestrator → Quota manager: checks status
scraper/youtube_scraper.py - Topic video search → Quota manager: checks and logs
scraper/youtube_scraper.py - Video details → Quota manager: checks and logs
scraper/youtube_scraper.py - Comment collection → Quota manager: checks and logs
scraper/youtube_scraper.py - Quota manager → PostgreSQL: reads and logs quota
scraper/quota_manager.py - Collection orchestrator → Video persistence: saves fetched videos
scraper/youtube_scraper.py - Collection orchestrator → Comment persistence: saves fetched comments
scraper/youtube_scraper.py - Video persistence → PostgreSQL: writes video records
scraper/youtube_scraper.py - Comment persistence → PostgreSQL: writes comment records
scraper/youtube_scraper.py - Collection orchestrator → Environment configuration: loads credentials
scraper/youtube_scraper.py - Quota manager → Environment configuration: loads database settings
scraper/quota_manager.py - Collection orchestrator → PostgreSQL: opens connection
scraper/youtube_scraper.py