# nunoom/social-analytics-angola architecture

> Architecture diagram of the GitHub repository nunoom/social-analytics-angola, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nunoom/social-analytics-angola
- Repository: https://github.com/nunoom/social-analytics-angola

## Overview

This repository implements a YouTube data-collection scraper: it searches topics, fetches video details and comments, tracks API quota, and stores collected records in PostgreSQL. Collection can run directly or on a six-hour schedule. The sampled source shows the scraper’s orchestration and database writes; the README is empty, so other product capabilities are not documented here.

Main entry points: [`scraper/youtube_scraper.py`](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py) (Collection orchestrator), [`scraper/quota_manager.py`](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/quota_manager.py) (Quota manager), [`scraper/scheduler.py`](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/scheduler.py) (Scheduled collection), [`scraper/config.py`](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/config.py) (Environment configuration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["YouTube collection"]
  node_scheduler["Scheduled collection<br/>[scheduler.py]"]
  node_collector["Collection orchestrator<br/>[youtube_scraper.py]"]
  node_search["Topic video search<br/>[youtube_scraper.py]"]
  node_video_details["Video details<br/>[youtube_scraper.py]"]
  node_comments["Comment collection<br/>[youtube_scraper.py]"]
end

subgraph group_operations["Scheduling and quota"]
  node_quota["Quota manager<br/>[quota_manager.py]"]
end

subgraph group_storage["Configuration and storage"]
  node_config["Environment configuration<br/>[config.py]"]
  node_video_store["Video persistence<br/>[youtube_scraper.py]"]
  node_comment_store["Comment persistence<br/>[youtube_scraper.py]"]
end

node_operator(("Operator"))
node_youtube_api["YouTube API"]
node_postgres[("PostgreSQL")]

node_operator -->|"starts directly"| node_collector
node_operator -->|"starts scheduler"| node_scheduler
node_scheduler -->|"runs collection"| node_collector
node_collector -->|"searches topics"| node_search
node_collector -->|"fetches details"| node_video_details
node_collector -->|"fetches comments"| node_comments
node_search -->|"searches videos"| node_youtube_api
node_video_details -->|"requests details"| node_youtube_api
node_comments -->|"requests comments"| node_youtube_api
node_collector -->|"checks status"| node_quota
node_search -->|"checks and logs"| node_quota
node_video_details -->|"checks and logs"| node_quota
node_comments -->|"checks and logs"| node_quota
node_quota -->|"reads and logs quota"| node_postgres
node_collector -->|"saves fetched videos"| node_video_store
node_collector -->|"saves fetched comments"| node_comment_store
node_video_store -->|"writes video records"| node_postgres
node_comment_store -->|"writes comment records"| node_postgres
node_collector -->|"loads credentials"| node_config
node_quota -->|"loads database settings"| node_config
node_collector -->|"opens connection"| node_postgres

click node_scheduler "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/scheduler.py"
click node_collector "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py"
click node_search "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py"
click node_video_details "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py"
click node_comments "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py"
click node_quota "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/quota_manager.py"
click node_config "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/config.py"
click node_video_store "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py"
click node_comment_store "https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scheduler,node_collector,node_search,node_video_details,node_comments toneBlue
class node_quota,node_postgres toneAmber
class node_config,node_video_store,node_comment_store,node_youtube_api toneMint
class node_operator toneIndigo
```

## Components

### YouTube collection

- [Scheduled collection](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/scheduler.py): `scraper/scheduler.py`
- [Collection orchestrator](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py): `scraper/youtube_scraper.py`
- [Topic video search](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py): `scraper/youtube_scraper.py`
- [Video details](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py): `scraper/youtube_scraper.py`
- [Comment collection](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py): `scraper/youtube_scraper.py`

### Scheduling and quota

- [Quota manager](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/quota_manager.py): `scraper/quota_manager.py`

### Configuration and storage

- [Environment configuration](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/config.py): `scraper/config.py`
- [Video persistence](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py): `scraper/youtube_scraper.py`
- [Comment persistence](https://github.com/nunoom/social-analytics-angola/blob/main/scraper/youtube_scraper.py): `scraper/youtube_scraper.py`

### Other components

- **Operator**
- **YouTube API**
- **PostgreSQL**

## Connections

- Operator → Collection orchestrator: starts directly (evidence: `scraper/youtube_scraper.py`)
- Operator → Scheduled collection: starts scheduler (evidence: `scraper/scheduler.py`)
- Scheduled collection → Collection orchestrator: runs collection (evidence: `scraper/scheduler.py`)
- Collection orchestrator → Topic video search: searches topics (evidence: `scraper/youtube_scraper.py`)
- Collection orchestrator → Video details: fetches details (evidence: `scraper/youtube_scraper.py`)
- Collection orchestrator → Comment collection: fetches comments (evidence: `scraper/youtube_scraper.py`)
- Topic video search → YouTube API: searches videos (evidence: `scraper/youtube_scraper.py`)
- Video details → YouTube API: requests details (evidence: `scraper/youtube_scraper.py`)
- Comment collection → YouTube API: requests comments (evidence: `scraper/youtube_scraper.py`)
- Collection orchestrator → Quota manager: checks status (evidence: `scraper/youtube_scraper.py`)
- Topic video search → Quota manager: checks and logs (evidence: `scraper/youtube_scraper.py`)
- Video details → Quota manager: checks and logs (evidence: `scraper/youtube_scraper.py`)
- Comment collection → Quota manager: checks and logs (evidence: `scraper/youtube_scraper.py`)
- Quota manager → PostgreSQL: reads and logs quota (evidence: `scraper/quota_manager.py`)
- Collection orchestrator → Video persistence: saves fetched videos (evidence: `scraper/youtube_scraper.py`)
- Collection orchestrator → Comment persistence: saves fetched comments (evidence: `scraper/youtube_scraper.py`)
- Video persistence → PostgreSQL: writes video records (evidence: `scraper/youtube_scraper.py`)
- Comment persistence → PostgreSQL: writes comment records (evidence: `scraper/youtube_scraper.py`)
- Collection orchestrator → Environment configuration: loads credentials (evidence: `scraper/youtube_scraper.py`)
- Quota manager → Environment configuration: loads database settings (evidence: `scraper/quota_manager.py`)
- Collection orchestrator → PostgreSQL: opens connection (evidence: `scraper/youtube_scraper.py`)
