# axbouf/yt_etl architecture

> Architecture diagram of the GitHub repository axbouf/yt_etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/axbouf/yt_etl
- Repository: https://github.com/axbouf/yt_etl

## Overview

YT_ETL collects YouTube channel video metadata, writes a dated JSON snapshot, then synchronizes that snapshot through PostgreSQL staging and transformed core tables before running Soda checks. Airflow DAGs coordinate these tasks, including a trigger from extraction to warehouse loading and onward to quality checks. The excerpts show the warehouse helpers and task wiring; the Soda check files are present in the tree but their contents were not sampled.

Main entry points: [`dags/datawarehouse/dwh.py`](https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/dwh.py) (Staging and core sync), [`dags/main.py`](https://github.com/axbouf/yt_etl/blob/main/dags/main.py) (Pipeline DAGs), [`dags/data_quality/soda.py`](https://github.com/axbouf/yt_etl/blob/main/dags/data_quality/soda.py) (Soda check runner), [`dags/api/video_stats.py`](https://github.com/axbouf/yt_etl/blob/main/dags/api/video_stats.py) (Video metadata extraction)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Airflow orchestration"]
  node_dag["Pipeline DAGs<br/>[main.py]"]
end

subgraph group_collection["YouTube collection"]
  node_extract["Video metadata extraction<br/>[video_stats.py]"]
  node_snapshot["JSON snapshot"]
end

subgraph group_warehouse["Warehouse processing"]
  node_warehouse["Staging and core sync<br/>[dwh.py]"]
  node_json_loader["Snapshot loader<br/>[data_loading.py]"]
  node_db_utils["Database utilities<br/>[data_utils.py]"]
  node_row_changes["Row insert/update/delete"]
  node_transform["Duration and type transform"]
end

subgraph group_quality["Data quality"]
  node_soda["Soda check runner<br/>[soda.py]"]
  node_check_config["Soda connection config<br/>[configuration.yaml]"]
  node_checks["Soda checks<br/>[check.yaml]"]
end

node_scheduler(("Airflow scheduler"))
node_youtube{{"YouTube API"}}
node_postgres[("PostgreSQL warehouse")]

node_scheduler -->|"schedules"| node_dag
node_dag -->|"runs collection"| node_extract
node_extract -->|"requests metadata"| node_youtube
node_extract -->|"writes snapshot"| node_snapshot
node_dag -->|"triggers update"| node_warehouse
node_warehouse -->|"loads snapshot"| node_json_loader
node_json_loader -->|"reads JSON"| node_snapshot
node_warehouse -->|"uses connection helpers"| node_db_utils
node_warehouse -->|"syncs rows"| node_row_changes
node_warehouse -->|"transforms core rows"| node_transform
node_warehouse -->|"reads and writes"| node_postgres
node_db_utils -->|"opens connection"| node_postgres
node_row_changes -->|"executes row SQL"| node_postgres
node_dag -->|"triggers checks"| node_soda
node_soda -->|"scans schemas"| node_postgres
node_soda -->|"configures scan"| node_check_config
node_soda -->|"runs checks"| node_checks

click node_dag "https://github.com/axbouf/yt_etl/blob/main/dags/main.py"
click node_extract "https://github.com/axbouf/yt_etl/blob/main/dags/api/video_stats.py"
click node_snapshot "https://github.com/axbouf/yt_etl/blob/main/data/YT_data_2026-04-22.json"
click node_warehouse "https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/dwh.py"
click node_json_loader "https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_loading.py"
click node_db_utils "https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_utils.py"
click node_row_changes "https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_modification.py"
click node_transform "https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_transformation.py"
click node_soda "https://github.com/axbouf/yt_etl/blob/main/dags/data_quality/soda.py"
click node_check_config "https://github.com/axbouf/yt_etl/blob/main/include/soda/configuration.yaml"
click node_checks "https://github.com/axbouf/yt_etl/blob/main/include/soda/check.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag toneBlue
class node_extract,node_snapshot,node_postgres toneAmber
class node_warehouse,node_json_loader,node_db_utils,node_row_changes,node_transform,node_youtube toneMint
class node_soda,node_check_config,node_checks,node_scheduler toneRose
```

## Components

### Airflow orchestration

- [Pipeline DAGs](https://github.com/axbouf/yt_etl/blob/main/dags/main.py): `dags/main.py`

### YouTube collection

- [Video metadata extraction](https://github.com/axbouf/yt_etl/blob/main/dags/api/video_stats.py): `dags/api/video_stats.py`
- [JSON snapshot](https://github.com/axbouf/yt_etl/blob/main/data/YT_data_2026-04-22.json): `data/YT_data_2026-04-22.json`

### Warehouse processing

- [Staging and core sync](https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/dwh.py): `dags/datawarehouse/dwh.py`
- [Snapshot loader](https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_loading.py): `dags/datawarehouse/data_loading.py`
- [Database utilities](https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_utils.py): `dags/datawarehouse/data_utils.py`
- [Row insert/update/delete](https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_modification.py): `dags/datawarehouse/data_modification.py`
- [Duration and type transform](https://github.com/axbouf/yt_etl/blob/main/dags/datawarehouse/data_transformation.py): `dags/datawarehouse/data_transformation.py`

### Data quality

- [Soda check runner](https://github.com/axbouf/yt_etl/blob/main/dags/data_quality/soda.py): `dags/data_quality/soda.py`
- [Soda connection config](https://github.com/axbouf/yt_etl/blob/main/include/soda/configuration.yaml): `include/soda/configuration.yaml`
- [Soda checks](https://github.com/axbouf/yt_etl/blob/main/include/soda/check.yaml): `include/soda/check.yaml`

### Other components

- **Airflow scheduler**
- **YouTube API**
- **PostgreSQL warehouse**

## Connections

- Airflow scheduler → Pipeline DAGs: schedules (evidence: `dags/main.py`)
- Pipeline DAGs → Video metadata extraction: runs collection (evidence: `dags/main.py`)
- Video metadata extraction → YouTube API: requests metadata (evidence: `dags/api/video_stats.py`)
- Video metadata extraction → JSON snapshot: writes snapshot (evidence: `dags/api/video_stats.py`)
- Pipeline DAGs → Staging and core sync: triggers update (evidence: `dags/main.py`)
- Staging and core sync → Snapshot loader: loads snapshot (evidence: `dags/datawarehouse/dwh.py`)
- Snapshot loader → JSON snapshot: reads JSON (evidence: `dags/datawarehouse/data_loading.py`)
- Staging and core sync → Database utilities: uses connection helpers (evidence: `dags/datawarehouse/dwh.py`)
- Staging and core sync → Row insert/update/delete: syncs rows (evidence: `dags/datawarehouse/dwh.py`)
- Staging and core sync → Duration and type transform: transforms core rows (evidence: `dags/datawarehouse/dwh.py`)
- Staging and core sync → PostgreSQL warehouse: reads and writes (evidence: `dags/datawarehouse/dwh.py`)
- Database utilities → PostgreSQL warehouse: opens connection (evidence: `dags/datawarehouse/data_utils.py`)
- Row insert/update/delete → PostgreSQL warehouse: executes row SQL (evidence: `dags/datawarehouse/data_modification.py`)
- Pipeline DAGs → Soda check runner: triggers checks (evidence: `dags/main.py`)
- Soda check runner → PostgreSQL warehouse: scans schemas (evidence: `dags/data_quality/soda.py`)
- Soda check runner → Soda connection config: configures scan (evidence: `dags/data_quality/soda.py`)
- Soda check runner → Soda checks: runs checks (evidence: `dags/data_quality/soda.py`)
