How axbouf/yt_etl works
YT_ETL collects YouTube channel video metadata, writes a dated JSON snapshot, then synchronizes that snapshot through PostgreSQL staging and transformed core tables before running Soda checks. Airflow DAGs coordinate these tasks, including a trigger from extraction to warehouse loading and onward to quality checks. The excerpts show the warehouse helpers and task wiring; the Soda check files are present in the tree but their contents were not sampled.
Main entry points: dwh.py Staging and core sync · main.py Pipeline DAGs · soda.py Soda check runner · video_stats.py Video metadata extraction
Components 14
Airflow orchestration
- Pipeline DAGs
dags/main.py
YouTube collection
- Video metadata extraction
dags/api/video_stats.py - JSON snapshot
data/YT_data_2026-04-22.json
Warehouse processing
- Staging and core sync
dags/datawarehouse/dwh.py - Snapshot loader
dags/datawarehouse/data_loading.py - Database utilities
dags/datawarehouse/data_utils.py - Row insert/update/delete
dags/datawarehouse/data_modification.py - Duration and type transform
dags/datawarehouse/data_transformation.py
Data quality
- Soda check runner
dags/data_quality/soda.py - Soda connection config
include/soda/configuration.yaml - Soda checks
include/soda/check.yaml
Other components
- Airflow scheduler
- YouTube API
- PostgreSQL warehouse
Connections 17
- Airflow scheduler → Pipeline DAGs: schedules
dags/main.py - Pipeline DAGs → Video metadata extraction: runs collection
dags/main.py - Video metadata extraction → YouTube API: requests metadata
dags/api/video_stats.py - Video metadata extraction → JSON snapshot: writes snapshot
dags/api/video_stats.py - Pipeline DAGs → Staging and core sync: triggers update
dags/main.py - Staging and core sync → Snapshot loader: loads snapshot
dags/datawarehouse/dwh.py - Snapshot loader → JSON snapshot: reads JSON
dags/datawarehouse/data_loading.py - Staging and core sync → Database utilities: uses connection helpers
dags/datawarehouse/dwh.py - Staging and core sync → Row insert/update/delete: syncs rows
dags/datawarehouse/dwh.py - Staging and core sync → Duration and type transform: transforms core rows
dags/datawarehouse/dwh.py - Staging and core sync → PostgreSQL warehouse: reads and writes
dags/datawarehouse/dwh.py - Database utilities → PostgreSQL warehouse: opens connection
dags/datawarehouse/data_utils.py - Row insert/update/delete → PostgreSQL warehouse: executes row SQL
dags/datawarehouse/data_modification.py - Pipeline DAGs → Soda check runner: triggers checks
dags/main.py - Soda check runner → PostgreSQL warehouse: scans schemas
dags/data_quality/soda.py - Soda check runner → Soda connection config: configures scan
dags/data_quality/soda.py - Soda check runner → Soda checks: runs checks
dags/data_quality/soda.py