How mattthedataengineer/yt_elt works
This repository is an Airflow-orchestrated YouTube ELT pipeline. Its main workflow retrieves channel uploads and video statistics, writes a dated JSON extract, upserts the data into PostgreSQL staging and core schemas, then runs Soda checks on both schemas. The warehouse separates loading, transformation, database utilities, and row modifications. The README documents these three DAG stages; the sampled source shows their task dependencies and key calls. Some deployment and quality-check details are documented but not fully sampled.
Main entry points: main.py DAG task flows · dwh.py Schema upserts · soda.py Soda scan task · video_stats.py Channel lookup
Components 17
Airflow orchestration
- DAG task flows
dags/main.py
YouTube extraction
- Channel lookup
dags/api/video_stats.py - Playlist scan
dags/api/video_stats.py - Video extraction
dags/api/video_stats.py
Data warehouse
- JSON loader
dags/datawarehouse/data_loading.py - Schema upserts
dags/datawarehouse/dwh.py - Core transformation
dags/datawarehouse/data_transformation.py - Database utilities
dags/datawarehouse/data_utils.py - Row modifications
dags/datawarehouse/data_modification.py
Data quality
- Soda scan task
dags/dataquality/soda.py - Quality checks
include/soda/checks.yml - Soda configuration
include/soda/configuration.yml
Other components
- Scheduled workflow
- YouTube API
- Dated JSON extract
data - PostgreSQL
- Soda CLI
Connections 23
- Scheduled workflow → DAG task flows: starts
dags/main.py - DAG task flows → Channel lookup: runs
dags/main.py - Channel lookup → Playlist scan: passes playlist
dags/main.py - Playlist scan → Video extraction: passes IDs
dags/main.py - Channel lookup → YouTube API: requests channel
dags/api/video_stats.py - Playlist scan → YouTube API: requests uploads
dags/api/video_stats.py - Video extraction → YouTube API: requests details
dags/api/video_stats.py - DAG task flows → Dated JSON extract: writes extract
dags/main.py - DAG task flows → Schema upserts: runs staging
dags/main.py - Schema upserts → JSON loader: loads JSON
dags/datawarehouse/dwh.py - JSON loader → Dated JSON extract: reads extract
dags/datawarehouse/data_loading.py - DAG task flows → Schema upserts: then runs core
dags/main.py - Schema upserts → Core transformation: transforms rows
dags/datawarehouse/dwh.py - Schema upserts → Database utilities: uses utilities
dags/datawarehouse/dwh.py - Schema upserts → Row modifications: upserts rows
dags/datawarehouse/dwh.py - Database utilities → PostgreSQL: connects
dags/datawarehouse/data_utils.py - Row modifications → PostgreSQL: writes rows
dags/datawarehouse/data_modification.py - DAG task flows → Soda scan task: runs staging check
dags/main.py - DAG task flows → Soda scan task: then checks core
dags/main.py - Soda scan task → Soda CLI: invokes scan
dags/dataquality/soda.py - Soda scan task → Quality checks: selects checks
dags/dataquality/soda.py - Soda scan task → Soda configuration: configures scan
dags/dataquality/soda.py - Soda CLI → PostgreSQL: checks schemas