# mattthedataengineer/yt_elt architecture

> Architecture diagram of the GitHub repository mattthedataengineer/yt_elt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mattthedataengineer/yt_elt
- Repository: https://github.com/mattthedataengineer/yt_elt

## Overview

This repository is an Airflow-orchestrated YouTube ELT pipeline. Its main workflow retrieves channel uploads and video statistics, writes a dated JSON extract, upserts the data into PostgreSQL staging and core schemas, then runs Soda checks on both schemas. The warehouse separates loading, transformation, database utilities, and row modifications. The README documents these three DAG stages; the sampled source shows their task dependencies and key calls. Some deployment and quality-check details are documented but not fully sampled.

Main entry points: [`dags/main.py`](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/main.py) (DAG task flows), [`dags/datawarehouse/dwh.py`](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/dwh.py) (Schema upserts), [`dags/dataquality/soda.py`](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/dataquality/soda.py) (Soda scan task), [`dags/api/video_stats.py`](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py) (Channel lookup)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Airflow orchestration"]
  node_dag_tasks["DAG task flows<br/>[main.py]"]
end

subgraph group_extraction["YouTube extraction"]
  node_channel_lookup["Channel lookup<br/>[video_stats.py]"]
  node_playlist_scan["Playlist scan<br/>[video_stats.py]"]
  node_video_extract["Video extraction<br/>[video_stats.py]"]
end

subgraph group_warehouse["Data warehouse"]
  node_json_loader["JSON loader<br/>[data_loading.py]"]
  node_staging_core["Schema upserts<br/>[dwh.py]"]
  node_transform["Core transformation"]
  node_db_helpers["Database utilities<br/>[data_utils.py]"]
  node_row_changes["Row modifications"]
end

subgraph group_quality["Data quality"]
  node_soda_task["Soda scan task<br/>[soda.py]"]
  node_quality_checks["Quality checks<br/>[checks.yml]"]
  node_soda_config["Soda configuration<br/>[configuration.yml]"]
end

node_schedule(("Scheduled workflow"))
node_youtube_api["YouTube API"]
node_json_output["Dated JSON extract"]
node_postgres[("PostgreSQL")]
node_soda_runner["Soda CLI"]

node_schedule -->|"starts"| node_dag_tasks
node_dag_tasks -->|"runs"| node_channel_lookup
node_channel_lookup -->|"passes playlist"| node_playlist_scan
node_playlist_scan -->|"passes IDs"| node_video_extract
node_channel_lookup -->|"requests channel"| node_youtube_api
node_playlist_scan -->|"requests uploads"| node_youtube_api
node_video_extract -->|"requests details"| node_youtube_api
node_dag_tasks -->|"writes extract"| node_json_output
node_dag_tasks -->|"runs staging"| node_staging_core
node_staging_core -->|"loads JSON"| node_json_loader
node_json_loader -->|"reads extract"| node_json_output
node_dag_tasks -->|"then runs core"| node_staging_core
node_staging_core -->|"transforms rows"| node_transform
node_staging_core -->|"uses utilities"| node_db_helpers
node_staging_core -->|"upserts rows"| node_row_changes
node_db_helpers -->|"connects"| node_postgres
node_row_changes -->|"writes rows"| node_postgres
node_dag_tasks -->|"runs staging check"| node_soda_task
node_dag_tasks -->|"then checks core"| node_soda_task
node_soda_task -->|"invokes scan"| node_soda_runner
node_soda_task -->|"selects checks"| node_quality_checks
node_soda_task -->|"configures scan"| node_soda_config
node_soda_runner -.->|"checks schemas"| node_postgres

click node_dag_tasks "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/main.py"
click node_channel_lookup "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py"
click node_playlist_scan "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py"
click node_video_extract "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py"
click node_json_output "https://github.com/mattthedataengineer/yt_elt/tree/main/data"
click node_json_loader "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_loading.py"
click node_staging_core "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/dwh.py"
click node_transform "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_transformation.py"
click node_db_helpers "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_utils.py"
click node_row_changes "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_modification.py"
click node_soda_task "https://github.com/mattthedataengineer/yt_elt/blob/main/dags/dataquality/soda.py"
click node_quality_checks "https://github.com/mattthedataengineer/yt_elt/blob/main/include/soda/checks.yml"
click node_soda_config "https://github.com/mattthedataengineer/yt_elt/blob/main/include/soda/configuration.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dag_tasks toneBlue
class node_channel_lookup,node_playlist_scan,node_video_extract,node_postgres toneAmber
class node_json_loader,node_staging_core,node_transform,node_db_helpers,node_row_changes,node_youtube_api toneMint
class node_soda_task,node_quality_checks,node_soda_config toneRose
class node_schedule,node_soda_runner toneIndigo
class node_json_output toneTeal
```

## Components

### Airflow orchestration

- [DAG task flows](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/main.py): `dags/main.py`

### YouTube extraction

- [Channel lookup](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py): `dags/api/video_stats.py`
- [Playlist scan](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py): `dags/api/video_stats.py`
- [Video extraction](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/api/video_stats.py): `dags/api/video_stats.py`

### Data warehouse

- [JSON loader](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_loading.py): `dags/datawarehouse/data_loading.py`
- [Schema upserts](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/dwh.py): `dags/datawarehouse/dwh.py`
- [Core transformation](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_transformation.py): `dags/datawarehouse/data_transformation.py`
- [Database utilities](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_utils.py): `dags/datawarehouse/data_utils.py`
- [Row modifications](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/datawarehouse/data_modification.py): `dags/datawarehouse/data_modification.py`

### Data quality

- [Soda scan task](https://github.com/mattthedataengineer/yt_elt/blob/main/dags/dataquality/soda.py): `dags/dataquality/soda.py`
- [Quality checks](https://github.com/mattthedataengineer/yt_elt/blob/main/include/soda/checks.yml): `include/soda/checks.yml`
- [Soda configuration](https://github.com/mattthedataengineer/yt_elt/blob/main/include/soda/configuration.yml): `include/soda/configuration.yml`

### Other components

- **Scheduled workflow**
- **YouTube API**
- [Dated JSON extract](https://github.com/mattthedataengineer/yt_elt/tree/main/data): `data`
- **PostgreSQL**
- **Soda CLI**

## Connections

- Scheduled workflow → DAG task flows: starts (evidence: `dags/main.py`)
- DAG task flows → Channel lookup: runs (evidence: `dags/main.py`)
- Channel lookup → Playlist scan: passes playlist (evidence: `dags/main.py`)
- Playlist scan → Video extraction: passes IDs (evidence: `dags/main.py`)
- Channel lookup → YouTube API: requests channel (evidence: `dags/api/video_stats.py`)
- Playlist scan → YouTube API: requests uploads (evidence: `dags/api/video_stats.py`)
- Video extraction → YouTube API: requests details (evidence: `dags/api/video_stats.py`)
- DAG task flows → Dated JSON extract: writes extract (evidence: `dags/main.py`)
- DAG task flows → Schema upserts: runs staging (evidence: `dags/main.py`)
- Schema upserts → JSON loader: loads JSON (evidence: `dags/datawarehouse/dwh.py`)
- JSON loader → Dated JSON extract: reads extract (evidence: `dags/datawarehouse/data_loading.py`)
- DAG task flows → Schema upserts: then runs core (evidence: `dags/main.py`)
- Schema upserts → Core transformation: transforms rows (evidence: `dags/datawarehouse/dwh.py`)
- Schema upserts → Database utilities: uses utilities (evidence: `dags/datawarehouse/dwh.py`)
- Schema upserts → Row modifications: upserts rows (evidence: `dags/datawarehouse/dwh.py`)
- Database utilities → PostgreSQL: connects (evidence: `dags/datawarehouse/data_utils.py`)
- Row modifications → PostgreSQL: writes rows (evidence: `dags/datawarehouse/data_modification.py`)
- DAG task flows → Soda scan task: runs staging check (evidence: `dags/main.py`)
- DAG task flows → Soda scan task: then checks core (evidence: `dags/main.py`)
- Soda scan task → Soda CLI: invokes scan (evidence: `dags/dataquality/soda.py`)
- Soda scan task → Quality checks: selects checks (evidence: `dags/dataquality/soda.py`)
- Soda scan task → Soda configuration: configures scan (evidence: `dags/dataquality/soda.py`)
- Soda CLI → PostgreSQL: checks schemas
