# tunguyenn99/son-tung-mtp-analytics architecture

> Architecture diagram of the GitHub repository tunguyenn99/son-tung-mtp-analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tunguyenn99/son-tung-mtp-analytics
- Repository: https://github.com/tunguyenn99/son-tung-mtp-analytics

## Overview

##### Project Type & Purpose
- This is a data-engineering pipeline rather than a traditional web app.
- It extracts YouTube data, loads it into Snowflake, transforms via dbt, orchestrates via Airflow/Cosmos, and visualizes in Power BI.
- Key stakeholders: data engineers, BI analysts.

##### File Structure to Derive Components
- Top-level scripts & configs
– python-get-data/: YouTube API extraction notebooks + CSV output (seed)
– snowflake-create-wh/: SQL script to provision Snowflake warehouse/database/schema
– dbt_youtube/: Core dbt project (models in bronze/silver/gold layers, seeds, snapshots, macros, tests)
– dbt_youtube_dag/: Airflow DAG (Cosmos/Astronomer) that invokes the dbt steps inside Docker
– power-bi-dashboard/: Power BI report file (.pbix)
- Support files: .env-sample, pyproject.toml, uv.lock, Dockerfile, requirements.txt, logs/.

##### README Insights
- End-to-end pipeline stages are explicitly listed
- Technologies used at each stage (YouTube Data API, Python, Snowflake, dbt-core/dbt-snowflake, Airflow/Astronomer/Cosmos, Power BI)
- Visual architecture image under imgs/00-architecture.png (reference for level of detail)

##### Main Components & Relationships
a. External Source:
– YouTube Data API v3
b. Ingestion Layer:
– python-get-data Jupyter Notebook → CSV files (stored in python-get-data/data/)
c. Data Warehouse:
– Snowflake (Warehouse → Database → Schema)
– Seeds loaded (CSV → staging tables)
d. Transformation Layer (dbt):
– Bronze (raw ingest models)
– Silver (staging + cleanup)
– Gold (aggregations, business metrics)
e. Orchestration:
– Apache Airflow (Astronomer/Cosmos) running a DAG that executes dbt commands inside a Docker container
f. Visualization:
– Power BI dashboard connecting to Snowflake gold tables
g. CI/CD / Dev Environment:
– Dockerfile / requirements.txt / astronaut CLI for local dev
– Environment variables managed via .env

##### Architectural Patterns & Design Principles
- ELT paradigm (Extract → Load → Transform)
- Layered data model (Bronze → Silver → Gold)
- Modular, declarative data transformations via dbt
- Containerized orchestration with Airflow in Docker for reproducibility
- Clear separation of concerns: extraction, transformation, orchestration, visualization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Source
    YTAPI["YouTube Data API v3"]:::external

    %% Data Ingestion
    subgraph "Data Ingestion"
        Notebook["Jupyter Notebook\n(get-data.ipynb)"]:::compute
        CSV["CSV Files\ndata/"]:::data
    end
    YTAPI -->|extracts data| Notebook
    Notebook -->|outputs CSV| CSV

    click Notebook "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/python-get-data/get-data.ipynb"
    click CSV "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/python-get-data/data/"

    %% Infrastructure Provisioning
    subgraph "Infrastructure Provisioning"
        snowSQL["Snowflake Provisioning\n(create-wh.sql)"]:::infra
    end
    snowSQL -->|provisions| Snowflake

    click snowSQL "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/snowflake-create-wh/create-wh.sql"

    %% Data Warehouse & Transformation
    subgraph "Data Warehouse & Transformation"
        Snowflake["Snowflake\nWarehouse/Database/Schema"]:::storage
        Seeds["dbt Seeds\n(channels.csv, playlist_items.csv,\nplaylists.csv, video_stats.csv)"]:::data
        Bronze["Bronze Models"]:::dbt
        Silver["Silver Models"]:::dbt
        Gold["Gold Models"]:::dbt
    end
    CSV -->|seed input| Seeds
    Seeds -->|loads into| Bronze
    Bronze -->|transforms to| Silver
    Silver -->|aggregates to| Gold
    Gold -->|stores in| Snowflake

    click Seeds "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/seeds/"
    click Bronze "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/models/bronze/"
    click Silver "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/models/silver/"
    click Gold "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/models/gold/"
    click dbtProject "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/dbt_youtube/dbt_project.yml"
    click snapshots "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/snapshots/"
    click macros "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/macros/"
    click tests "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube/tests/"
    click profiles "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/dbt_youtube/profiles.yml"
    click pkgs "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/dbt_youtube/packages.yml"

    %% Orchestration
    subgraph "Orchestration"
        Airflow["Airflow DAG\n(Astronomer/Cosmos)"]:::orchestration
        Docker["Docker Container\n(runs dbt)"]:::infra
    end
    Airflow -->|schedules| Docker
    Docker -->|executes dbt run/test/snapshot| Bronze
    Docker -->|executes dbt run/test/snapshot| Silver
    Docker -->|executes dbt run/test/snapshot| Gold

    click Airflow "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/dbt_youtube_dag/dags/dbt_dag.py"
    click dockerfile "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube_dag/Dockerfile"
    click reqDag "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/dbt_youtube_dag/requirements.txt"
    click astroCfg "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/dbt_youtube_dag/.astro/config.yaml"
    click dagFolder "https://github.com/tunguyenn99/son-tung-mtp-analytics/tree/main/dbt_youtube_dag/dags/dbt/"

    %% Visualization
    subgraph "Visualization"
        PowerBI["Power BI Dashboard\n(SonTungMTP_Dashboard.pbix)"]:::visual
    end
    Gold -->|query gold tables| PowerBI

    click PowerBI "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/power-bi-dashboard/SonTungMTP_Dashboard.pbix"

    %% CI/CD & Dev Environment
    subgraph "CI/CD & Dev Environment"
        EnvRoot[".env-sample (root)"]:::infra
        EnvPD[".env-sample (python-get-data)"]:::infra
        PyProj["pyproject.toml"]:::infra
        UVLock["uv.lock"]:::infra
        Logs["Logs\n(dbt.log)"]:::infra
    end

    click EnvRoot "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/.env-sample"
    click EnvPD "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/python-get-data/.env-sample"
    click PyProj "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/pyproject.toml"
    click UVLock "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/uv.lock"
    click Logs "https://github.com/tunguyenn99/son-tung-mtp-analytics/blob/main/logs/dbt.log"

    %% Styles
    classDef external fill:#AEDFF7,stroke:#0288D1,color:#000
    classDef compute fill:#C8E6C9,stroke:#388E3C,color:#000
    classDef data fill:#FFF9C4,stroke:#FBC02D,color:#000
    classDef storage fill:#B2EBF2,stroke:#0097A7,color:#000
    classDef dbt fill:#FFCC80,stroke:#F57C00,color:#000
    classDef orchestration fill:#E1BEE7,stroke:#8E24AA,color:#000
    classDef visual fill:#F8BBD0,stroke:#C2185B,color:#000
    classDef infra fill:#E0E0E0,stroke:#757575,color:#000
```
