How tunguyenn99/son-tung-mtp-analytics works
Project Type & Purpose
• This is a data-engineering pipeline rather than a traditional web app.
• It extracts YouTube data, loads it into Snowflake, transforms via dbt, orchestrates via Airflow/Cosmos, and visualizes in Power BI.
• Key stakeholders: data engineers, BI analysts.
File Structure to Derive Components
• Top-level scripts & configs
– python-get-data/: YouTube API extraction notebooks + CSV output (seed)
– snowflake-create-wh/: SQL script to provision Snowflake warehouse/database/schema
– dbt_youtube/: Core dbt project (models in bronze/silver/gold layers, seeds, snapshots, macros, tests)
– dbt_youtube_dag/: Airflow DAG (Cosmos/Astronomer) that invokes the dbt steps inside Docker
– power-bi-dashboard/: Power BI report file (.pbix)
• Support files: .env-sample, pyproject.toml, uv.lock, Dockerfile, requirements.txt, logs/.
README Insights
• End-to-end pipeline stages are explicitly listed
• Technologies used at each stage (YouTube Data API, Python, Snowflake, dbt-core/dbt-snowflake, Airflow/Astronomer/Cosmos, Power BI)
• Visual architecture image under imgs/00-architecture.png (reference for level of detail)
Main Components & Relationships
a. External Source:
– YouTube Data API v3
b. Ingestion Layer:
– python-get-data Jupyter Notebook → CSV files (stored in python-get-data/data/)
c. Data Warehouse:
– Snowflake (Warehouse → Database → Schema)
– Seeds loaded (CSV → staging tables)
d. Transformation Layer (dbt):
– Bronze (raw ingest models)
– Silver (staging + cleanup)
– Gold (aggregations, business metrics)
e. Orchestration:
– Apache Airflow (Astronomer/Cosmos) running a DAG that executes dbt commands inside a Docker container
f. Visualization:
– Power BI dashboard connecting to Snowflake gold tables
g. CI/CD / Dev Environment:
– Dockerfile / requirements.txt / astronaut CLI for local dev
– Environment variables managed via .env
Architectural Patterns & Design Principles
• ELT paradigm (Extract → Load → Transform)
• Layered data model (Bronze → Silver → Gold)
• Modular, declarative data transformations via dbt
• Containerized orchestration with Airflow in Docker for reproducibility
• Clear separation of concerns: extraction, transformation, orchestration, visualization