How jnkodithuwakku/nyc-taxi-medallion-etl works
This repository implements an Airflow-orchestrated NYC taxi ETL product. Monthly Yellow and Green Parquet files plus zone CSVs arrive in a landing area, are bulk-loaded with Polars and ADBC into PostgreSQL Bronze tables, quality-checked, transformed into a Silver star schema, and aggregated into Gold analytics. A report generator queries the warehouse and writes a standalone HTML dashboard for stakeholders. The graph preserves the documented medallion, dimensional modeling, quality, analytics, and dashboard stages; some SQL internals were sampled only partially.
Components 16
Orchestration
- Airflow DAG
dags/pipeline.py - Schema Setup
dags/pipeline.py - File Sensors
dags/pipeline.py - Quality Checks
dags/pipeline.py
Ingestion
- Landing Zone
- Bulk Loader
dags/pipeline.py - Polars ADBC
dags/pipeline.py
Warehouse
- Bronze Tables
dags/sql/init_bronze_schema.sql - Silver Star Schema
dags/sql/silver_transformations.sql - Gold Aggregations
dags/sql/gold_aggregations.sql
Analytics
- Report Generator
dags/generate_report.py - Fleet Analytics
dags/sql/gold_aggregations.sql - Behavior Analytics
dags/sql/gold_aggregations.sql
Presentation
- HTML Dashboard
data/gold_analytics_report.html
Other components
- Taxi Data Files
- Business Stakeholder
Connections 20
- Taxi Data Files → Landing Zone: lands files
- Airflow DAG → Schema Setup: runs setup
- Airflow DAG → File Sensors: runs sensors
- Airflow DAG → Bulk Loader: dispatches loads
- Schema Setup → Bronze Tables: creates tables
- Landing Zone → File Sensors: checks files
- File Sensors → Bulk Loader: unblocks load
- Bulk Loader → Polars ADBC: invokes ingestion
- Polars ADBC → Bronze Tables: writes rows
- Bronze Tables → Quality Checks: reads records
- Quality Checks → Silver Star Schema: gates transform
- Bronze Tables → Silver Star Schema: transforms data
- Silver Star Schema → Gold Aggregations: aggregates facts
- Silver Star Schema → Fleet Analytics: computes flows
- Silver Star Schema → Behavior Analytics: computes behavior
- Gold Aggregations → Report Generator: supplies metrics
- Fleet Analytics → Report Generator: supplies charts
- Behavior Analytics → Report Generator: supplies charts
- Report Generator → HTML Dashboard: writes report
- HTML Dashboard → Business Stakeholder: presents insights