How trannhatnguyen2/nyc_taxi_data_pipeline works
NYC Taxi Data Pipeline ingests TLC trip files through a scheduled batch ELT flow, stages processed data in PostgreSQL, validates it, and builds a warehouse star schema with dbt. A separate CDC/streaming path sends PostgreSQL changes through Debezium and Kafka into MinIO; the repository also includes streaming-read code and Trino configuration. The batch code and Airflow DAG were excerpted; CDC, dbt, validation, and Trino connections are included only where the README documents them.
Main entry points: elt_pipeline_dag.py Airflow DAG · datalake_to_dw.py Batch staging load · extract_load.py File extraction · transform_data.py Trip normalization
Components 19
Batch ingestion
- Airflow DAG
airflow/dags/elt_pipeline_dag.py - File extraction
scripts/extract_load.py - Trip normalization
scripts/transform_data.py - Delta conversion
scripts/convert_to_delta.py - Batch staging load
batch_processing/datalake_to_dw.py
Warehouse modeling
- Data validation
data_validation/gx - Star schema models
dbt_nyc/models/production - Warehouse descriptions
dbt_nyc/macros
CDC streaming
- CDC connector setup
debezium/configs/taxi-nyc-cdc.json - Connector registration
debezium/run.sh - Database event producer
utils/streaming_data_db.py - JSON event producer
utils/streaming_data_json.py - Stream lake writer
stream_processing/streaming_to_datalake.py - Parquet stream reader
stream_processing/read_parquet_streaming.py
Shared data services
- MinIO data lake
- PostgreSQL
- Kafka topics
- Trino query engine
trino/catalog/datalake.properties
Other components
- Pipeline operator
Connections 20
- Pipeline operator → Airflow DAG: starts
README.md - Airflow DAG → File extraction: calls
airflow/dags/elt_pipeline_dag.py - Airflow DAG → Trip normalization: calls
airflow/dags/elt_pipeline_dag.py - Airflow DAG → Delta conversion: calls
airflow/dags/elt_pipeline_dag.py - File extraction → MinIO data lake: uploads raw
scripts/extract_load.py - Trip normalization → MinIO data lake: writes processed
scripts/transform_data.py - Delta conversion → MinIO data lake: reads and writes
scripts/convert_to_delta.py - Batch staging load → MinIO data lake: reads parquet
batch_processing/datalake_to_dw.py - Batch staging load → PostgreSQL: writes staging
batch_processing/datalake_to_dw.py - PostgreSQL → Data validation: staging validated
README.md - PostgreSQL → Star schema models: staged data modeled
README.md - Star schema models → Warehouse descriptions: uses descriptions
- Pipeline operator → Connector registration: registers connector
README.md - Connector registration → CDC connector setup: uses configuration
- PostgreSQL → Kafka topics: changes captured
README.md - Database event producer → PostgreSQL: inserts records
utils/streaming_data_db.py - JSON event producer → Kafka topics: produces events
- Kafka topics → Stream lake writer: streams events
stream_processing/streaming_to_datalake.py - Stream lake writer → MinIO data lake: writes parquet
stream_processing/streaming_to_datalake.py - MinIO data lake → Parquet stream reader: provides parquet