How lau-jr/nyc-taxi-data-engineering works
NYC Taxi Data Engineering turns monthly TLC Yellow Taxi files into a validated, replay-safe DuckDB warehouse, then publishes aggregates for dashboard consumption and builds a separate hourly forecasting feature snapshot. The primary workflow is ingest → quality handling and transformation → star-schema loading → serving mart. Other documented branches include metric-driven dashboarding and baseline scoring. Relationships for sampled Python modules are supported by excerpts and the SOURCE INDEX; SQL and some feature details are represented only where the README or excerpts document them.
Main entry points: database.py Database operations · ingest.py Ingest runner · serving.py Serving reads · streamlit_app.py Streamlit dashboard
Components 20
Ingestion
- Ingest runner
src/ingest.py - Quality checks
src/validation.py - Trip transformation
src/transform.py
Warehouse
- Database operations
src/database.py - Star schema
sql/01_create_schema.sql - Dimensions
sql/02_create_dimensions.sql - Trip fact table
sql/03_create_fact.sql - Quarantine table
sql/05_create_quarantine.sql - Pipeline run records
sql/06_create_pipeline_run.sql
Serving
- Daily zone mart refresh
sql/07_refresh_mart_daily_zone_trips.sql - Published trip mart
sql/07_refresh_mart_daily_zone_trips.sql - Serving reads
src/serving.py - Metric calculations
src/metrics.py - Streamlit dashboard
streamlit_app.py
ML Features
- Feature builder
src/features.py - Hourly feature query
sql/08_build_features_zone_hour.sql - Versioned snapshots
data/features - Baseline scoring
src/baseline.py
Other components
- Pipeline operator
- Monthly TLC files
Connections 21
- Pipeline operator → Ingest runner: runs
README.md - Monthly TLC files → Ingest runner: loads
README.md - Ingest runner → Quality checks: checks rows
src/ingest.py - Ingest runner → Trip transformation: prepares trips
src/ingest.py - Quality checks → Trip transformation: uses trip IDs
src/validation.py - Ingest runner → Database operations: loads through
src/ingest.py - Database operations → Star schema: applies
src/database.py - Database operations → Dimensions: applies
src/database.py - Database operations → Trip fact table: inserts rows
src/database.py - Database operations → Quarantine table: inserts rejects
src/database.py - Database operations → Pipeline run records: records runs
README.md - Ingest runner → Daily zone mart refresh: rebuilds mart
README.md - Daily zone mart refresh → Published trip mart: publishes
README.md - Streamlit dashboard → Serving reads: reads mart
streamlit_app.py - Serving reads → Published trip mart: loads and checks
src/serving.py - Serving reads → Pipeline run records: checks freshness
src/serving.py - Streamlit dashboard → Metric calculations: summarizes
streamlit_app.py - Feature builder → Hourly feature query: executes query
src/features.py - Hourly feature query → Trip fact table: builds from trips
README.md - Feature builder → Versioned snapshots: writes snapshot
README.md - Baseline scoring → Versioned snapshots: scores snapshot
README.md