How lotfimay/nyc-taxi-lakehouse works
This repository implements an NYC Taxi lakehouse and a REST semantic-query API. Its main data workflow takes raw taxi files through Hive registration, Bronze Iceberg storage, dbt Silver cleanup, and Gold star-schema marts; dbt semantic models then support MetricFlow queries exposed over HTTP. The README documents the lakehouse stages, while the sampled FastAPI sources show API routing and engine setup. The detailed wiring among unsampled SQL and dbt models is represented only where the README establishes it.
Main entry points: routes.py API routes · stg_trips.sql Bronze Iceberg · gold Gold dimensions · engine_manager.py Engine manager
Components 22
Lakehouse storage
- Raw taxi files
- Hive table setup
sql/hive_queries.sql - Hive raw tables
- Bronze CTAS
sql/iceberg_queries.sql - Bronze Iceberg
dbt/nyc_taxi_lakehouse/models/bronze/stg_trips.sql - Iceberg catalog
dbt modeling
- Silver trips
dbt/nyc_taxi_lakehouse/models/silver/trips.sql - Gold dimensions
dbt/nyc_taxi_lakehouse/models/gold - Gold trip fact
dbt/nyc_taxi_lakehouse/models/gold/fact_trips.sql - Dimension seeds
dbt/nyc_taxi_lakehouse/seeds/gold - Semantic models
dbt/nyc_taxi_lakehouse/models/semantic_models - MetricFlow time spine
dbt/nyc_taxi_lakehouse/models/gold/metricflow_time_spine.sql
Semantic API
- FastAPI application
fastapi/app/main.py - API routes
fastapi/app/api/routes.py - API schemas
fastapi/app/api/schemas.py - Engine manager
fastapi/app/engine_manager.py - MetricFlow engine
Query infrastructure
- Trino
- MinIO object store
- PostgreSQL catalog
Other components
- Data uploader
- API consumer
Connections 28
- Data uploader → Raw taxi files: uploads
readme.md - Raw taxi files → MinIO object store: stored in
readme.md - Hive table setup → Hive raw tables: registers
readme.md - Hive raw tables → Bronze CTAS: read by CTAS
readme.md - Bronze CTAS → Bronze Iceberg: builds
readme.md - Bronze Iceberg → Silver trips: transforms
readme.md - Silver trips → Gold dimensions: models
readme.md - Silver trips → Gold trip fact: models
readme.md - Dimension seeds → Gold dimensions: provides values
readme.md - Gold dimensions → Gold trip fact: joined into
readme.md - Gold trip fact → Semantic models: described by
readme.md - Gold dimensions → Semantic models: described by
readme.md - MetricFlow time spine → Semantic models: supports time metrics
- Bronze Iceberg → Iceberg catalog: metadata managed by
readme.md - Bronze CTAS → Trino: executed through
readme.md - Hive raw tables → MinIO object store: reads raw files
readme.md - API consumer → FastAPI application: sends requests
fastapi/app/main.py - FastAPI application → API routes: registers router
fastapi/app/main.py - API routes → API schemas: validates and serializes
fastapi/app/api/routes.py - API routes → Engine manager: gets engine
fastapi/app/api/routes.py - FastAPI application → Engine manager: initializes adapter and manifest
fastapi/app/main.py - Engine manager → MetricFlow engine: creates engine
fastapi/app/engine_manager.py - API routes → MetricFlow engine: queries and lists metrics
fastapi/app/api/routes.py - Engine manager → Trino: connects via dbt adapter
- MetricFlow engine → Trino: executes warehouse queries
- Trino → Bronze Iceberg: queries Iceberg tables
readme.md - Trino → Iceberg catalog: uses catalog metadata
readme.md - Iceberg catalog → PostgreSQL catalog: stores metadata in
readme.md