# lotfimay/nyc-taxi-lakehouse architecture

> Architecture diagram of the GitHub repository lotfimay/nyc-taxi-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lotfimay/nyc-taxi-lakehouse
- Repository: https://github.com/lotfimay/nyc-taxi-lakehouse

## Overview

This repository implements an NYC Taxi lakehouse and a REST semantic-query API. Its main data workflow takes raw taxi files through Hive registration, Bronze Iceberg storage, dbt Silver cleanup, and Gold star-schema marts; dbt semantic models then support MetricFlow queries exposed over HTTP. The README documents the lakehouse stages, while the sampled FastAPI sources show API routing and engine setup. The detailed wiring among unsampled SQL and dbt models is represented only where the README establishes it.

Main entry points: [`fastapi/app/api/routes.py`](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/api/routes.py) (API routes), [`dbt/nyc_taxi_lakehouse/models/bronze/stg_trips.sql`](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/bronze/stg_trips.sql) (Bronze Iceberg), [`dbt/nyc_taxi_lakehouse/models/gold`](https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/models/gold) (Gold dimensions), [`fastapi/app/engine_manager.py`](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/engine_manager.py) (Engine manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_storage["Lakehouse storage"]
  node_raw_files[("Raw taxi files")]
  node_hive_sql["Hive table setup<br/>[hive_queries.sql]"]
  node_hive_tables[("Hive raw tables")]
  node_bronze_sql["Bronze CTAS"]
  node_bronze[("Bronze Iceberg<br/>[stg_trips.sql]")]
  node_iceberg_catalog[("Iceberg catalog")]
end

subgraph group_modeling["dbt modeling"]
  node_silver[("Silver trips<br/>[trips.sql]")]
  node_gold_dims[("Gold dimensions")]
  node_gold_fact[("Gold trip fact<br/>[fact_trips.sql]")]
  node_gold_seeds["Dimension seeds"]
  node_semantic_models["Semantic models"]
  node_time_spine[("MetricFlow time spine")]
end

subgraph group_semantic["Semantic API"]
  node_fastapi_app["FastAPI application<br/>[main.py]"]
  node_api_routes["API routes<br/>[routes.py]"]
  node_api_schemas["API schemas<br/>[schemas.py]"]
  node_engine_manager["Engine manager<br/>[engine_manager.py]"]
  node_metricflow["MetricFlow engine"]
end

subgraph group_services["Query infrastructure"]
  node_trino{{"Trino"}}
  node_minio[("MinIO object store")]
  node_postgres[("PostgreSQL catalog")]
end

node_data_uploader(("Data uploader"))
node_api_user(("API consumer"))

node_data_uploader -->|"uploads"| node_raw_files
node_raw_files -->|"stored in"| node_minio
node_hive_sql -->|"registers"| node_hive_tables
node_hive_tables -->|"read by CTAS"| node_bronze_sql
node_bronze_sql -->|"builds"| node_bronze
node_bronze -->|"transforms"| node_silver
node_silver -->|"models"| node_gold_dims
node_silver -->|"models"| node_gold_fact
node_gold_seeds -->|"provides values"| node_gold_dims
node_gold_dims -->|"joined into"| node_gold_fact
node_gold_fact -->|"described by"| node_semantic_models
node_gold_dims -->|"described by"| node_semantic_models
node_time_spine -->|"supports time metrics"| node_semantic_models
node_bronze -->|"metadata managed by"| node_iceberg_catalog
node_bronze_sql -->|"executed through"| node_trino
node_hive_tables -->|"reads raw files"| node_minio
node_api_user -->|"sends requests"| node_fastapi_app
node_fastapi_app -->|"registers router"| node_api_routes
node_api_routes -->|"validates and serializes"| node_api_schemas
node_api_routes -->|"gets engine"| node_engine_manager
node_fastapi_app -->|"initializes adapter and manifest"| node_engine_manager
node_engine_manager -->|"creates engine"| node_metricflow
node_api_routes -->|"queries and lists metrics"| node_metricflow
node_engine_manager -.->|"connects via dbt adapter"| node_trino
node_metricflow -.->|"executes warehouse queries"| node_trino
node_trino -->|"queries Iceberg tables"| node_bronze
node_trino -->|"uses catalog metadata"| node_iceberg_catalog
node_iceberg_catalog -->|"stores metadata in"| node_postgres

click node_hive_sql "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/sql/hive_queries.sql"
click node_bronze_sql "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/sql/iceberg_queries.sql"
click node_bronze "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/bronze/stg_trips.sql"
click node_silver "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/silver/trips.sql"
click node_gold_dims "https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/models/gold"
click node_gold_fact "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/gold/fact_trips.sql"
click node_gold_seeds "https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/seeds/gold"
click node_semantic_models "https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/models/semantic_models"
click node_time_spine "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/gold/metricflow_time_spine.sql"
click node_fastapi_app "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/main.py"
click node_api_routes "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/api/routes.py"
click node_api_schemas "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/api/schemas.py"
click node_engine_manager "https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/engine_manager.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_raw_files,node_hive_sql,node_hive_tables,node_bronze_sql,node_bronze,node_iceberg_catalog toneBlue
class node_silver,node_gold_dims,node_gold_fact,node_gold_seeds,node_semantic_models,node_time_spine toneAmber
class node_fastapi_app,node_api_routes,node_api_schemas,node_engine_manager,node_metricflow,node_api_user toneMint
class node_trino,node_minio,node_postgres toneRose
class node_data_uploader toneIndigo
```

## Components

### Lakehouse storage

- **Raw taxi files**
- [Hive table setup](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/sql/hive_queries.sql): `sql/hive_queries.sql`
- **Hive raw tables**
- [Bronze CTAS](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/sql/iceberg_queries.sql): `sql/iceberg_queries.sql`
- [Bronze Iceberg](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/bronze/stg_trips.sql): `dbt/nyc_taxi_lakehouse/models/bronze/stg_trips.sql`
- **Iceberg catalog**

### dbt modeling

- [Silver trips](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/silver/trips.sql): `dbt/nyc_taxi_lakehouse/models/silver/trips.sql`
- [Gold dimensions](https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/models/gold): `dbt/nyc_taxi_lakehouse/models/gold`
- [Gold trip fact](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/gold/fact_trips.sql): `dbt/nyc_taxi_lakehouse/models/gold/fact_trips.sql`
- [Dimension seeds](https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/seeds/gold): `dbt/nyc_taxi_lakehouse/seeds/gold`
- [Semantic models](https://github.com/lotfimay/nyc-taxi-lakehouse/tree/main/dbt/nyc_taxi_lakehouse/models/semantic_models): `dbt/nyc_taxi_lakehouse/models/semantic_models`
- [MetricFlow time spine](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/dbt/nyc_taxi_lakehouse/models/gold/metricflow_time_spine.sql): `dbt/nyc_taxi_lakehouse/models/gold/metricflow_time_spine.sql`

### Semantic API

- [FastAPI application](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/main.py): `fastapi/app/main.py`
- [API routes](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/api/routes.py): `fastapi/app/api/routes.py`
- [API schemas](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/api/schemas.py): `fastapi/app/api/schemas.py`
- [Engine manager](https://github.com/lotfimay/nyc-taxi-lakehouse/blob/main/fastapi/app/engine_manager.py): `fastapi/app/engine_manager.py`
- **MetricFlow engine**

### Query infrastructure

- **Trino**
- **MinIO object store**
- **PostgreSQL catalog**

### Other components

- **Data uploader**
- **API consumer**

## Connections

- Data uploader → Raw taxi files: uploads (evidence: `readme.md`)
- Raw taxi files → MinIO object store: stored in (evidence: `readme.md`)
- Hive table setup → Hive raw tables: registers (evidence: `readme.md`)
- Hive raw tables → Bronze CTAS: read by CTAS (evidence: `readme.md`)
- Bronze CTAS → Bronze Iceberg: builds (evidence: `readme.md`)
- Bronze Iceberg → Silver trips: transforms (evidence: `readme.md`)
- Silver trips → Gold dimensions: models (evidence: `readme.md`)
- Silver trips → Gold trip fact: models (evidence: `readme.md`)
- Dimension seeds → Gold dimensions: provides values (evidence: `readme.md`)
- Gold dimensions → Gold trip fact: joined into (evidence: `readme.md`)
- Gold trip fact → Semantic models: described by (evidence: `readme.md`)
- Gold dimensions → Semantic models: described by (evidence: `readme.md`)
- MetricFlow time spine → Semantic models: supports time metrics
- Bronze Iceberg → Iceberg catalog: metadata managed by (evidence: `readme.md`)
- Bronze CTAS → Trino: executed through (evidence: `readme.md`)
- Hive raw tables → MinIO object store: reads raw files (evidence: `readme.md`)
- API consumer → FastAPI application: sends requests (evidence: `fastapi/app/main.py`)
- FastAPI application → API routes: registers router (evidence: `fastapi/app/main.py`)
- API routes → API schemas: validates and serializes (evidence: `fastapi/app/api/routes.py`)
- API routes → Engine manager: gets engine (evidence: `fastapi/app/api/routes.py`)
- FastAPI application → Engine manager: initializes adapter and manifest (evidence: `fastapi/app/main.py`)
- Engine manager → MetricFlow engine: creates engine (evidence: `fastapi/app/engine_manager.py`)
- API routes → MetricFlow engine: queries and lists metrics (evidence: `fastapi/app/api/routes.py`)
- Engine manager → Trino: connects via dbt adapter
- MetricFlow engine → Trino: executes warehouse queries
- Trino → Bronze Iceberg: queries Iceberg tables (evidence: `readme.md`)
- Trino → Iceberg catalog: uses catalog metadata (evidence: `readme.md`)
- Iceberg catalog → PostgreSQL catalog: stores metadata in (evidence: `readme.md`)