# lau-jr/nyc-taxi-data-engineering architecture

> Architecture diagram of the GitHub repository lau-jr/nyc-taxi-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lau-jr/nyc-taxi-data-engineering
- Repository: https://github.com/lau-jr/nyc-taxi-data-engineering

## Overview

NYC Taxi Data Engineering turns monthly TLC Yellow Taxi files into a validated, replay-safe DuckDB warehouse, then publishes aggregates for dashboard consumption and builds a separate hourly forecasting feature snapshot. The primary workflow is ingest → quality handling and transformation → star-schema loading → serving mart. Other documented branches include metric-driven dashboarding and baseline scoring. Relationships for sampled Python modules are supported by excerpts and the SOURCE INDEX; SQL and some feature details are represented only where the README or excerpts document them.

Main entry points: [`src/database.py`](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/database.py) (Database operations), [`src/ingest.py`](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/ingest.py) (Ingest runner), [`src/serving.py`](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/serving.py) (Serving reads), [`streamlit_app.py`](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/streamlit_app.py) (Streamlit dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_ingest_runner["Ingest runner<br/>[ingest.py]"]
  node_quality["Quality checks<br/>[validation.py]"]
  node_transform["Trip transformation<br/>[transform.py]"]
end

subgraph group_warehouse["Warehouse"]
  node_db_api["Database operations<br/>[database.py]"]
  node_schema["Star schema"]
  node_dimensions["Dimensions"]
  node_facts[("Trip fact table<br/>[03_create_fact.sql]")]
  node_quarantine[("Quarantine table")]
  node_run_records[("Pipeline run records")]
end

subgraph group_serving["Serving"]
  node_mart_refresh["Daily zone mart refresh"]
  node_mart[("Published trip mart")]
  node_serving_api["Serving reads<br/>[serving.py]"]
  node_metrics["Metric calculations<br/>[metrics.py]"]
  node_dashboard["Streamlit dashboard<br/>[streamlit_app.py]"]
end

subgraph group_ml["ML Features"]
  node_feature_builder["Feature builder<br/>[features.py]"]
  node_feature_sql["Hourly feature query"]
  node_feature_snapshot["Versioned snapshots"]
  node_baseline["Baseline scoring<br/>[baseline.py]"]
end

node_operator(("Pipeline operator"))
node_raw["Monthly TLC files"]

node_operator -->|"runs"| node_ingest_runner
node_raw -->|"loads"| node_ingest_runner
node_ingest_runner -->|"checks rows"| node_quality
node_ingest_runner -->|"prepares trips"| node_transform
node_quality -->|"uses trip IDs"| node_transform
node_ingest_runner -->|"loads through"| node_db_api
node_db_api -->|"applies"| node_schema
node_db_api -->|"applies"| node_dimensions
node_db_api -->|"inserts rows"| node_facts
node_db_api -->|"inserts rejects"| node_quarantine
node_db_api -->|"records runs"| node_run_records
node_ingest_runner -->|"rebuilds mart"| node_mart_refresh
node_mart_refresh -->|"publishes"| node_mart
node_dashboard -->|"reads mart"| node_serving_api
node_serving_api -->|"loads and checks"| node_mart
node_serving_api -->|"checks freshness"| node_run_records
node_dashboard -->|"summarizes"| node_metrics
node_feature_builder -->|"executes query"| node_feature_sql
node_feature_sql -->|"builds from trips"| node_facts
node_feature_builder -->|"writes snapshot"| node_feature_snapshot
node_baseline -->|"scores snapshot"| node_feature_snapshot

click node_ingest_runner "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/ingest.py"
click node_quality "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/validation.py"
click node_transform "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/transform.py"
click node_db_api "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/database.py"
click node_schema "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/01_create_schema.sql"
click node_dimensions "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/02_create_dimensions.sql"
click node_facts "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/03_create_fact.sql"
click node_quarantine "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/05_create_quarantine.sql"
click node_run_records "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/06_create_pipeline_run.sql"
click node_mart_refresh "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/07_refresh_mart_daily_zone_trips.sql"
click node_mart "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/07_refresh_mart_daily_zone_trips.sql"
click node_serving_api "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/serving.py"
click node_metrics "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/metrics.py"
click node_dashboard "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/streamlit_app.py"
click node_feature_builder "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/features.py"
click node_feature_sql "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/08_build_features_zone_hour.sql"
click node_feature_snapshot "https://github.com/lau-jr/nyc-taxi-data-engineering/tree/main/data/features"
click node_baseline "https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/baseline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_runner,node_quality,node_transform toneBlue
class node_db_api,node_schema,node_dimensions,node_facts,node_quarantine,node_run_records toneAmber
class node_mart_refresh,node_mart,node_serving_api,node_metrics,node_dashboard toneMint
class node_feature_builder,node_feature_sql,node_feature_snapshot,node_baseline toneRose
class node_operator,node_raw toneIndigo
```

## Components

### Ingestion

- [Ingest runner](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/ingest.py): `src/ingest.py`
- [Quality checks](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/validation.py): `src/validation.py`
- [Trip transformation](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/transform.py): `src/transform.py`

### Warehouse

- [Database operations](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/database.py): `src/database.py`
- [Star schema](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/01_create_schema.sql): `sql/01_create_schema.sql`
- [Dimensions](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/02_create_dimensions.sql): `sql/02_create_dimensions.sql`
- [Trip fact table](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/03_create_fact.sql): `sql/03_create_fact.sql`
- [Quarantine table](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/05_create_quarantine.sql): `sql/05_create_quarantine.sql`
- [Pipeline run records](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/06_create_pipeline_run.sql): `sql/06_create_pipeline_run.sql`

### Serving

- [Daily zone mart refresh](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/07_refresh_mart_daily_zone_trips.sql): `sql/07_refresh_mart_daily_zone_trips.sql`
- [Published trip mart](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/07_refresh_mart_daily_zone_trips.sql): `sql/07_refresh_mart_daily_zone_trips.sql`
- [Serving reads](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/serving.py): `src/serving.py`
- [Metric calculations](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/metrics.py): `src/metrics.py`
- [Streamlit dashboard](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/streamlit_app.py): `streamlit_app.py`

### ML Features

- [Feature builder](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/features.py): `src/features.py`
- [Hourly feature query](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/sql/08_build_features_zone_hour.sql): `sql/08_build_features_zone_hour.sql`
- [Versioned snapshots](https://github.com/lau-jr/nyc-taxi-data-engineering/tree/main/data/features): `data/features`
- [Baseline scoring](https://github.com/lau-jr/nyc-taxi-data-engineering/blob/main/src/baseline.py): `src/baseline.py`

### Other components

- **Pipeline operator**
- **Monthly TLC files**

## Connections

- Pipeline operator → Ingest runner: runs (evidence: `README.md`)
- Monthly TLC files → Ingest runner: loads (evidence: `README.md`)
- Ingest runner → Quality checks: checks rows (evidence: `src/ingest.py`)
- Ingest runner → Trip transformation: prepares trips (evidence: `src/ingest.py`)
- Quality checks → Trip transformation: uses trip IDs (evidence: `src/validation.py`)
- Ingest runner → Database operations: loads through (evidence: `src/ingest.py`)
- Database operations → Star schema: applies (evidence: `src/database.py`)
- Database operations → Dimensions: applies (evidence: `src/database.py`)
- Database operations → Trip fact table: inserts rows (evidence: `src/database.py`)
- Database operations → Quarantine table: inserts rejects (evidence: `src/database.py`)
- Database operations → Pipeline run records: records runs (evidence: `README.md`)
- Ingest runner → Daily zone mart refresh: rebuilds mart (evidence: `README.md`)
- Daily zone mart refresh → Published trip mart: publishes (evidence: `README.md`)
- Streamlit dashboard → Serving reads: reads mart (evidence: `streamlit_app.py`)
- Serving reads → Published trip mart: loads and checks (evidence: `src/serving.py`)
- Serving reads → Pipeline run records: checks freshness (evidence: `src/serving.py`)
- Streamlit dashboard → Metric calculations: summarizes (evidence: `streamlit_app.py`)
- Feature builder → Hourly feature query: executes query (evidence: `src/features.py`)
- Hourly feature query → Trip fact table: builds from trips (evidence: `README.md`)
- Feature builder → Versioned snapshots: writes snapshot (evidence: `README.md`)
- Baseline scoring → Versioned snapshots: scores snapshot (evidence: `README.md`)
