# leticia-gerperangel/data-engineering-project architecture

> Architecture diagram of the GitHub repository leticia-gerperangel/data-engineering-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/leticia-gerperangel/data-engineering-project
- Repository: https://github.com/leticia-gerperangel/data-engineering-project

## Overview

This repository is an end-to-end NYC taxi analytics data pipeline. Its principal workflow ingests TLC trip data into PostgreSQL, uses Kestra to coordinate cloud loading into GCS and BigQuery, transforms warehouse data with dbt, and exposes marts to Looker Studio. Separate BigQuery SQL supports warehouse setup and quality checks; a local Spark demonstration processes downloaded Parquet files independently of the production path. The README documents the high-level handoffs, while excerpts show ingestion, schema reuse, and Spark behavior; unexcerpted orchestration and dbt internals are represented at their documented boundaries.

Main entry points: [`dw/01-external-tables.sql`](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dw/01-external-tables.sql) (BigQuery warehouse), [`ingestion/ingest_data.py`](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/ingestion/ingest_data.py) (CSV ingestion), [`orchestration/08-gcp-taxi.yaml`](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/08-gcp-taxi.yaml) (Kestra flows), [`orchestration/07-gcp-setup.yaml`](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/07-gcp-setup.yaml) (Cloud setup flows)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data ingestion"]
  node_ingest_tool["CSV ingestion<br/>[ingest_data.py]"]
  node_schemas["Trip schemas<br/>[schemas.py]"]
  node_postgres[("PostgreSQL staging")]
end

subgraph group_orchestrate["Pipeline orchestration"]
  node_kestra["Kestra flows<br/>[08-gcp-taxi.yaml]"]
  node_cloud_setup["Cloud setup flows<br/>[07-gcp-setup.yaml]"]
  node_gcp_values["GCP KV setup"]
  node_postgres_flow["Postgres flow"]
end

subgraph group_warehouse["Warehouse and processing"]
  node_gcs[("Google Cloud Storage")]
  node_bigquery[("BigQuery warehouse")]
  node_warehouse_checks["Warehouse checks"]
  node_spark_batch["Spark batch demo<br/>[taxi_schema.py]"]
end

subgraph group_analytics["Analytics and reporting"]
  node_dbt_staging["dbt staging"]
  node_dbt_intermediate["Trip union"]
  node_dbt_marts["Dimensions and facts<br/>[fact_trips.sql]"]
  node_reports["Analytics reports"]
  node_dashboard["Looker Studio"]
end

node_tlc["NYC TLC data"]
node_pipeline_operator(("Pipeline operator"))

node_tlc -->|"supplies CSV"| node_ingest_tool
node_ingest_tool -->|"imports types"| node_schemas
node_ingest_tool -->|"loads chunks"| node_postgres
node_pipeline_operator -->|"runs flows"| node_kestra
node_pipeline_operator -->|"registers values"| node_gcp_values
node_gcp_values -->|"provides configuration"| node_cloud_setup
node_cloud_setup -->|"creates bucket"| node_gcs
node_cloud_setup -->|"creates dataset"| node_bigquery
node_postgres_flow -->|"loads and merges"| node_postgres
node_kestra -->|"uploads data"| node_gcs
node_kestra -->|"loads tables"| node_bigquery
node_gcs -->|"backs external tables"| node_bigquery
node_bigquery -->|"checks quality"| node_warehouse_checks
node_bigquery -->|"transforms source data"| node_dbt_staging
node_dbt_staging -->|"feeds union"| node_dbt_intermediate
node_dbt_intermediate -->|"builds marts"| node_dbt_marts
node_dbt_marts -->|"feeds reports"| node_reports
node_reports -->|"queries marts"| node_dashboard
node_tlc -->|"downloads Parquet"| node_spark_batch

click node_ingest_tool "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/ingestion/ingest_data.py"
click node_schemas "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/ingestion/schemas.py"
click node_kestra "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/08-gcp-taxi.yaml"
click node_cloud_setup "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/07-gcp-setup.yaml"
click node_gcp_values "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/06-gcp-kv-example.yaml"
click node_postgres_flow "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/04-postgres-taxi.yaml"
click node_bigquery "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dw/01-external-tables.sql"
click node_warehouse_checks "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dw/04-data-quality-check.sql"
click node_spark_batch "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/batch/notebook/taxi_trip/taxi_schema.py"
click node_dbt_staging "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/staging/stg_yellow_tripdata.sql"
click node_dbt_intermediate "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/intermediate/int_trips_union.sql"
click node_dbt_marts "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/marts/fact_trips.sql"
click node_reports "https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/marts/reports/mart_month_revenue.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ingest_tool,node_schemas,node_postgres toneBlue
class node_kestra,node_cloud_setup,node_gcp_values,node_postgres_flow toneAmber
class node_gcs,node_bigquery,node_warehouse_checks,node_spark_batch toneMint
class node_dbt_staging,node_dbt_intermediate,node_dbt_marts,node_reports,node_dashboard toneRose
class node_tlc,node_pipeline_operator toneIndigo
```

## Components

### Data ingestion

- [CSV ingestion](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/ingestion/ingest_data.py): `ingestion/ingest_data.py`
- [Trip schemas](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/ingestion/schemas.py): `ingestion/schemas.py`
- **PostgreSQL staging**

### Pipeline orchestration

- [Kestra flows](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/08-gcp-taxi.yaml): `orchestration/08-gcp-taxi.yaml`
- [Cloud setup flows](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/07-gcp-setup.yaml): `orchestration/07-gcp-setup.yaml`
- [GCP KV setup](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/06-gcp-kv-example.yaml): `orchestration/06-gcp-kv-example.yaml`
- [Postgres flow](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/orchestration/04-postgres-taxi.yaml): `orchestration/04-postgres-taxi.yaml`

### Warehouse and processing

- **Google Cloud Storage**
- [BigQuery warehouse](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dw/01-external-tables.sql): `dw/01-external-tables.sql`
- [Warehouse checks](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dw/04-data-quality-check.sql): `dw/04-data-quality-check.sql`
- [Spark batch demo](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/batch/notebook/taxi_trip/taxi_schema.py): `batch/notebook/taxi_trip/taxi_schema.py`

### Analytics and reporting

- [dbt staging](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/staging/stg_yellow_tripdata.sql): `dbt-project/taxi_rides_ny/models/staging/stg_yellow_tripdata.sql`
- [Trip union](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/intermediate/int_trips_union.sql): `dbt-project/taxi_rides_ny/models/intermediate/int_trips_union.sql`
- [Dimensions and facts](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/marts/fact_trips.sql): `dbt-project/taxi_rides_ny/models/marts/fact_trips.sql`
- [Analytics reports](https://github.com/leticia-gerperangel/data-engineering-project/blob/main/dbt-project/taxi_rides_ny/models/marts/reports/mart_month_revenue.sql): `dbt-project/taxi_rides_ny/models/marts/reports/mart_month_revenue.sql`
- **Looker Studio**

### Other components

- **NYC TLC data**
- **Pipeline operator**

## Connections

- NYC TLC data → CSV ingestion: supplies CSV (evidence: `README.md`)
- CSV ingestion → Trip schemas: imports types (evidence: `ingestion/ingest_data.py`)
- CSV ingestion → PostgreSQL staging: loads chunks (evidence: `README.md`)
- Pipeline operator → Kestra flows: runs flows (evidence: `README.md`)
- Pipeline operator → GCP KV setup: registers values (evidence: `README.md`)
- GCP KV setup → Cloud setup flows: provides configuration (evidence: `README.md`)
- Cloud setup flows → Google Cloud Storage: creates bucket (evidence: `README.md`)
- Cloud setup flows → BigQuery warehouse: creates dataset (evidence: `README.md`)
- Postgres flow → PostgreSQL staging: loads and merges (evidence: `README.md`)
- Kestra flows → Google Cloud Storage: uploads data (evidence: `README.md`)
- Kestra flows → BigQuery warehouse: loads tables (evidence: `README.md`)
- Google Cloud Storage → BigQuery warehouse: backs external tables (evidence: `README.md`)
- BigQuery warehouse → Warehouse checks: checks quality (evidence: `README.md`)
- BigQuery warehouse → dbt staging: transforms source data (evidence: `README.md`)
- dbt staging → Trip union: feeds union (evidence: `README.md`)
- Trip union → Dimensions and facts: builds marts (evidence: `README.md`)
- Dimensions and facts → Analytics reports: feeds reports (evidence: `README.md`)
- Analytics reports → Looker Studio: queries marts (evidence: `README.md`)
- NYC TLC data → Spark batch demo: downloads Parquet (evidence: `batch/notebook/taxi_trip/taxi_schema.py`)
