How leticia-gerperangel/data-engineering-project works
This repository is an end-to-end NYC taxi analytics data pipeline. Its principal workflow ingests TLC trip data into PostgreSQL, uses Kestra to coordinate cloud loading into GCS and BigQuery, transforms warehouse data with dbt, and exposes marts to Looker Studio. Separate BigQuery SQL supports warehouse setup and quality checks; a local Spark demonstration processes downloaded Parquet files independently of the production path. The README documents the high-level handoffs, while excerpts show ingestion, schema reuse, and Spark behavior; unexcerpted orchestration and dbt internals are represented at their documented boundaries.
Main entry points: 01-external-tables.sql BigQuery warehouse · ingest_data.py CSV ingestion · 08-gcp-taxi.yaml Kestra flows · 07-gcp-setup.yaml Cloud setup flows
Components 18
Data ingestion
- CSV ingestion
ingestion/ingest_data.py - Trip schemas
ingestion/schemas.py - PostgreSQL staging
Pipeline orchestration
- Kestra flows
orchestration/08-gcp-taxi.yaml - Cloud setup flows
orchestration/07-gcp-setup.yaml - GCP KV setup
orchestration/06-gcp-kv-example.yaml - Postgres flow
orchestration/04-postgres-taxi.yaml
Warehouse and processing
- Google Cloud Storage
- BigQuery warehouse
dw/01-external-tables.sql - Warehouse checks
dw/04-data-quality-check.sql - Spark batch demo
batch/notebook/taxi_trip/taxi_schema.py
Analytics and reporting
- dbt staging
dbt-project/taxi_rides_ny/models/staging/stg_yellow_tripdata.sql - Trip union
dbt-project/taxi_rides_ny/models/intermediate/int_trips_union.sql - Dimensions and facts
dbt-project/taxi_rides_ny/models/marts/fact_trips.sql - Analytics reports
dbt-project/taxi_rides_ny/models/marts/reports/mart_month_revenue.sql - Looker Studio
Other components
- NYC TLC data
- Pipeline operator
Connections 19
- NYC TLC data → CSV ingestion: supplies CSV
README.md - CSV ingestion → Trip schemas: imports types
ingestion/ingest_data.py - CSV ingestion → PostgreSQL staging: loads chunks
README.md - Pipeline operator → Kestra flows: runs flows
README.md - Pipeline operator → GCP KV setup: registers values
README.md - GCP KV setup → Cloud setup flows: provides configuration
README.md - Cloud setup flows → Google Cloud Storage: creates bucket
README.md - Cloud setup flows → BigQuery warehouse: creates dataset
README.md - Postgres flow → PostgreSQL staging: loads and merges
README.md - Kestra flows → Google Cloud Storage: uploads data
README.md - Kestra flows → BigQuery warehouse: loads tables
README.md - Google Cloud Storage → BigQuery warehouse: backs external tables
README.md - BigQuery warehouse → Warehouse checks: checks quality
README.md - BigQuery warehouse → dbt staging: transforms source data
README.md - dbt staging → Trip union: feeds union
README.md - Trip union → Dimensions and facts: builds marts
README.md - Dimensions and facts → Analytics reports: feeds reports
README.md - Analytics reports → Looker Studio: queries marts
README.md - NYC TLC data → Spark batch demo: downloads Parquet
batch/notebook/taxi_trip/taxi_schema.py