How sagar8080/data-fusion-engineering works
Project Type and Purpose
• This is a cloud-based data engineering pipeline (“data fusion”) that ingests multiple external datasets (traffic, taxi, weather, crashes, persons, vehicles), transforms them via Spark/Beam, and loads them into BigQuery for downstream analysis and dashboarding.
• Purpose: analyze how weather, traffic, and taxi usage affect road safety in NYC.
File Structure Analysis
• Top-level:
• ingest/ — Python Cloud Function code for each data source
• load/ — PySpark and Beam jobs to load landing-zone files into raw BigQuery tables
• transform/ — PySpark scripts to clean/enrich and write production tables
• / — SQL model and analytical queries
• utils/ — infra helper (create_infra.py), constants
• *.tf, variables.tf — Terraform IaC for GCS buckets, Cloud Functions, schedulers
• setup.sh, run_load_to_raw.sh, run_transformations.sh — shell scripts for initial setup and cron jobs
• Patterns: ELT pipeline with schema-on-read landing zone in GCS → raw tables → transformed prd tables → unified model → dashboards.
• Config/infra files: main.tf, variables.tf, setup.sh, utils/create_infra.py
README Insights
• Data flow: event-driven ingestion → batch processing → scheduled transformations → analysis & dashboard.
a. Main Components
• Data Sources (external APIs & CSV endpoints)
• Cloud Scheduler
• Cloud Functions (one per data source)
• Google Cloud Storage (landing-zone buckets: pre-processed, processed)
• Dataproc clusters (PySpark jobs in load/ and transform/)
• BigQuery (raw dataset, production dataset, catalog)
• Compute Engine VM (cron scripts + Superset dashboard)
• Data Catalog (metadata)
• Superset (visualization)
b. Relationships & Data Flow
1. Scheduler → triggers Cloud Functions
2. Cloud Functions → fetch data → upload CSV/JSON to GCS pre-processed bucket & optionally write minimal state to BigQuery
3. Cron on Compute Engine → runs run_load_to_raw.sh → starts Dataproc → executes load/*.py → writes to BigQuery raw tables
4. Cron → runs run_transformations.sh → starts Dataproc → executes transform/*.py → writes to BigQuery production tables (partitioned by year/month)
5. BigQuery tables → Data Catalog registers metadata → Analysts / Superset queries prd tables → dashboards
c. Architectural Patterns/Principles
• ELT (extract to GCS, load to raw, transform in warehouse)
• Micro-batches with scheduling (cron & Cloud Scheduler)
• IaC for reproducible infra (Terraform + Python utils)
• Separation of concerns: ingest, load, transform, analyze
• Modularization per data domain (crashes, persons, traffic, taxi, weather, vehicles)
d. Key Technologies
• Python (Cloud Functions, scripts)
• Apache Spark on Dataproc (batch transformations)
• Apache Beam (experimental, later replaced)
• Google Cloud Functions & Scheduler (ingest orchestration)
• Google Cloud Storage (data lake)
• BigQuery (data warehouse)
• Terraform & Python utils for infra provisioning
• Compute Engine VM for cron jobs and Superset hosting
• Superset (dashboard)