# sagar8080/data-fusion-engineering architecture

> Architecture diagram of the GitHub repository sagar8080/data-fusion-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sagar8080/data-fusion-engineering
- Repository: https://github.com/sagar8080/data-fusion-engineering

## Overview

##### Project Type and Purpose
- This is a cloud-based data engineering pipeline (“data fusion”) that ingests multiple external datasets (traffic, taxi, weather, crashes, persons, vehicles), transforms them via Spark/Beam, and loads them into BigQuery for downstream analysis and dashboarding.
- Purpose: analyze how weather, traffic, and taxi usage affect road safety in NYC.

##### File Structure Analysis
- Top-level:
- ingest/ — Python Cloud Function code for each data source
- load/ — PySpark and Beam jobs to load landing-zone files into raw BigQuery tables
- transform/ — PySpark scripts to clean/enrich and write production tables
- / — SQL model and analytical queries
- utils/ — infra helper (create_infra.py), constants
- *.tf, variables.tf — Terraform IaC for GCS buckets, Cloud Functions, schedulers
- setup.sh, run_load_to_raw.sh, run_transformations.sh — shell scripts for initial setup and cron jobs
- Patterns: ELT pipeline with schema-on-read landing zone in GCS → raw tables → transformed prd tables → unified model → dashboards.
- Config/infra files: main.tf, variables.tf, setup.sh, utils/create_infra.py

##### README Insights
- Data flow: event-driven ingestion → batch processing → scheduled transformations → analysis & dashboard.

a. Main Components
- Data Sources (external APIs & CSV endpoints)
- Cloud Scheduler
- Cloud Functions (one per data source)
- Google Cloud Storage (landing-zone buckets: pre-processed, processed)
- Dataproc clusters (PySpark jobs in load/ and transform/)
- BigQuery (raw dataset, production dataset, catalog)
- Compute Engine VM (cron scripts + Superset dashboard)
- Data Catalog (metadata)
- Superset (visualization)
b. Relationships & Data Flow
1. Scheduler → triggers Cloud Functions
2. Cloud Functions → fetch data → upload CSV/JSON to GCS pre-processed bucket & optionally write minimal state to BigQuery
3. Cron on Compute Engine → runs run_load_to_raw.sh → starts Dataproc → executes load/*.py → writes to BigQuery raw tables
4. Cron → runs run_transformations.sh → starts Dataproc → executes transform/*.py → writes to BigQuery production tables (partitioned by year/month)
5. BigQuery tables → Data Catalog registers metadata → Analysts / Superset queries prd tables → dashboards
c. Architectural Patterns/Principles
- ELT (extract to GCS, load to raw, transform in warehouse)
- Micro-batches with scheduling (cron & Cloud Scheduler)
- IaC for reproducible infra (Terraform + Python utils)
- Separation of concerns: ingest, load, transform, analyze
- Modularization per data domain (crashes, persons, traffic, taxi, weather, vehicles)
d. Key Technologies
- Python (Cloud Functions, scripts)
- Apache Spark on Dataproc (batch transformations)
- Apache Beam (experimental, later replaced)
- Google Cloud Functions & Scheduler (ingest orchestration)
- Google Cloud Storage (data lake)
- BigQuery (data warehouse)
- Terraform & Python utils for infra provisioning
- Compute Engine VM for cron jobs and Superset hosting
- Superset (dashboard)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingest Layer
    subgraph "Ingest Layer"
        direction TB
        Scheduler["Cloud Scheduler"]:::gcp
        subgraph "Cloud Functions"
            direction TB
            CF_crashes["Crashes Function"]:::gcp
            CF_persons["Persons Function"]:::gcp
            CF_taxi["Taxi Function"]:::gcp
            CF_traffic["Traffic Function"]:::gcp
            CF_vehicles["Vehicles Function"]:::gcp
            CF_weather["Weather Function"]:::gcp
        end
    end

    %% Storage Landing
    subgraph "Landing Zone"
        direction TB
        GCS_pre["GCS Pre-Processed Bucket"]:::storage
    end

    %% Processing Layer
    subgraph "Processing Layer"
        direction TB
        VM["Compute Engine VM (cron & Superset)"]:::compute
        subgraph "Dataproc Load Jobs"
            direction TB
            DL_crashes["Load Crashes PySpark"]:::script
            DL_taxi_py["Load Taxi PySpark"]:::script
            DL_taxi_beam["Load Taxi Beam"]:::script
            DL_traffic_py["Load Traffic PySpark"]:::script
            DL_traffic_beam["Load Traffic Beam"]:::script
            DL_wpv["Load Weather/Persons/Vehicles PySpark"]:::script
        end
        subgraph "Dataproc Transform Jobs"
            direction TB
            DT_crashes["Transform Crashes"]:::script
            DT_persons["Transform Persons"]:::script
            DT_taxi_py["Transform Taxi (Py)"]:::script
            DT_taxi_sql["Transform Taxi (SQL)"]:::script
            DT_traffic["Transform Traffic"]:::script
            DT_vehicles["Transform Vehicles"]:::script
            DT_weather["Transform Weather"]:::script
        end
    end

    %% Storage Raw and Production
    subgraph "BigQuery"
        direction TB
        BQ_raw["Raw Dataset"]:::storage
        BQ_prd["Production Dataset"]:::storage
        DCatalog["Data Catalog"]:::gcp
    end

    %% Analysis Layer
    subgraph "Analysis Layer"
        direction TB
        Superset["Superset Dashboard"]:::compute
        AQ["Analytical Queries"]:::script
        CM["Combined Data Model"]:::script
        PA["Potential Analysis"]:::script
    end

    %% Infra
    subgraph "Infrastructure as Code"
        direction TB
        TF_main["main.tf"]:::script
        TF_vars["variables.tf"]:::script
        Infra_helper["create_infra.py"]:::script
        Constants["constants.py"]:::script
        Setup["setup.sh"]:::script
        Cron_load["run_load_to_raw.sh"]:::script
        Cron_tr["run_transformations.sh"]:::script
    end

    %% Data Flow
    Scheduler -->|triggers| CF_crashes
    Scheduler -->|triggers| CF_persons
    Scheduler -->|triggers| CF_taxi
    Scheduler -->|triggers| CF_traffic
    Scheduler -->|triggers| CF_vehicles
    Scheduler -->|triggers| CF_weather

    CF_crashes -->|writes CSV/JSON| GCS_pre
    CF_persons -->|writes CSV/JSON| GCS_pre
    CF_taxi -->|writes CSV/JSON| GCS_pre
    CF_traffic -->|writes CSV/JSON| GCS_pre
    CF_vehicles -->|writes CSV/JSON| GCS_pre
    CF_weather -->|writes CSV/JSON| GCS_pre

    VM -->|cron starts| Cron_load
    Cron_load -->|submits| DL_crashes
    Cron_load -->|submits| DL_taxi_py
    Cron_load -->|submits| DL_taxi_beam
    Cron_load -->|submits| DL_traffic_py
    Cron_load -->|submits| DL_traffic_beam
    Cron_load -->|submits| DL_wpv

    DL_crashes -->|writes| BQ_raw
    DL_taxi_py -->|writes| BQ_raw
    DL_taxi_beam -->|writes| BQ_raw
    DL_traffic_py -->|writes| BQ_raw
    DL_traffic_beam -->|writes| BQ_raw
    DL_wpv -->|writes| BQ_raw

    VM -->|cron starts| Cron_tr
    Cron_tr -->|submits| DT_crashes
    Cron_tr -->|submits| DT_persons
    Cron_tr -->|submits| DT_taxi_py
    Cron_tr -->|submits| DT_taxi_sql
    Cron_tr -->|submits| DT_traffic
    Cron_tr -->|submits| DT_vehicles
    Cron_tr -->|submits| DT_weather

    DT_crashes -->|writes| BQ_prd
    DT_persons -->|writes| BQ_prd
    DT_taxi_py -->|writes| BQ_prd
    DT_taxi_sql -->|writes| BQ_prd
    DT_traffic -->|writes| BQ_prd
    DT_vehicles -->|writes| BQ_prd
    DT_weather -->|writes| BQ_prd

    BQ_prd -->|register metadata| DCatalog
    Superset -->|queries| BQ_prd
    AQ -->|runs on| BQ_prd
    CM -->|runs on| BQ_prd
    PA -->|runs on| BQ_prd

    Infra_helper -->|used in| TF_main
    TF_main -->|defines| Scheduler
    TF_main -->|defines| CF_crashes
    TF_main -->|defines| CF_persons
    TF_main -->|defines| CF_taxi
    TF_main -->|defines| CF_traffic
    TF_main -->|defines| CF_vehicles
    TF_main -->|defines| CF_weather
    TF_main -->|defines| GCS_pre

    %% Click Events
    click CF_crashes "https://github.com/sagar8080/data-fusion-engineering/blob/main/ingest/crashes_data/main.py"
    click CF_persons "https://github.com/sagar8080/data-fusion-engineering/blob/main/ingest/persons_data/main.py"
    click CF_taxi "https://github.com/sagar8080/data-fusion-engineering/blob/main/ingest/taxi_data/main.py"
    click CF_traffic "https://github.com/sagar8080/data-fusion-engineering/blob/main/ingest/traffic_data/main.py"
    click CF_vehicles "https://github.com/sagar8080/data-fusion-engineering/blob/main/ingest/vehicles_data/main.py"
    click CF_weather "https://github.com/sagar8080/data-fusion-engineering/blob/main/ingest/weather_data/main.py"

    click DL_crashes "https://github.com/sagar8080/data-fusion-engineering/blob/main/load/load_crashes_data_pyspark.py"
    click DL_taxi_py "https://github.com/sagar8080/data-fusion-engineering/blob/main/load/load_taxi_data_pyspark.py"
    click DL_taxi_beam "https://github.com/sagar8080/data-fusion-engineering/blob/main/load/load_taxi_data_beam.py"
    click DL_traffic_py "https://github.com/sagar8080/data-fusion-engineering/blob/main/load/load_traffic_data_pyspark.py"
    click DL_traffic_beam "https://github.com/sagar8080/data-fusion-engineering/blob/main/load/load_traffic_data_beam.py"
    click DL_wpv "https://github.com/sagar8080/data-fusion-engineering/blob/main/load/load_weather_persons_vehicles_data.py"

    click DT_crashes "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_crashes_data.py"
    click DT_persons "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_persons_data.py"
    click DT_taxi_py "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_taxi_data.py"
    click DT_taxi_sql "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_taxi_data.SQL"
    click DT_traffic "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_traffic_data.py"
    click DT_vehicles "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_vehicles_data.py"
    click DT_weather "https://github.com/sagar8080/data-fusion-engineering/blob/main/transform/transform_weather_data.py"

    click AQ "https://github.com/sagar8080/data-fusion-engineering/blob/main/analyze/analytical_queries.sql"
    click CM "https://github.com/sagar8080/data-fusion-engineering/blob/main/analyze/combined_data_model.sql"
    click PA "https://github.com/sagar8080/data-fusion-engineering/blob/main/analyze/potential_analysis.sql"

    click TF_main "https://github.com/sagar8080/data-fusion-engineering/blob/main/main.tf"
    click TF_vars "https://github.com/sagar8080/data-fusion-engineering/blob/main/variables.tf"
    click Infra_helper "https://github.com/sagar8080/data-fusion-engineering/blob/main/utils/create_infra.py"
    click Constants "https://github.com/sagar8080/data-fusion-engineering/blob/main/utils/constants.py"
    click Setup "https://github.com/sagar8080/data-fusion-engineering/blob/main/setup.sh"
    click Cron_load "https://github.com/sagar8080/data-fusion-engineering/blob/main/run_load_to_raw.sh"
    click Cron_tr "https://github.com/sagar8080/data-fusion-engineering/blob/main/run_transformations.sh"

    %% Styles
    classDef gcp fill:#dae8fc,stroke:#6c8ebf,color:#0b5394
    classDef compute fill:#d5e8d4,stroke:#82b366,color:#274e13
    classDef storage fill:#fff2cc,stroke:#d6b656,color:#7f6000
    classDef script fill:#f8cecc,stroke:#b85450,color:#a61c17
    class CF_crashes,CF_persons,CF_taxi,CF_traffic,CF_vehicles,CF_weather gcp
    class Scheduler,DCatalog gcp
    class VM,Superset compute
    class GCS_pre,BQ_raw,BQ_prd storage
    class DL_crashes,DL_taxi_py,DL_taxi_beam,DL_traffic_py,DL_traffic_beam,DL_wpv script
    class DT_crashes,DT_persons,DT_taxi_py,DT_taxi_sql,DT_traffic,DT_vehicles,DT_weather script
    class AQ,CM,PA script
    class TF_main,TF_vars,Infra_helper,Constants,Setup,Cron_load,Cron_tr script
```
