# darshilparmar/uber-etl-pipeline-data-engineering-project architecture

> Architecture diagram of the GitHub repository darshilparmar/uber-etl-pipeline-data-engineering-project, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darshilparmar/uber-etl-pipeline-data-engineering-project
- Repository: https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project

## Overview

##### Main Components
- Data Source: The CSV file(s) under the data folder, representing Uber data.
- ETL Pipeline: The mage-files directory highlights the three-stage process:
- Extract (extract.py)
- Transform (transform.py)
- Load (load.py)
- Compute Instance: The GCP Compute Instance where Python and ETL scripts are executed.
- Cloud Storage: Google Storage used for staging or raw data storage.
- Data Warehouse: BigQuery for storing and querying processed data.
- Data Visualization: Looker Studio for analytics dashboards.
- Notebooks: Uber Data Pipeline notebooks that likely demonstrate or validate the pipeline operation and data analytics queries.
- SQL Analysis: The analytics_query.sql file that represents custom querying on the final data store.

##### Relationships and Interactions
- The flow begins with raw data (Uber data CSV) being pulled (extracted) using the extract.py script.
- Next, data transformation occurs using transform.py, preparing the data for loading.
- The load.py script then pushes the processed data into BigQuery either via an intermediary (GCS) or directly.
- The Compute Instance is the execution environment that runs these ETL processes.
- Once loaded into BigQuery, data is made available for analysis, including running SQL queries (analytics_query.sql) and generating visualizations in Looker Studio.
- The notebooks (Uber Data Pipeline notebooks) tie these stages together by either documenting experiments, presenting sample ETL runs, or validating the process.

##### Architectural Patterns or Design Principles
- ETL Pipeline: A clear separation of concerns is used by dividing the tasks into extraction, transformation, and loading.
- Cloud-Native Architecture: Using GCP components, the overall design is modular, scalable, and leverages managed services.
- Integration of Open-source Tools: The project uses Mage (a modern data pipeline tool) that drives the ETL process.
- Data Warehouse Design: The processed data is centralized in BigQuery which acts as the analytical backend.

##### Relevant Technologies, Frameworks, or Libraries
- Programming Language: Python (used in the ETL scripts and notebooks)
- Mage Data Pipeline Tool used for orchestrating ETL
- Google Cloud Platform components: Google Storage, Compute Instance, BigQuery
- Looker Studio for BI and visualization tasks
- SQL for querying data in the warehouse

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    DS["Uber CSV"]:::datasource

    %% ETL Pipeline Subgraph
    subgraph "ETL Pipeline"
        EX["Extract (mage-files/extract.py)"]:::pipeline
        TR["Transform (mage-files/transform.py)"]:::pipeline
        LD["Load (mage-files/load.py)"]:::pipeline
    end

    %% GCP Infrastructure Subgraph
    subgraph "GCP Infrastructure"
        CI["Compute Instance"]:::compute
        GS["Google Storage"]:::storage
        BQ["BigQuery"]:::db
    end

    %% Analytics & Reporting Subgraph
    subgraph "Analytics & Reporting"
        LS["Looker Studio"]:::analytics
        NB1["Uber Data Pipeline (Fixed Version) Notebook"]:::analytics
        NB2["Uber Data Pipeline (Video Version) Notebook"]:::analytics
        SQL["SQL Analysis (analytics_query.sql)"]:::analytics
    end

    %% Data Flow
    DS -->|"extracts"| EX
    EX -->|"transforms"| TR
    TR -->|"loads"| LD
    LD -->|"stages data"| GS
    GS -->|"stores data"| BQ
    BQ -->|"visualizes"| LS
    BQ -->|"query results"| SQL
    BQ -->|"demonstrates data"| NB1
    BQ -->|"demonstrates data"| NB2

    %% Execution Environment Connections (dashed to indicate control/trigger)
    CI -.-> EX
    CI -.-> TR
    CI -.-> LD

    %% Styles
    classDef datasource fill:#ffcccc,stroke:#b30000,stroke-width:2px;
    classDef pipeline fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef compute fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef storage fill:#e2e3e5,stroke:#383d41,stroke-width:2px;
    classDef db fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef analytics fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;

    %% Click Events
    click DS "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/data/uber_data.csv"
    click EX "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/mage-files/extract.py"
    click TR "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/mage-files/transform.py"
    click LD "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/mage-files/load.py"
    click NB1 "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/Uber Data Pipeline (Fixed Version).ipynb"
    click NB2 "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/Uber Data Pipeline (Video Version).ipynb"
    click SQL "https://github.com/darshilparmar/uber-etl-pipeline-data-engineering-project/blob/main/analytics_query.sql"
```
