# devanshiprajapati/uber_data_analytics architecture

> Architecture diagram of the GitHub repository devanshiprajapati/uber_data_analytics, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devanshiprajapati/uber_data_analytics
- Repository: https://github.com/devanshiprajapati/uber_data_analytics

## Overview

##### Main Components

a) Data Sources Layer:
- Source data (Uber TLC Trip Records)
- CSV files notation

b) Storage Layer:
- Google Cloud Storage bucket
- Connection to raw data files

c) Processing Layer (Mage ETL):
- Create three connected blocks representing:
- Data Loader (data-loader.py)
- Data Transformer (data-transformer.py)
- Data Exporter (data-exporter.py)

d) Data Warehouse Layer:
- Google BigQuery representation
- Modeled tables structure
- Fact and dimension tables

e) Visualization Layer:
- Looker Studio dashboard
- Connection to BigQuery

##### Component Relationships

- Raw Data → GCP Storage
- GCP Storage → Mage ETL Pipeline
- Mage Pipeline → BigQuery
- BigQuery → Looker Studio

##### Infrastructure Elements

A separate section showing:
- GCP VM Instance hosting Mage
- Network connections between services
- Security boundaries

##### Technical Details

For each component, add annotations showing:
- File formats (CSV)
- Data transformation steps
- Query processing
- Key metrics being calculated

**[Data Sources] → [Storage] → [Processing] → [Data Warehouse] → [Visualization]**

##### Additional Elements

- Component types

##### Notes Section

A small notes section indicating:
- Data refresh frequency
- Volume of data processed
- Key transformations performed

- The complete data flow from source to visualization
- The role of each GCP service
- The ETL pipeline structure
- How components interact with each other

Remember to emphasize the cloud-native nature of the solution and how each component in the file tree (particularly the Mage files) fits into the larger architecture.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Define styles
    classDef source fill:#f9f,stroke:#333,stroke-width:2px
    classDef storage fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef processing fill:#98FB98,stroke:#333,stroke-width:2px
    classDef warehouse fill:#4169E1,stroke:#333,stroke-width:2px
    classDef visualization fill:#FFA500,stroke:#333,stroke-width:2px
    classDef infrastructure fill:#D3D3D3,stroke:#333,stroke-width:2px

    %% Data Source Layer
    subgraph DataSource["Data Source Layer"]
        RawData["Uber TLC Trip Records (CSV)"]:::source
    end
    click RawData "https://github.com/DevanshiPrajapati/Uber_Data_Analytics/blob/main/data/uber_data.csv"

    %% Storage Layer
    subgraph Storage["Google Cloud Platform"]
        GCS["Google Cloud Storage"]:::storage
    end

    %% Processing Layer
    subgraph Processing["Mage ETL Pipeline"]
        Loader["Data Loader"]:::processing
        Transformer["Data Transformer"]:::processing
        Exporter["Data Exporter"]:::processing
    end
    click Loader "https://github.com/DevanshiPrajapati/Uber_Data_Analytics/blob/main/mage-files/data-loader.py"
    click Transformer "https://github.com/DevanshiPrajapati/Uber_Data_Analytics/blob/main/mage-files/data-transformer.py"
    click Exporter "https://github.com/DevanshiPrajapati/Uber_Data_Analytics/blob/main/mage-files/data-exporter.py"

    %% Data Warehouse Layer
    subgraph Warehouse["Data Warehouse"]
        BQ["BigQuery"]:::warehouse
        Queries["SQL Queries"]:::warehouse
    end
    click Queries "https://github.com/DevanshiPrajapati/Uber_Data_Analytics/blob/main/queries.txt"

    %% Visualization Layer
    subgraph Visualization["Analytics Layer"]
        Looker["Looker Studio Dashboard"]:::visualization
    end

    %% Infrastructure
    subgraph Infrastructure["Infrastructure"]
        VM["GCP VM Instance"]:::infrastructure
        Pipeline["Pipeline Documentation"]:::infrastructure
    end
    click Pipeline "https://github.com/DevanshiPrajapati/Uber_Data_Analytics/blob/main/UberDataPipeline.ipynb"

    %% Relationships
    RawData --> GCS
    GCS --> Loader
    Loader --> Transformer
    Transformer --> Exporter
    Exporter --> BQ
    BQ --> Queries
    BQ --> Looker
    VM --> Processing

    %% Legend
    subgraph Legend["Legend"]
        L1["Data Source"]:::source
        L2["Storage Component"]:::storage
        L3["Processing Step"]:::processing
        L4["Data Warehouse"]:::warehouse
        L5["Visualization"]:::visualization
        L6["Infrastructure"]:::infrastructure
    end
```
