# kvj-harsha/earthquakepipeline architecture

> Architecture diagram of the GitHub repository kvj-harsha/earthquakepipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kvj-harsha/earthquakepipeline
- Repository: https://github.com/kvj-harsha/earthquakepipeline

## Overview

##### File structure
- Top-level folders:
- notebooks/ – contains PySpark notebooks for Bronze, Silver, Gold layers
- pipeline/ – contains the Azure Data Factory (ADF) pipeline JSON
- synapse/ – contains the Synapse ingestion script
- powerbi/ – contains the Power BI dashboard files
- Key config files: pipeline/adf_pipeline.json, synapse/synapse_ingestion.sql
- Notebooks follow the Medallion layering: bronze_notebook, silver_notebook, gold_notebook

##### README
- Lists technologies: Azure Data Factory, Databricks, Data Lake Gen2, Synapse Analytics, Delta Lake, Power BI
- Notes orchestration (ADF triggers Databricks jobs) and data transformations

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Source
    USGS["USGS Earthquake API"]

    %% Orchestration
    ADF["ADF Pipeline"]:::orchestration
    click ADF "https://github.com/kvj-harsha/earthquakepipeline/blob/main/pipeline/pipeline1.json"

    %% Databricks Compute and Notebooks
    subgraph "Azure Databricks Cluster"
        direction TB
        BNb["Bronze Notebook"]:::bronze
        SNb["Silver Notebook"]:::silver
        GNb["Gold Notebook"]:::gold
    end
    click BNb "https://github.com/kvj-harsha/earthquakepipeline/blob/main/notebooks/Bronze Notebook.ipynb"
    click SNb "https://github.com/kvj-harsha/earthquakepipeline/blob/main/notebooks/Silver Notebook.ipynb"
    click GNb "https://github.com/kvj-harsha/earthquakepipeline/blob/main/notebooks/Gold Notebook.ipynb"

    %% Data Lake Storage
    subgraph "Azure Data Lake Storage Gen2"
        direction TB
        DLbronze["Bronze Container"]:::bronze
        DLsilver["Silver Container"]:::silver
        DLgold["Gold Container"]:::gold
    end

    %% Synapse Layer
    subgraph "Azure Synapse Analytics"
        direction TB
        SynSQL["SQL Endpoint"]:::serve
        SynDB["Synapse Tables"]:::storage
    end
    click SynSQL "https://github.com/kvj-harsha/earthquakepipeline/blob/main/synapse/synapse_ingestion.sql"

    %% Visualization
    subgraph "Power BI"
        direction TB
        PBIDB["Dashboard (.pbix)"]:::serve
        PBIT["Template (.pbit)"]:::serve
    end
    click PBIDB "https://github.com/kvj-harsha/earthquakepipeline/blob/main/powerbi/dashboard.pbix"
    click PBIT "https://github.com/kvj-harsha/earthquakepipeline/blob/main/powerbi/earthquake_dashboard.pbit"

    %% Documentation
    README["README.md"]:::docs
    click README "https://github.com/kvj-harsha/earthquakepipeline/blob/main/README.md"

    %% Data Flow
    USGS -->|"Daily Trigger"| ADF
    ADF -->|"Trigger Databricks Jobs"| BNb
    BNb -->|"Raw JSON to Bronze"| DLbronze
    DLbronze -->|"Read & Clean"| SNb
    SNb -->|"Structured Data to Silver"| DLsilver
    DLsilver -->|"Aggregate Metrics"| GNb
    GNb -->|"Gold Tables to Lake"| DLgold
    DLgold -->|"COPY INTO"| SynSQL
    SynSQL -->|"Load into Tables"| SynDB
    SynDB -->|"Live Query"| PBIDB
    PBIT -.->|Template based| PBIDB

    %% References to docs
    ADF -.-> README
    BNb -.-> README
    SNb -.-> README
    GNb -.-> README

    %% Styles
    classDef bronze fill:#CD7F32,stroke:#333,stroke-width:1px
    classDef silver fill:#C0C0C0,stroke:#333,stroke-width:1px
    classDef gold fill:#FFD700,stroke:#333,stroke-width:1px
    classDef orchestration fill:#1E90FF,stroke:#333,stroke-width:1px
    classDef serve fill:#32CD32,stroke:#333,stroke-width:1px
    classDef storage fill:#87CEFA,stroke:#333,stroke-width:1px
    classDef docs fill:#EFEFEF,stroke:#999,stroke-width:1px
```
