# abhishekshah25/3-layer-medallion-data-pipeline architecture

> Architecture diagram of the GitHub repository abhishekshah25/3-layer-medallion-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abhishekshah25/3-layer-medallion-data-pipeline
- Repository: https://github.com/abhishekshah25/3-layer-medallion-data-pipeline

## Overview

##### Project type and purpose
- This is a data-engineering project implementing a three-layer Medallion Architecture pipeline on Azure.
- It uses Azure Data Factory for orchestration, Azure Databricks for Spark-based processing, and DBT for modular SQL transformations.

##### File structure
- Top-level:
- Procedure.pdf: setup/run instructions
- databricks_base_notebook.py: base code for Databricks notebooks
- medallion_dbt_spark/: DBT project
- medallion_dbt_spark/ contains:
- dbt_project.yml: project configuration
- models/staging/: “bronze” raw data staging layer
- models/marts/…/dim_*.sql, sales.sql: “gold” presentation-ready marts
- macros/, analyses/, seeds/, snapshots/: DBT features for reusable code, ad-hoc analysis, static data, change-data-capture snapshots
- No frontend or backend directories – it’s purely an ETL pipeline.

##### README for additional insights
- Confirms a 3-layer system (bronze/silver/gold) on Azure
- Highlights Azure Data Factory, Databricks, DBT components
- Emphasizes modularity, scalability, automated workflows, version control

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    DataSources["Data Sources\n(on-prem, APIs, files)"]:::external

    %% Orchestration Layer
    subgraph "Orchestration"
        ADF["Azure Data Factory\n(Orchestration)"]:::azure
    end

    %% Bronze Layer Storage
    Bronze["Bronze Zone\n(Raw Data Landing)"]:::storage

    %% Compute Layer
    Databricks["Azure Databricks\n(Spark Notebooks)"]:::compute

    %% Silver Layer Storage
    Silver["Silver Zone\n(CDC/Snapshot Tables)"]:::storage

    %% DBT Layer
    subgraph "DBT on Databricks"
        DBTCompute["DBT on Databricks\n(Modular SQL Transforms)"]:::dbtCompute
        DBTConfig["DBT Project Config\ndbt_project.yml"]:::dbtConfig
        subgraph "DBT Project Structure"
            Staging["Staging Models\n(Bronze Layer)"]:::dbtModels
            Snapshots["Snapshots\n(CDC Tables)"]:::dbtModels
            Marts["Gold Marts\n(Presentation Layer)"]:::dbtModels
            Customer["dim_customer.sql"]:::mart
            Product["dim_product.sql"]:::mart
            Sales["sales.sql"]:::mart
            Macros["Macros\n(reusable SQL logic)"]:::dbtSupport
            Seeds["Seeds\n(static reference data)"]:::dbtSupport
            Analyses["Analyses\n(ad-hoc queries)"]:::dbtSupport
            Tests["Tests\n(model validations)"]:::dbtSupport
        end
    end

    %% Gold & Consumption
    Gold["Gold Zone\n(Presentation Marts)"]:::storage
    BI["BI & Downstream\n(Power BI, Synapse)"]:::bi

    %% Documentation
    Doc["README.md\n(Project Docs)"]:::doc

    %% Data Flow
    DataSources --> ADF
    ADF --> Bronze
    Bronze --> Databricks
    Databricks --> Silver
    Silver --> DBTCompute
    DBTCompute --> Marts
    Marts --> Gold
    Gold --> BI

    %% Click Events
    click ADF "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/Procedure.pdf"
    click Databricks "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/databricks_base_notebook.py"
    click DBTConfig "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/medallion_dbt_spark/dbt_project.yml"
    click Staging "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/models/staging"
    click Snapshots "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/snapshots"
    click Marts "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/models/marts"
    click Customer "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/medallion_dbt_spark/models/marts/customer/dim_customer.sql"
    click Product "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/medallion_dbt_spark/models/marts/product/dim_product.sql"
    click Sales "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/medallion_dbt_spark/models/marts/sales/sales.sql"
    click Macros "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/macros"
    click Seeds "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/seeds"
    click Analyses "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/analyses"
    click Tests "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/tree/main/medallion_dbt_spark/tests"
    click Doc "https://github.com/abhishekshah25/3-layer-medallion-data-pipeline/blob/main/README.md"

    %% Styles
    classDef azure fill:#007FFF,stroke:#005F9E,color:#fff
    classDef storage fill:#8FBC8F,stroke:#6E8B3D,color:#000
    classDef compute fill:#FFA500,stroke:#CC8400,color:#000
    classDef dbtCompute fill:#228B22,stroke:#196619,color:#fff
    classDef dbtConfig fill:#ADFF2F,stroke:#7FBF17,color:#000
    classDef dbtModels fill:#90EE90,stroke:#60C060,color:#000
    classDef dbtSupport fill:#98FB98,stroke:#70C070,color:#000
    classDef mart fill:#32CD32,stroke:#28A428,color:#fff
    classDef bi fill:#9370DB,stroke:#6654A6,color:#fff
    classDef external fill:#778899,stroke:#556B2F,color:#fff
    classDef doc fill:#D3D3D3,stroke:#A9A9A9,color:#000
```
