# akshatdev2002/azure-databricks-lakehouse-data-engineering architecture

> Architecture diagram of the GitHub repository akshatdev2002/azure-databricks-lakehouse-data-engineering, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/akshatdev2002/azure-databricks-lakehouse-data-engineering
- Repository: https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering
- Video tour (about a minute): https://gitdiagram.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/video

## Overview

This repository documents an Azure Databricks lakehouse workflow: raw files are ingested into Bronze, refined into Silver datasets, then shaped into Gold analytics tables and a star schema. Silver includes customer, order, product, and region transformations; Gold documents customer SCD Type 1 and product SCD Type 2, alongside orders and analytics-ready modeling. Unity Catalog provides governance and table registration. The README describes these relationships, but notebook source was not available to verify finer-grained calls or table wiring.

Main entry points: [`Bronze-Layer.ipynb`](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Bronze-Layer.ipynb) (Bronze ingestion), [`Silver-Layer/Silver_Customers.ipynb`](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Customers.ipynb) (Silver customers), [`Gold-Layer/Gold_Customers.ipynb`](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Customers.ipynb) (Gold customers), [`Gold-Layer/Gold_Products.ipynb`](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Products.ipynb) (Gold products)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Bronze ingestion"]
  node_bronze["Bronze ingestion<br/>[Bronze-Layer.ipynb]"]
end

subgraph group_refinement["Silver refinement"]
  node_silver_customers["Silver customers"]
  node_silver_orders["Silver orders"]
  node_silver_products["Silver products"]
  node_silver_regions["Silver regions"]
end

subgraph group_analytics["Gold analytics"]
  node_gold_customers["Gold customers"]
  node_gold_orders["Gold orders<br/>[Gold_Orders.ipynb]"]
  node_gold_products["Gold products"]
  node_customer_dim[("Customers dimension")]
  node_orders_fact[("Orders fact<br/>[Gold_Orders.ipynb]")]
  node_products_dim[("Products dimension")]
  node_regions_dim[("Regions dimension")]
end

subgraph group_governance["Governance"]
  node_unity_catalog[("Unity Catalog")]
end

node_adls[("ADLS Gen2")]
node_bi_users(("BI users"))
node_databricks{{"Azure Databricks"}}

node_adls -->|"provides files"| node_bronze
node_bronze -->|"ingests customer data"| node_silver_customers
node_bronze -->|"ingests order data"| node_silver_orders
node_bronze -->|"ingests product data"| node_silver_products
node_bronze -->|"ingests region data"| node_silver_regions
node_silver_customers -->|"refines customer data"| node_gold_customers
node_silver_orders -->|"refines order data"| node_gold_orders
node_silver_products -->|"refines product data"| node_gold_products
node_gold_customers -->|"maintains SCD 1"| node_customer_dim
node_gold_orders -->|"produces fact"| node_orders_fact
node_gold_products -->|"maintains SCD 2"| node_products_dim
node_gold_customers -.->|"registers tables"| node_unity_catalog
node_gold_products -.->|"registers tables"| node_unity_catalog
node_silver_customers -.->|"registers tables"| node_unity_catalog
node_silver_regions -.->|"registers tables"| node_unity_catalog
node_databricks -->|"runs notebook"| node_bronze
node_databricks -->|"runs notebook"| node_silver_customers
node_databricks -->|"runs notebook"| node_silver_orders
node_databricks -->|"runs notebook"| node_silver_products
node_databricks -->|"runs notebook"| node_silver_regions
node_databricks -->|"runs notebook"| node_gold_customers
node_databricks -->|"runs notebook"| node_gold_orders
node_databricks -->|"runs notebook"| node_gold_products
node_unity_catalog -.->|"governs access"| node_bi_users
node_orders_fact -->|"supports analysis"| node_bi_users
node_customer_dim -->|"supports analysis"| node_bi_users
node_products_dim -->|"supports analysis"| node_bi_users
node_regions_dim -->|"supports analysis"| node_bi_users

click node_bronze "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Bronze-Layer.ipynb"
click node_silver_customers "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Customers.ipynb"
click node_silver_orders "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Orders.ipynb"
click node_silver_products "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Products.ipynb"
click node_silver_regions "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Regions.ipynb"
click node_gold_customers "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Customers.ipynb"
click node_gold_orders "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Orders.ipynb"
click node_gold_products "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Products.ipynb"
click node_customer_dim "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Customers.ipynb"
click node_orders_fact "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Orders.ipynb"
click node_products_dim "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Products.ipynb"
click node_regions_dim "https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/tree/main/Gold-Layer"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_bronze,node_bi_users toneBlue
class node_silver_customers,node_silver_orders,node_silver_products,node_silver_regions,node_adls toneAmber
class node_gold_customers,node_gold_orders,node_gold_products,node_customer_dim,node_orders_fact,node_products_dim,node_regions_dim toneMint
class node_unity_catalog toneRose
class node_databricks toneIndigo
```

## Components

### Bronze ingestion

- [Bronze ingestion](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Bronze-Layer.ipynb): `Bronze-Layer.ipynb`

### Silver refinement

- [Silver customers](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Customers.ipynb): `Silver-Layer/Silver_Customers.ipynb`
- [Silver orders](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Orders.ipynb): `Silver-Layer/Silver_Orders.ipynb`
- [Silver products](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Products.ipynb): `Silver-Layer/Silver_Products.ipynb`
- [Silver regions](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Silver-Layer/Silver_Regions.ipynb): `Silver-Layer/Silver_Regions.ipynb`

### Gold analytics

- [Gold customers](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Customers.ipynb): `Gold-Layer/Gold_Customers.ipynb`
- [Gold orders](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Orders.ipynb): `Gold-Layer/Gold_Orders.ipynb`
- [Gold products](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Products.ipynb): `Gold-Layer/Gold_Products.ipynb`
- [Customers dimension](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Customers.ipynb): `Gold-Layer/Gold_Customers.ipynb`
- [Orders fact](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Orders.ipynb): `Gold-Layer/Gold_Orders.ipynb`
- [Products dimension](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/blob/main/Gold-Layer/Gold_Products.ipynb): `Gold-Layer/Gold_Products.ipynb`
- [Regions dimension](https://github.com/akshatdev2002/azure-databricks-lakehouse-data-engineering/tree/main/Gold-Layer): `Gold-Layer`

### Governance

- **Unity Catalog**

### Other components

- **ADLS Gen2**
- **BI users**
- **Azure Databricks**

## Connections

- ADLS Gen2 → Bronze ingestion: provides files
- Bronze ingestion → Silver customers: ingests customer data
- Bronze ingestion → Silver orders: ingests order data
- Bronze ingestion → Silver products: ingests product data
- Bronze ingestion → Silver regions: ingests region data
- Silver customers → Gold customers: refines customer data
- Silver orders → Gold orders: refines order data
- Silver products → Gold products: refines product data
- Gold customers → Customers dimension: maintains SCD 1
- Gold orders → Orders fact: produces fact
- Gold products → Products dimension: maintains SCD 2
- Gold customers → Unity Catalog: registers tables
- Gold products → Unity Catalog: registers tables
- Silver customers → Unity Catalog: registers tables
- Silver regions → Unity Catalog: registers tables
- Azure Databricks → Bronze ingestion: runs notebook
- Azure Databricks → Silver customers: runs notebook
- Azure Databricks → Silver orders: runs notebook
- Azure Databricks → Silver products: runs notebook
- Azure Databricks → Silver regions: runs notebook
- Azure Databricks → Gold customers: runs notebook
- Azure Databricks → Gold orders: runs notebook
- Azure Databricks → Gold products: runs notebook
- Unity Catalog → BI users: governs access
- Orders fact → BI users: supports analysis
- Customers dimension → BI users: supports analysis
- Products dimension → BI users: supports analysis
- Regions dimension → BI users: supports analysis