How akshatdev2002/azure-databricks-lakehouse-data-engineering works
This repository documents an Azure Databricks lakehouse workflow: raw files are ingested into Bronze, refined into Silver datasets, then shaped into Gold analytics tables and a star schema. Silver includes customer, order, product, and region transformations; Gold documents customer SCD Type 1 and product SCD Type 2, alongside orders and analytics-ready modeling. Unity Catalog provides governance and table registration. The README describes these relationships, but notebook source was not available to verify finer-grained calls or table wiring.
Main entry points: Bronze-Layer.ipynb Bronze ingestion · Silver_Customers.ipynb Silver customers · Gold_Customers.ipynb Gold customers · Gold_Products.ipynb Gold products
Components 16
Bronze ingestion
- Bronze ingestion
Bronze-Layer.ipynb
Silver refinement
- Silver customers
Silver-Layer/Silver_Customers.ipynb - Silver orders
Silver-Layer/Silver_Orders.ipynb - Silver products
Silver-Layer/Silver_Products.ipynb - Silver regions
Silver-Layer/Silver_Regions.ipynb
Gold analytics
- Gold customers
Gold-Layer/Gold_Customers.ipynb - Gold orders
Gold-Layer/Gold_Orders.ipynb - Gold products
Gold-Layer/Gold_Products.ipynb - Customers dimension
Gold-Layer/Gold_Customers.ipynb - Orders fact
Gold-Layer/Gold_Orders.ipynb - Products dimension
Gold-Layer/Gold_Products.ipynb - Regions dimension
Gold-Layer
Governance
- Unity Catalog
Other components
- ADLS Gen2
- BI users
- Azure Databricks
Connections 28
- ADLS Gen2 → Bronze ingestion: provides files
- Bronze ingestion → Silver customers: ingests customer data
- Bronze ingestion → Silver orders: ingests order data
- Bronze ingestion → Silver products: ingests product data
- Bronze ingestion → Silver regions: ingests region data
- Silver customers → Gold customers: refines customer data
- Silver orders → Gold orders: refines order data
- Silver products → Gold products: refines product data
- Gold customers → Customers dimension: maintains SCD 1
- Gold orders → Orders fact: produces fact
- Gold products → Products dimension: maintains SCD 2
- Gold customers → Unity Catalog: registers tables
- Gold products → Unity Catalog: registers tables
- Silver customers → Unity Catalog: registers tables
- Silver regions → Unity Catalog: registers tables
- Azure Databricks → Bronze ingestion: runs notebook
- Azure Databricks → Silver customers: runs notebook
- Azure Databricks → Silver orders: runs notebook
- Azure Databricks → Silver products: runs notebook
- Azure Databricks → Silver regions: runs notebook
- Azure Databricks → Gold customers: runs notebook
- Azure Databricks → Gold orders: runs notebook
- Azure Databricks → Gold products: runs notebook
- Unity Catalog → BI users: governs access
- Orders fact → BI users: supports analysis
- Customers dimension → BI users: supports analysis
- Products dimension → BI users: supports analysis
- Regions dimension → BI users: supports analysis