How hruday-45/olist-azure-medallion-lakehouse works
This repository describes an Azure Olist lakehouse: ADF orchestrates CSV ingestion into Bronze, PySpark transformations into Silver, and a curated Gold star schema for analytics. The documented control flow includes a master pipeline, a sequential sub-orchestrator, notebook workers, parallel Gold processing, and failure alerts through a Logic App. Source excerpts were not provided, so relationships below are limited to those stated in the README; notebook-to-table details are not inferred.
Main entry points: pl_orchestrate_medallion.json Layer Orchestrator · notebooks Databricks Notebooks · pl_master_olist_daily.json Master Pipeline · pl_load_bronze_raw.json Bronze Worker
Components 16
ADF Orchestration
- Master Pipeline
pipelines/pl_master_olist_daily.json - Layer Orchestrator
pipelines/pl_orchestrate_medallion.json - Bronze Worker
pipelines/pl_load_bronze_raw.json - Silver Worker
pipelines/pl_transform_silver_standardized.json - Gold Worker
pipelines/pl_curated_gold_star_schema.json
Medallion Data Layers
- Databricks Notebooks
notebooks - Bronze Raw
notebooks/Ingesting_source_in_bronze_tables.ipynb - Silver Standardized
notebooks/silver_dimensions_scd2.ipynb - Silver Fact Upserts
notebooks/silver_facts_upsert.ipynb - Gold Star Schema
notebooks/fact_sales.ipynb - Gold Dimensions
notebooks/dim_customers.ipynb
Lakehouse Storage
- ADLS Gen2
Failure Alerting
- Azure Logic App
logi-app/workflow_definition.json - Alert Email
logi-app/alert_template.html
Other components
- Olist CSVs
- Data Team
Connections 16
- Master Pipeline → Layer Orchestrator: triggers
README.md - Layer Orchestrator → Bronze Worker: runs first
README.md - Layer Orchestrator → Silver Worker: runs next
README.md - Layer Orchestrator → Gold Worker: runs last
README.md - Bronze Worker → Databricks Notebooks: executes notebooks
README.md - Silver Worker → Databricks Notebooks: executes notebooks
README.md - Gold Worker → Databricks Notebooks: executes in parallel
README.md - Olist CSVs → Bronze Raw: ingests
README.md - Bronze Raw → ADLS Gen2: stores raw data
README.md - Silver Standardized → ADLS Gen2: writes cleaned data
README.md - Silver Fact Upserts → ADLS Gen2: upserts data
- Gold Star Schema → ADLS Gen2: curates tables
README.md - Gold Dimensions → ADLS Gen2: curates tables
- Master Pipeline → Azure Logic App: sends failure alert
README.md - Azure Logic App → Alert Email: formats notification
README.md - Alert Email → Data Team: notifies
README.md