# hruday-45/olist-azure-medallion-lakehouse architecture

> Architecture diagram of the GitHub repository hruday-45/olist-azure-medallion-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hruday-45/olist-azure-medallion-lakehouse
- Repository: https://github.com/hruday-45/olist-azure-medallion-lakehouse

## Overview

This repository describes an Azure Olist lakehouse: ADF orchestrates CSV ingestion into Bronze, PySpark transformations into Silver, and a curated Gold star schema for analytics. The documented control flow includes a master pipeline, a sequential sub-orchestrator, notebook workers, parallel Gold processing, and failure alerts through a Logic App. Source excerpts were not provided, so relationships below are limited to those stated in the README; notebook-to-table details are not inferred.

Main entry points: [`pipelines/pl_orchestrate_medallion.json`](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_orchestrate_medallion.json) (Layer Orchestrator), [`notebooks`](https://github.com/hruday-45/olist-azure-medallion-lakehouse/tree/main/notebooks) (Databricks Notebooks), [`pipelines/pl_master_olist_daily.json`](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_master_olist_daily.json) (Master Pipeline), [`pipelines/pl_load_bronze_raw.json`](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_load_bronze_raw.json) (Bronze Worker)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["ADF Orchestration"]
  node_master["Master Pipeline"]
  node_sub["Layer Orchestrator"]
  node_bronze_worker["Bronze Worker"]
  node_silver_worker["Silver Worker"]
  node_gold_worker["Gold Worker"]
end

subgraph group_layers["Medallion Data Layers"]
  node_notebooks["Databricks Notebooks"]
  node_bronze["Bronze Raw"]
  node_silver["Silver Standardized"]
  node_silver_facts["Silver Fact Upserts"]
  node_gold["Gold Star Schema<br/>[fact_sales.ipynb]"]
  node_gold_dimensions["Gold Dimensions"]
end

subgraph group_storage["Lakehouse Storage"]
  node_adls[("ADLS Gen2")]
end

subgraph group_alerts["Failure Alerting"]
  node_logic_app["Azure Logic App"]
  node_email["Alert Email"]
end

node_csv["Olist CSVs"]
node_data_team(("Data Team"))

node_master -->|"triggers"| node_sub
node_sub -->|"runs first"| node_bronze_worker
node_sub -->|"runs next"| node_silver_worker
node_sub -->|"runs last"| node_gold_worker
node_bronze_worker -->|"executes notebooks"| node_notebooks
node_silver_worker -->|"executes notebooks"| node_notebooks
node_gold_worker -->|"executes in parallel"| node_notebooks
node_csv -->|"ingests"| node_bronze
node_bronze -->|"stores raw data"| node_adls
node_silver -->|"writes cleaned data"| node_adls
node_silver_facts -.->|"upserts data"| node_adls
node_gold -->|"curates tables"| node_adls
node_gold_dimensions -.->|"curates tables"| node_adls
node_master -->|"sends failure alert"| node_logic_app
node_logic_app -->|"formats notification"| node_email
node_email -->|"notifies"| node_data_team

click node_master "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_master_olist_daily.json"
click node_sub "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_orchestrate_medallion.json"
click node_bronze_worker "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_load_bronze_raw.json"
click node_silver_worker "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_transform_silver_standardized.json"
click node_gold_worker "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_curated_gold_star_schema.json"
click node_notebooks "https://github.com/hruday-45/olist-azure-medallion-lakehouse/tree/main/notebooks"
click node_bronze "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/Ingesting_source_in_bronze_tables.ipynb"
click node_silver "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/silver_dimensions_scd2.ipynb"
click node_silver_facts "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/silver_facts_upsert.ipynb"
click node_gold "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/fact_sales.ipynb"
click node_gold_dimensions "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/dim_customers.ipynb"
click node_logic_app "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/logi-app/workflow_definition.json"
click node_email "https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/logi-app/alert_template.html"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_master,node_sub,node_bronze_worker,node_silver_worker,node_gold_worker toneBlue
class node_notebooks,node_bronze,node_silver,node_silver_facts,node_gold,node_gold_dimensions toneAmber
class node_adls toneMint
class node_logic_app,node_email toneRose
class node_csv,node_data_team toneIndigo
```

## Components

### ADF Orchestration

- [Master Pipeline](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_master_olist_daily.json): `pipelines/pl_master_olist_daily.json`
- [Layer Orchestrator](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_orchestrate_medallion.json): `pipelines/pl_orchestrate_medallion.json`
- [Bronze Worker](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_load_bronze_raw.json): `pipelines/pl_load_bronze_raw.json`
- [Silver Worker](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_transform_silver_standardized.json): `pipelines/pl_transform_silver_standardized.json`
- [Gold Worker](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/pipelines/pl_curated_gold_star_schema.json): `pipelines/pl_curated_gold_star_schema.json`

### Medallion Data Layers

- [Databricks Notebooks](https://github.com/hruday-45/olist-azure-medallion-lakehouse/tree/main/notebooks): `notebooks`
- [Bronze Raw](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/Ingesting_source_in_bronze_tables.ipynb): `notebooks/Ingesting_source_in_bronze_tables.ipynb`
- [Silver Standardized](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/silver_dimensions_scd2.ipynb): `notebooks/silver_dimensions_scd2.ipynb`
- [Silver Fact Upserts](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/silver_facts_upsert.ipynb): `notebooks/silver_facts_upsert.ipynb`
- [Gold Star Schema](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/fact_sales.ipynb): `notebooks/fact_sales.ipynb`
- [Gold Dimensions](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/notebooks/dim_customers.ipynb): `notebooks/dim_customers.ipynb`

### Lakehouse Storage

- **ADLS Gen2**

### Failure Alerting

- [Azure Logic App](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/logi-app/workflow_definition.json): `logi-app/workflow_definition.json`
- [Alert Email](https://github.com/hruday-45/olist-azure-medallion-lakehouse/blob/main/logi-app/alert_template.html): `logi-app/alert_template.html`

### Other components

- **Olist CSVs**
- **Data Team**

## Connections

- Master Pipeline → Layer Orchestrator: triggers (evidence: `README.md`)
- Layer Orchestrator → Bronze Worker: runs first (evidence: `README.md`)
- Layer Orchestrator → Silver Worker: runs next (evidence: `README.md`)
- Layer Orchestrator → Gold Worker: runs last (evidence: `README.md`)
- Bronze Worker → Databricks Notebooks: executes notebooks (evidence: `README.md`)
- Silver Worker → Databricks Notebooks: executes notebooks (evidence: `README.md`)
- Gold Worker → Databricks Notebooks: executes in parallel (evidence: `README.md`)
- Olist CSVs → Bronze Raw: ingests (evidence: `README.md`)
- Bronze Raw → ADLS Gen2: stores raw data (evidence: `README.md`)
- Silver Standardized → ADLS Gen2: writes cleaned data (evidence: `README.md`)
- Silver Fact Upserts → ADLS Gen2: upserts data
- Gold Star Schema → ADLS Gen2: curates tables (evidence: `README.md`)
- Gold Dimensions → ADLS Gen2: curates tables
- Master Pipeline → Azure Logic App: sends failure alert (evidence: `README.md`)
- Azure Logic App → Alert Email: formats notification (evidence: `README.md`)
- Alert Email → Data Team: notifies (evidence: `README.md`)
