# nabilkendri/lakehouse-forge architecture

> Architecture diagram of the GitHub repository nabilkendri/lakehouse-forge, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nabilkendri/lakehouse-forge
- Repository: https://github.com/nabilkendri/lakehouse-forge

## Overview

Lakehouse Forge is a Databricks/PySpark ETL learning project. Its principal workflow ingests SQL orders and JSON web events into Bronze, cleans and joins them in Silver while retaining rejected events, validates Silver, and publishes customer metrics in Gold. The README documents the notebook stages and reusable validation module; sampled source also shows the web-event cleaning and rejection functions. Notebook internals were not sampled, so links between their stages are based on the README.

Main entry points: [`notebooks/11_transform_silver.ipynb`](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb) (Silver transformation), [`notebooks/12_validate_silver.ipynb`](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/12_validate_silver.ipynb) (Silver validation), [`notebooks/10_ingest_bronze.ipynb`](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/10_ingest_bronze.ipynb) (Bronze ingestion), [`notebooks/13_build_gold.ipynb`](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/13_build_gold.ipynb) (Gold aggregation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_sources["Data sources"]
  node_orders_source[("Sample orders")]
  node_events_source["Web events<br/>[web_events.jsonl]"]
end

subgraph group_pipeline["ETL pipeline"]
  node_bronze["Bronze ingestion"]
  node_bronze_tables[("Bronze tables")]
  node_silver["Silver transformation"]
  node_event_cleaning["Event cleaning<br/>[transform.py]"]
  node_clean_events[("Clean events")]
  node_rejected_events[("Rejected events")]
  node_clean_orders[("Clean orders")]
  node_customer_events[("Customer events")]
  node_gold["Gold aggregation"]
  node_customer_summary[("Customer summary")]
end

subgraph group_quality["Data quality"]
  node_validation["Silver validation"]
  node_validation_checks["Validation checks<br/>[validate.py]"]
  node_integrity["Delta integrity demo"]
end

node_workflow(("Databricks workflow"))

node_orders_source -->|"ingests"| node_bronze
node_events_source -->|"ingests"| node_bronze
node_bronze -->|"writes"| node_bronze_tables
node_workflow -->|"runs first"| node_bronze
node_workflow -->|"runs second"| node_silver
node_workflow -->|"runs third"| node_validation
node_workflow -->|"runs fourth"| node_gold
node_bronze_tables -->|"reads"| node_silver
node_silver -->|"uses"| node_event_cleaning
node_silver -->|"cleans"| node_clean_orders
node_silver -->|"joins"| node_customer_events
node_silver -->|"writes"| node_clean_events
node_silver -->|"retains"| node_rejected_events
node_silver -->|"writes"| node_clean_orders
node_customer_events -->|"checks"| node_validation
node_validation -->|"uses"| node_validation_checks
node_validation -->|"precedes"| node_gold
node_customer_events -->|"aggregates"| node_gold
node_gold -->|"publishes"| node_customer_summary
node_integrity -->|"demonstrates constraints"| node_clean_events

click node_events_source "https://github.com/nabilkendri/lakehouse-forge/blob/main/data/web_events.jsonl"
click node_bronze "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/10_ingest_bronze.ipynb"
click node_silver "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb"
click node_event_cleaning "https://github.com/nabilkendri/lakehouse-forge/blob/main/src/transform.py"
click node_clean_orders "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb"
click node_customer_events "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb"
click node_validation "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/12_validate_silver.ipynb"
click node_validation_checks "https://github.com/nabilkendri/lakehouse-forge/blob/main/src/validate.py"
click node_gold "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/13_build_gold.ipynb"
click node_integrity "https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/04_data_integrity.ipynb"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_orders_source,node_events_source toneBlue
class node_bronze,node_bronze_tables,node_silver,node_event_cleaning,node_clean_events,node_rejected_events,node_clean_orders,node_customer_events,node_gold,node_customer_summary toneAmber
class node_validation,node_validation_checks,node_integrity toneMint
class node_workflow toneIndigo
```

## Components

### Data sources

- **Sample orders**
- [Web events](https://github.com/nabilkendri/lakehouse-forge/blob/main/data/web_events.jsonl): `data/web_events.jsonl`

### ETL pipeline

- [Bronze ingestion](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/10_ingest_bronze.ipynb): `notebooks/10_ingest_bronze.ipynb`
- **Bronze tables**
- [Silver transformation](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb): `notebooks/11_transform_silver.ipynb`
- [Event cleaning](https://github.com/nabilkendri/lakehouse-forge/blob/main/src/transform.py): `src/transform.py`
- **Clean events**
- **Rejected events**
- [Clean orders](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb): `notebooks/11_transform_silver.ipynb`
- [Customer events](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/11_transform_silver.ipynb): `notebooks/11_transform_silver.ipynb`
- [Gold aggregation](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/13_build_gold.ipynb): `notebooks/13_build_gold.ipynb`
- **Customer summary**

### Data quality

- [Silver validation](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/12_validate_silver.ipynb): `notebooks/12_validate_silver.ipynb`
- [Validation checks](https://github.com/nabilkendri/lakehouse-forge/blob/main/src/validate.py): `src/validate.py`
- [Delta integrity demo](https://github.com/nabilkendri/lakehouse-forge/blob/main/notebooks/04_data_integrity.ipynb): `notebooks/04_data_integrity.ipynb`

### Other components

- **Databricks workflow**

## Connections

- Sample orders → Bronze ingestion: ingests (evidence: `README.md`)
- Web events → Bronze ingestion: ingests (evidence: `README.md`)
- Bronze ingestion → Bronze tables: writes (evidence: `README.md`)
- Databricks workflow → Bronze ingestion: runs first (evidence: `README.md`)
- Databricks workflow → Silver transformation: runs second (evidence: `README.md`)
- Databricks workflow → Silver validation: runs third (evidence: `README.md`)
- Databricks workflow → Gold aggregation: runs fourth (evidence: `README.md`)
- Bronze tables → Silver transformation: reads (evidence: `README.md`)
- Silver transformation → Event cleaning: uses (evidence: `README.md`)
- Silver transformation → Clean orders: cleans (evidence: `README.md`)
- Silver transformation → Customer events: joins (evidence: `README.md`)
- Silver transformation → Clean events: writes (evidence: `README.md`)
- Silver transformation → Rejected events: retains (evidence: `README.md`)
- Silver transformation → Clean orders: writes (evidence: `README.md`)
- Customer events → Silver validation: checks (evidence: `README.md`)
- Silver validation → Validation checks: uses (evidence: `README.md`)
- Silver validation → Gold aggregation: precedes (evidence: `README.md`)
- Customer events → Gold aggregation: aggregates (evidence: `README.md`)
- Gold aggregation → Customer summary: publishes (evidence: `README.md`)
- Delta integrity demo → Clean events: demonstrates constraints (evidence: `README.md`)
