How nabilkendri/lakehouse-forge works
Lakehouse Forge is a Databricks/PySpark ETL learning project. Its principal workflow ingests SQL orders and JSON web events into Bronze, cleans and joins them in Silver while retaining rejected events, validates Silver, and publishes customer metrics in Gold. The README documents the notebook stages and reusable validation module; sampled source also shows the web-event cleaning and rejection functions. Notebook internals were not sampled, so links between their stages are based on the README.
Main entry points: 11_transform_silver.ipynb Silver transformation · 12_validate_silver.ipynb Silver validation · 10_ingest_bronze.ipynb Bronze ingestion · 13_build_gold.ipynb Gold aggregation
Components 16
Data sources
- Sample orders
- Web events
data/web_events.jsonl
ETL pipeline
- Bronze ingestion
notebooks/10_ingest_bronze.ipynb - Bronze tables
- Silver transformation
notebooks/11_transform_silver.ipynb - Event cleaning
src/transform.py - Clean events
- Rejected events
- Clean orders
notebooks/11_transform_silver.ipynb - Customer events
notebooks/11_transform_silver.ipynb - Gold aggregation
notebooks/13_build_gold.ipynb - Customer summary
Data quality
- Silver validation
notebooks/12_validate_silver.ipynb - Validation checks
src/validate.py - Delta integrity demo
notebooks/04_data_integrity.ipynb
Other components
- Databricks workflow
Connections 20
- Sample orders → Bronze ingestion: ingests
README.md - Web events → Bronze ingestion: ingests
README.md - Bronze ingestion → Bronze tables: writes
README.md - Databricks workflow → Bronze ingestion: runs first
README.md - Databricks workflow → Silver transformation: runs second
README.md - Databricks workflow → Silver validation: runs third
README.md - Databricks workflow → Gold aggregation: runs fourth
README.md - Bronze tables → Silver transformation: reads
README.md - Silver transformation → Event cleaning: uses
README.md - Silver transformation → Clean orders: cleans
README.md - Silver transformation → Customer events: joins
README.md - Silver transformation → Clean events: writes
README.md - Silver transformation → Rejected events: retains
README.md - Silver transformation → Clean orders: writes
README.md - Customer events → Silver validation: checks
README.md - Silver validation → Validation checks: uses
README.md - Silver validation → Gold aggregation: precedes
README.md - Customer events → Gold aggregation: aggregates
README.md - Gold aggregation → Customer summary: publishes
README.md - Delta integrity demo → Clean events: demonstrates constraints
README.md