How mikolajsedek/lakeflow_declarative_pipelines_demo works
This repository is a Databricks Lakeflow Declarative Pipelines demo. Its user-to-result path generates synthetic, joinable CSV datasets in a Unity Catalog Volume, then demonstrates a simple CSV-backed view and an advanced Bronze-to-Silver-to-Gold pipeline with KPI views; a separate SCD Type 2 pipeline preserves order history. DDL construction and dataframe transformations are the main internal reusable layers. The README documents several pipeline behaviors whose implementation excerpts are incomplete, so those relationships cite the README; unsampled implementation details are not inferred.
Main entry points: 03.advanced_pipeline.py Medallion pipeline · 01.create_fake_data.py Fake data notebook · 02.simple_pipeline.py Simple views · 04.scd_genie_code_pipeline.py SCD Type 2 history
Components 16
Data generation
- Fake data notebook
src/python/notebooks/01.create_fake_data.py - CSV source volume
Lakeflow pipelines
- Simple views
src/python/notebooks/02.simple_pipeline.py - Medallion pipeline
src/python/notebooks/03.advanced_pipeline.py - Bronze ingestion
src/python/notebooks/03.advanced_pipeline.py - Silver cleansing
src/python/notebooks/03.advanced_pipeline.py - Silver tables
Analytics and history
- Gold CDC entities
src/python/notebooks/03.advanced_pipeline.py - Gold KPI views
src/python/notebooks/03.advanced_pipeline.py - Summary statistics
src/python/notebooks/02.simple_pipeline.py - SCD Type 2 history
src/python/notebooks/04.scd_genie_code_pipeline.py - Gold tables
Reusable helpers
- Synthetic row generation
src/python/notebooks/modules/data_generation.py - Catalog DDL builder
src/python/notebooks/modules/ddl_helpers.py - Transform functions
src/python/notebooks/modules/transformations.py
Other components
- Demo operator
Connections 22
- Demo operator → Fake data notebook: runs
README.md - Fake data notebook → Synthetic row generation: generates rows
src/python/notebooks/01.create_fake_data.py - Fake data notebook → Catalog DDL builder: builds setup DDL
src/python/notebooks/01.create_fake_data.py - Fake data notebook → CSV source volume: writes CSV
src/python/notebooks/01.create_fake_data.py - Demo operator → Simple views: runs
README.md - Simple views → CSV source volume: reads CSV
README.md - Demo operator → Medallion pipeline: runs
README.md - Medallion pipeline → Bronze ingestion: registers ingestion
README.md - Bronze ingestion → CSV source volume: loads source files
README.md - Medallion pipeline → Silver cleansing: registers staging
README.md - Silver cleansing → Bronze ingestion: reads stream
src/python/notebooks/03.advanced_pipeline.py - Silver cleansing → Transform functions: applies transforms
src/python/notebooks/03.advanced_pipeline.py - Silver cleansing → Silver tables: writes tables
README.md - Medallion pipeline → Gold CDC entities: registers CDC
src/python/notebooks/03.advanced_pipeline.py - Gold CDC entities → Silver tables: merges source
README.md - Gold CDC entities → Gold tables: writes entities
README.md - Medallion pipeline → Gold KPI views: registers views
src/python/notebooks/03.advanced_pipeline.py - Gold KPI views → Gold tables: reads gold tables
src/python/notebooks/03.advanced_pipeline.py - Simple views → Summary statistics: creates views
README.md - Demo operator → SCD Type 2 history: runs
README.md - SCD Type 2 history → Silver tables: reads Silver orders
README.md - SCD Type 2 history → Gold tables: writes history
src/python/notebooks/04.scd_genie_code_pipeline.py