# mikolajsedek/lakeflow_declarative_pipelines_demo architecture

> Architecture diagram of the GitHub repository mikolajsedek/lakeflow_declarative_pipelines_demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mikolajsedek/lakeflow_declarative_pipelines_demo
- Repository: https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo

## Overview

This repository is a Databricks Lakeflow Declarative Pipelines demo. Its user-to-result path generates synthetic, joinable CSV datasets in a Unity Catalog Volume, then demonstrates a simple CSV-backed view and an advanced Bronze-to-Silver-to-Gold pipeline with KPI views; a separate SCD Type 2 pipeline preserves order history. DDL construction and dataframe transformations are the main internal reusable layers. The README documents several pipeline behaviors whose implementation excerpts are incomplete, so those relationships cite the README; unsampled implementation details are not inferred.

Main entry points: [`src/python/notebooks/03.advanced_pipeline.py`](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py) (Medallion pipeline), [`src/python/notebooks/01.create_fake_data.py`](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/01.create_fake_data.py) (Fake data notebook), [`src/python/notebooks/02.simple_pipeline.py`](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/02.simple_pipeline.py) (Simple views), [`src/python/notebooks/04.scd_genie_code_pipeline.py`](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/04.scd_genie_code_pipeline.py) (SCD Type 2 history)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_generation["Data generation"]
  node_generator["Fake data notebook"]
  node_volume[("CSV source volume")]
end

subgraph group_pipeline["Lakeflow pipelines"]
  node_simple["Simple views"]
  node_advanced["Medallion pipeline"]
  node_bronze["Bronze ingestion"]
  node_silver["Silver cleansing"]
  node_silverstore[("Silver tables")]
end

subgraph group_analytics["Analytics and history"]
  node_gold["Gold CDC entities"]
  node_kpis["Gold KPI views"]
  node_summary["Summary statistics"]
  node_scd2["SCD Type 2 history"]
  node_goldstore[("Gold tables")]
end

subgraph group_helpers["Reusable helpers"]
  node_rowgen["Synthetic row generation<br/>[data_generation.py]"]
  node_ddl["Catalog DDL builder<br/>[ddl_helpers.py]"]
  node_transform["Transform functions<br/>[transformations.py]"]
end

node_operator(("Demo operator"))

node_operator -->|"runs"| node_generator
node_generator -->|"generates rows"| node_rowgen
node_generator -->|"builds setup DDL"| node_ddl
node_generator -->|"writes CSV"| node_volume
node_operator -->|"runs"| node_simple
node_simple -->|"reads CSV"| node_volume
node_operator -->|"runs"| node_advanced
node_advanced -->|"registers ingestion"| node_bronze
node_bronze -->|"loads source files"| node_volume
node_advanced -->|"registers staging"| node_silver
node_silver -->|"reads stream"| node_bronze
node_silver -->|"applies transforms"| node_transform
node_silver -->|"writes tables"| node_silverstore
node_advanced -->|"registers CDC"| node_gold
node_gold -->|"merges source"| node_silverstore
node_gold -->|"writes entities"| node_goldstore
node_advanced -->|"registers views"| node_kpis
node_kpis -->|"reads gold tables"| node_goldstore
node_simple -->|"creates views"| node_summary
node_operator -->|"runs"| node_scd2
node_scd2 -->|"reads Silver orders"| node_silverstore
node_scd2 -->|"writes history"| node_goldstore

click node_generator "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/01.create_fake_data.py"
click node_rowgen "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/modules/data_generation.py"
click node_ddl "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/modules/ddl_helpers.py"
click node_simple "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/02.simple_pipeline.py"
click node_advanced "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py"
click node_bronze "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py"
click node_silver "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py"
click node_transform "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/modules/transformations.py"
click node_gold "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py"
click node_kpis "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py"
click node_summary "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/02.simple_pipeline.py"
click node_scd2 "https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/04.scd_genie_code_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_volume toneBlue
class node_simple,node_advanced,node_bronze,node_silver,node_silverstore toneAmber
class node_gold,node_kpis,node_summary,node_scd2,node_goldstore toneMint
class node_rowgen,node_ddl,node_transform toneRose
class node_operator toneIndigo
```

## Components

### Data generation

- [Fake data notebook](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/01.create_fake_data.py): `src/python/notebooks/01.create_fake_data.py`
- **CSV source volume**

### Lakeflow pipelines

- [Simple views](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/02.simple_pipeline.py): `src/python/notebooks/02.simple_pipeline.py`
- [Medallion pipeline](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py): `src/python/notebooks/03.advanced_pipeline.py`
- [Bronze ingestion](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py): `src/python/notebooks/03.advanced_pipeline.py`
- [Silver cleansing](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py): `src/python/notebooks/03.advanced_pipeline.py`
- **Silver tables**

### Analytics and history

- [Gold CDC entities](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py): `src/python/notebooks/03.advanced_pipeline.py`
- [Gold KPI views](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/03.advanced_pipeline.py): `src/python/notebooks/03.advanced_pipeline.py`
- [Summary statistics](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/02.simple_pipeline.py): `src/python/notebooks/02.simple_pipeline.py`
- [SCD Type 2 history](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/04.scd_genie_code_pipeline.py): `src/python/notebooks/04.scd_genie_code_pipeline.py`
- **Gold tables**

### Reusable helpers

- [Synthetic row generation](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/modules/data_generation.py): `src/python/notebooks/modules/data_generation.py`
- [Catalog DDL builder](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/modules/ddl_helpers.py): `src/python/notebooks/modules/ddl_helpers.py`
- [Transform functions](https://github.com/mikolajsedek/lakeflow_declarative_pipelines_demo/blob/main/src/python/notebooks/modules/transformations.py): `src/python/notebooks/modules/transformations.py`

### Other components

- **Demo operator**

## Connections

- Demo operator → Fake data notebook: runs (evidence: `README.md`)
- Fake data notebook → Synthetic row generation: generates rows (evidence: `src/python/notebooks/01.create_fake_data.py`)
- Fake data notebook → Catalog DDL builder: builds setup DDL (evidence: `src/python/notebooks/01.create_fake_data.py`)
- Fake data notebook → CSV source volume: writes CSV (evidence: `src/python/notebooks/01.create_fake_data.py`)
- Demo operator → Simple views: runs (evidence: `README.md`)
- Simple views → CSV source volume: reads CSV (evidence: `README.md`)
- Demo operator → Medallion pipeline: runs (evidence: `README.md`)
- Medallion pipeline → Bronze ingestion: registers ingestion (evidence: `README.md`)
- Bronze ingestion → CSV source volume: loads source files (evidence: `README.md`)
- Medallion pipeline → Silver cleansing: registers staging (evidence: `README.md`)
- Silver cleansing → Bronze ingestion: reads stream (evidence: `src/python/notebooks/03.advanced_pipeline.py`)
- Silver cleansing → Transform functions: applies transforms (evidence: `src/python/notebooks/03.advanced_pipeline.py`)
- Silver cleansing → Silver tables: writes tables (evidence: `README.md`)
- Medallion pipeline → Gold CDC entities: registers CDC (evidence: `src/python/notebooks/03.advanced_pipeline.py`)
- Gold CDC entities → Silver tables: merges source (evidence: `README.md`)
- Gold CDC entities → Gold tables: writes entities (evidence: `README.md`)
- Medallion pipeline → Gold KPI views: registers views (evidence: `src/python/notebooks/03.advanced_pipeline.py`)
- Gold KPI views → Gold tables: reads gold tables (evidence: `src/python/notebooks/03.advanced_pipeline.py`)
- Simple views → Summary statistics: creates views (evidence: `README.md`)
- Demo operator → SCD Type 2 history: runs (evidence: `README.md`)
- SCD Type 2 history → Silver tables: reads Silver orders (evidence: `README.md`)
- SCD Type 2 history → Gold tables: writes history (evidence: `src/python/notebooks/04.scd_genie_code_pipeline.py`)
