# tanor-dataskills/lakeflow_declarative_pipeline architecture

> Architecture diagram of the GitHub repository tanor-dataskills/lakeflow_declarative_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tanor-dataskills/lakeflow_declarative_pipeline
- Repository: https://github.com/tanor-dataskills/lakeflow_declarative_pipeline

## Overview

RetailQ is a Databricks retail analytics project. Its documented workflow brings PostgreSQL, Salesforce, and blob transactions into Bronze, cleans and standardizes them in Silver, builds a Gold sales model and calendar, and exposes metrics for dashboards and natural-language analysis. The excerpts show the blob ingestion and several transformations directly; connector setup, some Gold dimensions, orchestration, governance, and end-user tools are documented but not implemented in the sampled files.

Main entry points: [`01_RetailQ_Project/02_ETL_Pipeline`](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline) (Managed connectors), [`01_RetailQ_Project/01_Notebook/04_Metric View.py`](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/04_Metric%20View.py) (Retail metric view), [`01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver`](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver) (Postgres Bronze), [`01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py`](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py) (Sales fact)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source ingestion"]
  node_connector_ingest["Managed connectors"]
  node_blob_ingest["Auto Loader"]
  node_bronze_postgres[("Postgres Bronze")]
  node_bronze_salesforce[("Salesforce Bronze")]
  node_bronze_blob[("Blob Bronze")]
end

subgraph group_transform["Medallion data"]
  node_account["Account cleaning<br/>[account.py]"]
  node_opportunity["Opportunity cleaning<br/>[opportunity.py]"]
  node_inventory["Inventory cleaning<br/>[inventory.py]"]
  node_product["Product cleaning<br/>[product_catalog.py]"]
  node_transactions["Transaction cleaning<br/>[transactions.py]"]
end

subgraph group_analytics["Analytics layer"]
  node_fact[("Sales fact<br/>[fact_sales.py]")]
  node_dimensions[("Gold dimensions")]
  node_calendar[("Calendar table<br/>[03_calendar.py]")]
  node_metric[("Retail metric view<br/>[04_Metric View.py]")]
end

subgraph group_consume["Data consumption"]
  node_dashboards["Databricks dashboards"]
  node_genie["Genie workspace"]
  node_assistant_notebooks["Assistant examples<br/>[01_Chat_Mode.py]"]
  node_edit_mode["Edit mode examples<br/>[02_Edit_Mode.py]"]
  node_agent_mode["Agent mode examples<br/>[03_Agent_Mode.py]"]
end

node_postgres(("PostgreSQL"))
node_salesforce(("Salesforce"))
node_blob(("Blob transactions"))
node_user(("Retail analyst"))

node_postgres -.->|"ingested via"| node_connector_ingest
node_salesforce -.->|"ingested via"| node_connector_ingest
node_connector_ingest -.->|"loads"| node_bronze_postgres
node_connector_ingest -.->|"loads"| node_bronze_salesforce
node_blob -->|"read from volume"| node_blob_ingest
node_blob_ingest -->|"writes table"| node_bronze_blob
node_bronze_salesforce -->|"streams table"| node_account
node_bronze_salesforce -->|"streams table"| node_opportunity
node_bronze_postgres -->|"streams table"| node_inventory
node_bronze_postgres -->|"streams table"| node_product
node_bronze_blob -->|"reads table"| node_transactions
node_transactions -->|"reads table"| node_fact
node_opportunity -->|"reads table"| node_fact
node_fact -->|"metric source"| node_metric
node_dimensions -->|"dimension joins"| node_metric
node_calendar -.->|"supports calendar dimension"| node_dimensions
node_user -.->|"explores"| node_dashboards
node_user -.->|"asks questions"| node_genie
node_metric -.->|"feeds analytics"| node_dashboards
node_metric -.->|"supports self-service"| node_genie
node_user -->|"uses examples"| node_assistant_notebooks
node_user -->|"uses examples"| node_edit_mode
node_user -->|"uses examples"| node_agent_mode

click node_connector_ingest "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline"
click node_blob_ingest "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py"
click node_bronze_postgres "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver"
click node_bronze_salesforce "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver"
click node_bronze_blob "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py"
click node_account "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/account.py"
click node_opportunity "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/opportunity.py"
click node_inventory "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/inventory.py"
click node_product "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/product_catalog.py"
click node_transactions "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/transactions.py"
click node_fact "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py"
click node_dimensions "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold"
click node_calendar "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/03_calendar.py"
click node_metric "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/04_Metric%20View.py"
click node_assistant_notebooks "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/03_Genie/01_Chat_Mode.py"
click node_edit_mode "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/03_Genie/02_Edit_Mode.py"
click node_agent_mode "https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/03_Genie/03_Agent_Mode.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_connector_ingest,node_blob_ingest,node_bronze_postgres,node_bronze_salesforce,node_bronze_blob toneBlue
class node_account,node_opportunity,node_inventory,node_product,node_transactions,node_postgres toneAmber
class node_fact,node_dimensions,node_calendar,node_metric toneMint
class node_dashboards,node_genie,node_assistant_notebooks,node_edit_mode,node_agent_mode toneRose
class node_salesforce,node_blob,node_user toneIndigo
```

## Components

### Source ingestion

- [Managed connectors](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline): `01_RetailQ_Project/02_ETL_Pipeline`
- [Auto Loader](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py): `01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py`
- [Postgres Bronze](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver`
- [Salesforce Bronze](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver`
- [Blob Bronze](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py): `01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py`

### Medallion data

- [Account cleaning](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/account.py): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/account.py`
- [Opportunity cleaning](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/opportunity.py): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/opportunity.py`
- [Inventory cleaning](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/inventory.py): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/inventory.py`
- [Product cleaning](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/product_catalog.py): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/product_catalog.py`
- [Transaction cleaning](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/transactions.py): `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/transactions.py`

### Analytics layer

- [Sales fact](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py): `01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py`
- [Gold dimensions](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/tree/main/01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold): `01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold`
- [Calendar table](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/03_calendar.py): `01_RetailQ_Project/01_Notebook/03_calendar.py`
- [Retail metric view](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/01_Notebook/04_Metric%20View.py): `01_RetailQ_Project/01_Notebook/04_Metric View.py`

### Data consumption

- **Databricks dashboards**
- **Genie workspace**
- [Assistant examples](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/03_Genie/01_Chat_Mode.py): `01_RetailQ_Project/03_Genie/01_Chat_Mode.py`
- [Edit mode examples](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/03_Genie/02_Edit_Mode.py): `01_RetailQ_Project/03_Genie/02_Edit_Mode.py`
- [Agent mode examples](https://github.com/tanor-dataskills/lakeflow_declarative_pipeline/blob/main/01_RetailQ_Project/03_Genie/03_Agent_Mode.py): `01_RetailQ_Project/03_Genie/03_Agent_Mode.py`

### Other components

- **PostgreSQL**
- **Salesforce**
- **Blob transactions**
- **Retail analyst**

## Connections

- PostgreSQL → Managed connectors: ingested via (evidence: `README.md`)
- Salesforce → Managed connectors: ingested via (evidence: `README.md`)
- Managed connectors → Postgres Bronze: loads (evidence: `README.md`)
- Managed connectors → Salesforce Bronze: loads (evidence: `README.md`)
- Blob transactions → Auto Loader: read from volume (evidence: `01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py`)
- Auto Loader → Blob Bronze: writes table (evidence: `01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py`)
- Salesforce Bronze → Account cleaning: streams table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/account.py`)
- Salesforce Bronze → Opportunity cleaning: streams table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/opportunity.py`)
- Postgres Bronze → Inventory cleaning: streams table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/inventory.py`)
- Postgres Bronze → Product cleaning: streams table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/product_catalog.py`)
- Blob Bronze → Transaction cleaning: reads table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/transactions.py`)
- Transaction cleaning → Sales fact: reads table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py`)
- Opportunity cleaning → Sales fact: reads table (evidence: `01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py`)
- Sales fact → Retail metric view: metric source (evidence: `01_RetailQ_Project/01_Notebook/04_Metric View.py`)
- Gold dimensions → Retail metric view: dimension joins (evidence: `01_RetailQ_Project/01_Notebook/04_Metric View.py`)
- Calendar table → Gold dimensions: supports calendar dimension
- Retail analyst → Databricks dashboards: explores (evidence: `README.md`)
- Retail analyst → Genie workspace: asks questions (evidence: `README.md`)
- Retail metric view → Databricks dashboards: feeds analytics (evidence: `README.md`)
- Retail metric view → Genie workspace: supports self-service (evidence: `README.md`)
- Retail analyst → Assistant examples: uses examples (evidence: `01_RetailQ_Project/03_Genie/01_Chat_Mode.py`)
- Retail analyst → Edit mode examples: uses examples (evidence: `01_RetailQ_Project/03_Genie/02_Edit_Mode.py`)
- Retail analyst → Agent mode examples: uses examples (evidence: `01_RetailQ_Project/03_Genie/03_Agent_Mode.py`)
