How tanor-dataskills/lakeflow_declarative_pipeline works
RetailQ is a Databricks retail analytics project. Its documented workflow brings PostgreSQL, Salesforce, and blob transactions into Bronze, cleans and standardizes them in Silver, builds a Gold sales model and calendar, and exposes metrics for dashboards and natural-language analysis. The excerpts show the blob ingestion and several transformations directly; connector setup, some Gold dimensions, orchestration, governance, and end-user tools are documented but not implemented in the sampled files.
Main entry points: 02_ETL_Pipeline Managed connectors · 04_Metric View.py Retail metric view · 00_Bronze_to_silver Postgres Bronze · fact_sales.py Sales fact
Components 23
Source ingestion
- Managed connectors
01_RetailQ_Project/02_ETL_Pipeline - Auto Loader
01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py - Postgres Bronze
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver - Salesforce Bronze
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver - Blob Bronze
01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py
Medallion data
- Account cleaning
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/account.py - Opportunity cleaning
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/opportunity.py - Inventory cleaning
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/inventory.py - Product cleaning
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/product_catalog.py - Transaction cleaning
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/transactions.py
Analytics layer
- Sales fact
01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py - Gold dimensions
01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold - Calendar table
01_RetailQ_Project/01_Notebook/03_calendar.py - Retail metric view
01_RetailQ_Project/01_Notebook/04_Metric View.py
Data consumption
- Databricks dashboards
- Genie workspace
- Assistant examples
01_RetailQ_Project/03_Genie/01_Chat_Mode.py - Edit mode examples
01_RetailQ_Project/03_Genie/02_Edit_Mode.py - Agent mode examples
01_RetailQ_Project/03_Genie/03_Agent_Mode.py
Other components
- PostgreSQL
- Salesforce
- Blob transactions
- Retail analyst
Connections 23
- PostgreSQL → Managed connectors: ingested via
README.md - Salesforce → Managed connectors: ingested via
README.md - Managed connectors → Postgres Bronze: loads
README.md - Managed connectors → Salesforce Bronze: loads
README.md - Blob transactions → Auto Loader: read from volume
01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py - Auto Loader → Blob Bronze: writes table
01_RetailQ_Project/01_Notebook/01_blob_to_bronze.py - Salesforce Bronze → Account cleaning: streams table
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/account.py - Salesforce Bronze → Opportunity cleaning: streams table
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/opportunity.py - Postgres Bronze → Inventory cleaning: streams table
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/inventory.py - Postgres Bronze → Product cleaning: streams table
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/product_catalog.py - Blob Bronze → Transaction cleaning: reads table
01_RetailQ_Project/02_ETL_Pipeline/00_Bronze_to_silver/transactions.py - Transaction cleaning → Sales fact: reads table
01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py - Opportunity cleaning → Sales fact: reads table
01_RetailQ_Project/02_ETL_Pipeline/01_Silver_to_gold/fact_sales.py - Sales fact → Retail metric view: metric source
01_RetailQ_Project/01_Notebook/04_Metric View.py - Gold dimensions → Retail metric view: dimension joins
01_RetailQ_Project/01_Notebook/04_Metric View.py - Calendar table → Gold dimensions: supports calendar dimension
- Retail analyst → Databricks dashboards: explores
README.md - Retail analyst → Genie workspace: asks questions
README.md - Retail metric view → Databricks dashboards: feeds analytics
README.md - Retail metric view → Genie workspace: supports self-service
README.md - Retail analyst → Assistant examples: uses examples
01_RetailQ_Project/03_Genie/01_Chat_Mode.py - Retail analyst → Edit mode examples: uses examples
01_RetailQ_Project/03_Genie/02_Edit_Mode.py - Retail analyst → Agent mode examples: uses examples
01_RetailQ_Project/03_Genie/03_Agent_Mode.py