How srajan2627/uber-eats-lakehouse works
This is a Databricks lakehouse reference implementation for marketplace analytics. Its main workflow generates synthetic marketplace feeds, incrementally ingests them into Bronze, standardizes them into Silver, checks and quarantines data-quality failures, builds historical dimensions and Gold facts, then publishes analyst-facing Genie views. The graph also includes optimization as a documented downstream capability. Relationships are limited to those shown in excerpts or explicitly stated in the README; unsampled notebook logic is not treated as inspected.
Main entry points: 02_build_gold_facts.py Gold fact build · 01_build_silver.py Silver build · 01_run_data_quality.py Quality checks · 01_autoloader_ingest.py Auto Loader
Components 20
Data Ingestion
- Sample generator
data_generator/generate_uber_eats_data.py - Landing files
generated_data - Auto Loader
notebooks/01_bronze/01_autoloader_ingest.py - Bronze tables
notebooks/01_bronze/01_autoloader_ingest.py
Lakehouse Modeling
- Silver build
notebooks/02_silver/01_build_silver.py - Silver transforms
src/uber_eats_lakehouse/silver_transforms.py - Silver tables
notebooks/02_silver/01_build_silver.py - SCD2 dimensions
notebooks/04_gold/01_build_dimensions_scd2.py - SCD2 logic
src/uber_eats_lakehouse/scd2.py - Gold dimensions
notebooks/04_gold/01_build_dimensions_scd2.py - Gold fact build
notebooks/04_gold/02_build_gold_facts.py - Fact transforms
src/uber_eats_lakehouse/gold_transforms.py - Gold facts
notebooks/04_gold/02_build_gold_facts.py
Quality Controls
- Quality checks
notebooks/03_dq/01_run_data_quality.py - Quality rules
src/uber_eats_lakehouse/data_quality.py - DQ results
notebooks/03_dq/01_run_data_quality.py
Analytics Access
- Gold optimization
notebooks/05_optimization/01_optimize_gold_tables.py - Genie views
notebooks/06_genie/01_publish_genie_views.py
Other components
- Data operator
- Analytics user
Connections 21
- Data operator → Sample generator: runs generator
- Sample generator → Landing files: writes feeds
data_generator/generate_uber_eats_data.py - Data operator → Auto Loader: starts pipeline
- Auto Loader → Landing files: loads files
notebooks/01_bronze/01_autoloader_ingest.py - Auto Loader → Bronze tables: writes raw tables
notebooks/01_bronze/01_autoloader_ingest.py - Silver build → Bronze tables: reads raw tables
notebooks/02_silver/01_build_silver.py - Silver build → Silver transforms: calls transforms
notebooks/02_silver/01_build_silver.py - Silver build → Silver tables: merges cleaned data
notebooks/02_silver/01_build_silver.py - Quality checks → Quality rules: loads rules
notebooks/03_dq/01_run_data_quality.py - Quality checks → Silver tables: checks records
- Quality checks → DQ results: persists results
notebooks/03_dq/01_run_data_quality.py - SCD2 dimensions → SCD2 logic: uses SCD2 logic
notebooks/04_gold/01_build_dimensions_scd2.py - SCD2 dimensions → Gold dimensions: builds history
README.md - Gold fact build → Fact transforms: calls fact transform
notebooks/04_gold/02_build_gold_facts.py - Gold fact build → Gold dimensions: resolves versions
- Gold fact build → Silver tables: reads silver data
notebooks/04_gold/02_build_gold_facts.py - Gold fact build → Gold facts: writes facts
notebooks/04_gold/02_build_gold_facts.py - Gold optimization → Gold facts: optimizes gold
README.md - Genie views → Gold facts: publishes analytics
notebooks/06_genie/01_publish_genie_views.py - Genie views → DQ results: exposes quality trends
notebooks/06_genie/01_publish_genie_views.py - Analytics user → Genie views: asks questions
README.md