# srajan2627/uber-eats-lakehouse architecture

> Architecture diagram of the GitHub repository srajan2627/uber-eats-lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/srajan2627/uber-eats-lakehouse
- Repository: https://github.com/srajan2627/uber-eats-lakehouse

## Overview

This is a Databricks lakehouse reference implementation for marketplace analytics. Its main workflow generates synthetic marketplace feeds, incrementally ingests them into Bronze, standardizes them into Silver, checks and quarantines data-quality failures, builds historical dimensions and Gold facts, then publishes analyst-facing Genie views. The graph also includes optimization as a documented downstream capability. Relationships are limited to those shown in excerpts or explicitly stated in the README; unsampled notebook logic is not treated as inspected.

Main entry points: [`notebooks/04_gold/02_build_gold_facts.py`](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/02_build_gold_facts.py) (Gold fact build), [`notebooks/02_silver/01_build_silver.py`](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/02_silver/01_build_silver.py) (Silver build), [`notebooks/03_dq/01_run_data_quality.py`](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/03_dq/01_run_data_quality.py) (Quality checks), [`notebooks/01_bronze/01_autoloader_ingest.py`](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/01_bronze/01_autoloader_ingest.py) (Auto Loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Data Ingestion"]
  node_generator["Sample generator"]
  node_landing["Landing files"]
  node_autoloader["Auto Loader"]
  node_bronze[("Bronze tables")]
end

subgraph group_model["Lakehouse Modeling"]
  node_silverjob["Silver build<br/>[01_build_silver.py]"]
  node_silvertransforms["Silver transforms"]
  node_silver[("Silver tables<br/>[01_build_silver.py]")]
  node_dimjob["SCD2 dimensions"]
  node_scd2["SCD2 logic<br/>[scd2.py]"]
  node_dimensions[("Gold dimensions")]
  node_factjob["Gold fact build"]
  node_goldtransforms["Fact transforms<br/>[gold_transforms.py]"]
  node_facts[("Gold facts")]
end

subgraph group_quality["Quality Controls"]
  node_dqjob["Quality checks"]
  node_dqrules["Quality rules<br/>[data_quality.py]"]
  node_dqstore[("DQ results")]
end

subgraph group_analytics["Analytics Access"]
  node_optimization["Gold optimization"]
  node_genie[("Genie views")]
end

node_operator(("Data operator"))
node_analyst(("Analytics user"))

node_operator -.->|"runs generator"| node_generator
node_generator -->|"writes feeds"| node_landing
node_operator -.->|"starts pipeline"| node_autoloader
node_autoloader -->|"loads files"| node_landing
node_autoloader -->|"writes raw tables"| node_bronze
node_silverjob -->|"reads raw tables"| node_bronze
node_silverjob -->|"calls transforms"| node_silvertransforms
node_silverjob -->|"merges cleaned data"| node_silver
node_dqjob -->|"loads rules"| node_dqrules
node_dqjob -->|"checks records"| node_silver
node_dqjob -->|"persists results"| node_dqstore
node_dimjob -->|"uses SCD2 logic"| node_scd2
node_dimjob -->|"builds history"| node_dimensions
node_factjob -->|"calls fact transform"| node_goldtransforms
node_factjob -->|"resolves versions"| node_dimensions
node_factjob -->|"reads silver data"| node_silver
node_factjob -->|"writes facts"| node_facts
node_optimization -.->|"optimizes gold"| node_facts
node_genie -->|"publishes analytics"| node_facts
node_genie -->|"exposes quality trends"| node_dqstore
node_analyst -->|"asks questions"| node_genie

click node_generator "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/data_generator/generate_uber_eats_data.py"
click node_landing "https://github.com/srajan2627/uber-eats-lakehouse/tree/main/generated_data"
click node_autoloader "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/01_bronze/01_autoloader_ingest.py"
click node_bronze "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/01_bronze/01_autoloader_ingest.py"
click node_silverjob "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/02_silver/01_build_silver.py"
click node_silvertransforms "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/silver_transforms.py"
click node_silver "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/02_silver/01_build_silver.py"
click node_dqjob "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/03_dq/01_run_data_quality.py"
click node_dqrules "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/data_quality.py"
click node_dqstore "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/03_dq/01_run_data_quality.py"
click node_dimjob "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/01_build_dimensions_scd2.py"
click node_scd2 "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/scd2.py"
click node_dimensions "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/01_build_dimensions_scd2.py"
click node_factjob "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/02_build_gold_facts.py"
click node_goldtransforms "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/gold_transforms.py"
click node_facts "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/02_build_gold_facts.py"
click node_optimization "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/05_optimization/01_optimize_gold_tables.py"
click node_genie "https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/06_genie/01_publish_genie_views.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_landing,node_autoloader,node_bronze,node_analyst toneBlue
class node_silverjob,node_silvertransforms,node_silver,node_dimjob,node_scd2,node_dimensions,node_factjob,node_goldtransforms,node_facts toneAmber
class node_dqjob,node_dqrules,node_dqstore toneMint
class node_optimization,node_genie toneRose
class node_operator toneIndigo
```

## Components

### Data Ingestion

- [Sample generator](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/data_generator/generate_uber_eats_data.py): `data_generator/generate_uber_eats_data.py`
- [Landing files](https://github.com/srajan2627/uber-eats-lakehouse/tree/main/generated_data): `generated_data`
- [Auto Loader](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/01_bronze/01_autoloader_ingest.py): `notebooks/01_bronze/01_autoloader_ingest.py`
- [Bronze tables](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/01_bronze/01_autoloader_ingest.py): `notebooks/01_bronze/01_autoloader_ingest.py`

### Lakehouse Modeling

- [Silver build](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/02_silver/01_build_silver.py): `notebooks/02_silver/01_build_silver.py`
- [Silver transforms](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/silver_transforms.py): `src/uber_eats_lakehouse/silver_transforms.py`
- [Silver tables](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/02_silver/01_build_silver.py): `notebooks/02_silver/01_build_silver.py`
- [SCD2 dimensions](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/01_build_dimensions_scd2.py): `notebooks/04_gold/01_build_dimensions_scd2.py`
- [SCD2 logic](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/scd2.py): `src/uber_eats_lakehouse/scd2.py`
- [Gold dimensions](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/01_build_dimensions_scd2.py): `notebooks/04_gold/01_build_dimensions_scd2.py`
- [Gold fact build](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/02_build_gold_facts.py): `notebooks/04_gold/02_build_gold_facts.py`
- [Fact transforms](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/gold_transforms.py): `src/uber_eats_lakehouse/gold_transforms.py`
- [Gold facts](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/04_gold/02_build_gold_facts.py): `notebooks/04_gold/02_build_gold_facts.py`

### Quality Controls

- [Quality checks](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/03_dq/01_run_data_quality.py): `notebooks/03_dq/01_run_data_quality.py`
- [Quality rules](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/src/uber_eats_lakehouse/data_quality.py): `src/uber_eats_lakehouse/data_quality.py`
- [DQ results](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/03_dq/01_run_data_quality.py): `notebooks/03_dq/01_run_data_quality.py`

### Analytics Access

- [Gold optimization](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/05_optimization/01_optimize_gold_tables.py): `notebooks/05_optimization/01_optimize_gold_tables.py`
- [Genie views](https://github.com/srajan2627/uber-eats-lakehouse/blob/main/notebooks/06_genie/01_publish_genie_views.py): `notebooks/06_genie/01_publish_genie_views.py`

### Other components

- **Data operator**
- **Analytics user**

## Connections

- Data operator → Sample generator: runs generator
- Sample generator → Landing files: writes feeds (evidence: `data_generator/generate_uber_eats_data.py`)
- Data operator → Auto Loader: starts pipeline
- Auto Loader → Landing files: loads files (evidence: `notebooks/01_bronze/01_autoloader_ingest.py`)
- Auto Loader → Bronze tables: writes raw tables (evidence: `notebooks/01_bronze/01_autoloader_ingest.py`)
- Silver build → Bronze tables: reads raw tables (evidence: `notebooks/02_silver/01_build_silver.py`)
- Silver build → Silver transforms: calls transforms (evidence: `notebooks/02_silver/01_build_silver.py`)
- Silver build → Silver tables: merges cleaned data (evidence: `notebooks/02_silver/01_build_silver.py`)
- Quality checks → Quality rules: loads rules (evidence: `notebooks/03_dq/01_run_data_quality.py`)
- Quality checks → Silver tables: checks records
- Quality checks → DQ results: persists results (evidence: `notebooks/03_dq/01_run_data_quality.py`)
- SCD2 dimensions → SCD2 logic: uses SCD2 logic (evidence: `notebooks/04_gold/01_build_dimensions_scd2.py`)
- SCD2 dimensions → Gold dimensions: builds history (evidence: `README.md`)
- Gold fact build → Fact transforms: calls fact transform (evidence: `notebooks/04_gold/02_build_gold_facts.py`)
- Gold fact build → Gold dimensions: resolves versions
- Gold fact build → Silver tables: reads silver data (evidence: `notebooks/04_gold/02_build_gold_facts.py`)
- Gold fact build → Gold facts: writes facts (evidence: `notebooks/04_gold/02_build_gold_facts.py`)
- Gold optimization → Gold facts: optimizes gold (evidence: `README.md`)
- Genie views → Gold facts: publishes analytics (evidence: `notebooks/06_genie/01_publish_genie_views.py`)
- Genie views → DQ results: exposes quality trends (evidence: `notebooks/06_genie/01_publish_genie_views.py`)
- Analytics user → Genie views: asks questions (evidence: `README.md`)