# devbhalodia/ecommerce_data_platform architecture

> Architecture diagram of the GitHub repository devbhalodia/ecommerce_data_platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/devbhalodia/ecommerce_data_platform
- Repository: https://github.com/devbhalodia/ecommerce_data_platform

## Overview

This is an ecommerce analytics data platform: CDC data moves from an OLTP source through Bronze and Silver processing into dbt Gold models and dashboard KPIs. The graph preserves documented quality controls, analytics stages, and maintenance. Most links are documented in the README; sampled Silver notebooks show per-entity processing, while unsampled dbt internals are represented at their documented directory paths.

Main entry points: [`silver_layer`](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/silver_layer) (Silver Stream Processing), [`alerting/Schema_Drift_Alert.sql`](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/alerting/Schema_Drift_Alert.sql) (Schema Drift Alert), [`bronze_layer`](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/bronze_layer) (Bronze Delta), [`dbt_dev/models/staging`](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/staging) (dbt Staging)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["CDC Ingestion"]
  node_fivetran["Fivetran CDC"]
  node_bronze[("Bronze Delta")]
  node_registration["Bronze Registration"]
end

subgraph group_quality["Quality and Silver"]
  node_drift_alert["Schema Drift Alert"]
  node_baseline[("Drift Baseline<br/>[monitoring.py]")]
  node_silver["Silver Stream Processing"]
  node_silver_tables[("Silver Tables")]
  node_quarantine[("Quarantine Tables")]
end

subgraph group_analytics["Gold Analytics"]
  node_dbt_staging["dbt Staging"]
  node_dbt_tests["dbt Data Tests"]
  node_intermediate["Sales Intermediate"]
  node_dimensions["Dimensions"]
  node_facts["Fact Marts"]
  node_metrics["KPI Metrics"]
end

subgraph group_operations["Operations and Reporting"]
  node_jobs["Lakeflow Jobs"]
  node_dashboard["Databricks Dashboard"]
  node_maintenance["Delta Maintenance<br/>[maintenance.py]"]
end

node_oltp[("NeonDB OLTP")]
node_email["Email Notifications"]

node_oltp -->|"supplies CDC"| node_fivetran
node_fivetran -->|"lands Delta"| node_bronze
node_bronze -->|"registers tables"| node_registration
node_bronze -->|"checks schema"| node_drift_alert
node_baseline -->|"provides baseline"| node_drift_alert
node_drift_alert -->|"sends alert"| node_email
node_drift_alert -->|"precedes processing"| node_silver
node_bronze -->|"streams batches"| node_silver
node_silver -->|"upserts valid rows"| node_silver_tables
node_silver -->|"writes rejected rows"| node_quarantine
node_silver -->|"feeds transformations"| node_dbt_staging
node_dbt_staging -->|"validates models"| node_dbt_tests
node_dbt_staging -->|"feeds model"| node_intermediate
node_intermediate -->|"supports marts"| node_facts
node_facts -->|"joins dimensions"| node_dimensions
node_facts -->|"feeds metrics"| node_metrics
node_dimensions -->|"joins facts"| node_facts
node_metrics -->|"visualizes KPIs"| node_dashboard
node_jobs -->|"orchestrates first"| node_drift_alert
node_jobs -->|"runs notebooks"| node_silver
node_jobs -->|"runs dbt"| node_dbt_staging
node_jobs -->|"refreshes dashboard"| node_dashboard
node_jobs -->|"sends notifications"| node_email
node_maintenance -->|"optimizes tables"| node_silver_tables

click node_bronze "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/bronze_layer"
click node_registration "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/bronze_layer/external_table_registration.py"
click node_drift_alert "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/alerting/Schema_Drift_Alert.sql"
click node_baseline "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/monitoring.py"
click node_silver "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/silver_layer"
click node_silver_tables "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/table_definition.py"
click node_quarantine "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/table_definition.py"
click node_dbt_staging "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/staging"
click node_dbt_tests "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/tests"
click node_intermediate "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/dbt_dev/models/intermediate/int_sales_events.sql"
click node_dimensions "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/marts/dimensions"
click node_facts "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/marts"
click node_metrics "https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/marts/metrics"
click node_maintenance "https://github.com/devbhalodia/ecommerce_data_platform/blob/main/maintenance.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fivetran,node_bronze,node_registration toneBlue
class node_drift_alert,node_baseline,node_silver,node_silver_tables,node_quarantine,node_oltp toneAmber
class node_dbt_staging,node_dbt_tests,node_intermediate,node_dimensions,node_facts,node_metrics toneMint
class node_jobs,node_dashboard,node_maintenance toneRose
class node_email toneIndigo
```

## Components

### CDC Ingestion

- **Fivetran CDC**
- [Bronze Delta](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/bronze_layer): `bronze_layer`
- [Bronze Registration](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/bronze_layer/external_table_registration.py): `bronze_layer/external_table_registration.py`

### Quality and Silver

- [Schema Drift Alert](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/alerting/Schema_Drift_Alert.sql): `alerting/Schema_Drift_Alert.sql`
- [Drift Baseline](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/monitoring.py): `monitoring.py`
- [Silver Stream Processing](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/silver_layer): `silver_layer`
- [Silver Tables](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/table_definition.py): `table_definition.py`
- [Quarantine Tables](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/table_definition.py): `table_definition.py`

### Gold Analytics

- [dbt Staging](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/staging): `dbt_dev/models/staging`
- [dbt Data Tests](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/tests): `dbt_dev/tests`
- [Sales Intermediate](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/dbt_dev/models/intermediate/int_sales_events.sql): `dbt_dev/models/intermediate/int_sales_events.sql`
- [Dimensions](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/marts/dimensions): `dbt_dev/models/marts/dimensions`
- [Fact Marts](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/marts): `dbt_dev/models/marts`
- [KPI Metrics](https://github.com/devbhalodia/ecommerce_data_platform/tree/main/dbt_dev/models/marts/metrics): `dbt_dev/models/marts/metrics`

### Operations and Reporting

- **Lakeflow Jobs**
- **Databricks Dashboard**
- [Delta Maintenance](https://github.com/devbhalodia/ecommerce_data_platform/blob/main/maintenance.py): `maintenance.py`

### Other components

- **NeonDB OLTP**
- **Email Notifications**

## Connections

- NeonDB OLTP → Fivetran CDC: supplies CDC (evidence: `README.md`)
- Fivetran CDC → Bronze Delta: lands Delta (evidence: `README.md`)
- Bronze Delta → Bronze Registration: registers tables (evidence: `bronze_layer/external_table_registration.py`)
- Bronze Delta → Schema Drift Alert: checks schema (evidence: `README.md`)
- Drift Baseline → Schema Drift Alert: provides baseline (evidence: `README.md`)
- Schema Drift Alert → Email Notifications: sends alert (evidence: `README.md`)
- Schema Drift Alert → Silver Stream Processing: precedes processing (evidence: `README.md`)
- Bronze Delta → Silver Stream Processing: streams batches (evidence: `README.md`)
- Silver Stream Processing → Silver Tables: upserts valid rows (evidence: `silver_layer/shipments.py`)
- Silver Stream Processing → Quarantine Tables: writes rejected rows (evidence: `silver_layer/shipments.py`)
- Silver Stream Processing → dbt Staging: feeds transformations (evidence: `README.md`)
- dbt Staging → dbt Data Tests: validates models (evidence: `README.md`)
- dbt Staging → Sales Intermediate: feeds model (evidence: `README.md`)
- Sales Intermediate → Fact Marts: supports marts (evidence: `README.md`)
- Fact Marts → Dimensions: joins dimensions (evidence: `README.md`)
- Fact Marts → KPI Metrics: feeds metrics (evidence: `README.md`)
- Dimensions → Fact Marts: joins facts (evidence: `README.md`)
- KPI Metrics → Databricks Dashboard: visualizes KPIs (evidence: `README.md`)
- Lakeflow Jobs → Schema Drift Alert: orchestrates first (evidence: `README.md`)
- Lakeflow Jobs → Silver Stream Processing: runs notebooks (evidence: `README.md`)
- Lakeflow Jobs → dbt Staging: runs dbt (evidence: `README.md`)
- Lakeflow Jobs → Databricks Dashboard: refreshes dashboard (evidence: `README.md`)
- Lakeflow Jobs → Email Notifications: sends notifications (evidence: `README.md`)
- Delta Maintenance → Silver Tables: optimizes tables (evidence: `maintenance.py`)
