How devbhalodia/ecommerce_data_platform works
This is an ecommerce analytics data platform: CDC data moves from an OLTP source through Bronze and Silver processing into dbt Gold models and dashboard KPIs. The graph preserves documented quality controls, analytics stages, and maintenance. Most links are documented in the README; sampled Silver notebooks show per-entity processing, while unsampled dbt internals are represented at their documented directory paths.
Main entry points: silver_layer Silver Stream Processing · Schema_Drift_Alert.sql Schema Drift Alert · bronze_layer Bronze Delta · staging dbt Staging
Components 19
CDC Ingestion
- Fivetran CDC
- Bronze Delta
bronze_layer - Bronze Registration
bronze_layer/external_table_registration.py
Quality and Silver
- Schema Drift Alert
alerting/Schema_Drift_Alert.sql - Drift Baseline
monitoring.py - Silver Stream Processing
silver_layer - Silver Tables
table_definition.py - Quarantine Tables
table_definition.py
Gold Analytics
- dbt Staging
dbt_dev/models/staging - dbt Data Tests
dbt_dev/tests - Sales Intermediate
dbt_dev/models/intermediate/int_sales_events.sql - Dimensions
dbt_dev/models/marts/dimensions - Fact Marts
dbt_dev/models/marts - KPI Metrics
dbt_dev/models/marts/metrics
Operations and Reporting
- Lakeflow Jobs
- Databricks Dashboard
- Delta Maintenance
maintenance.py
Other components
- NeonDB OLTP
- Email Notifications
Connections 24
- NeonDB OLTP → Fivetran CDC: supplies CDC
README.md - Fivetran CDC → Bronze Delta: lands Delta
README.md - Bronze Delta → Bronze Registration: registers tables
bronze_layer/external_table_registration.py - Bronze Delta → Schema Drift Alert: checks schema
README.md - Drift Baseline → Schema Drift Alert: provides baseline
README.md - Schema Drift Alert → Email Notifications: sends alert
README.md - Schema Drift Alert → Silver Stream Processing: precedes processing
README.md - Bronze Delta → Silver Stream Processing: streams batches
README.md - Silver Stream Processing → Silver Tables: upserts valid rows
silver_layer/shipments.py - Silver Stream Processing → Quarantine Tables: writes rejected rows
silver_layer/shipments.py - Silver Stream Processing → dbt Staging: feeds transformations
README.md - dbt Staging → dbt Data Tests: validates models
README.md - dbt Staging → Sales Intermediate: feeds model
README.md - Sales Intermediate → Fact Marts: supports marts
README.md - Fact Marts → Dimensions: joins dimensions
README.md - Fact Marts → KPI Metrics: feeds metrics
README.md - Dimensions → Fact Marts: joins facts
README.md - KPI Metrics → Databricks Dashboard: visualizes KPIs
README.md - Lakeflow Jobs → Schema Drift Alert: orchestrates first
README.md - Lakeflow Jobs → Silver Stream Processing: runs notebooks
README.md - Lakeflow Jobs → dbt Staging: runs dbt
README.md - Lakeflow Jobs → Databricks Dashboard: refreshes dashboard
README.md - Lakeflow Jobs → Email Notifications: sends notifications
README.md - Delta Maintenance → Silver Tables: optimizes tables
maintenance.py