How shivatejabadugu/ecommerce-analytics-platform works
This is an Azure/Databricks e-commerce analytics lakehouse. Its main workflow incrementally extracts order and reference data, processes batch and clickstream data through bronze, silver, and gold stages, then serves business metrics to Power BI. The graph also shows quality/audit state and monitoring. README documentation supports several integrations whose implementation was not sampled; their links are shown only where documented, and the unsampled monitoring and BI paths use existing directories.
Main entry points: batch_job.py Batch job · silver/batch.py Silver batch · gold/batch.py Gold batch · pipeline ADF orchestration
Components 24
Source ingestion
- ADF orchestration
adf/pipeline - Landing files
adf/dataset - Event Hubs
Lakehouse processing
- Batch job
src/ecom_platform/jobs/batch_job.py - Batch ingestion
src/ecom_platform/bronze/batch_ingest.py - Stream ingestion
src/ecom_platform/bronze/stream_ingest.py - Bronze tables
src/ecom_platform/bronze - Silver batch
src/ecom_platform/silver/batch.py - Clickstream silver
src/ecom_platform/silver/clickstream.py - Typed transforms
src/ecom_platform/silver/transforms.py - Silver tables
src/ecom_platform/silver
Analytics serving
- Gold batch
src/ecom_platform/gold/batch.py - Facts and dimensions
src/ecom_platform/gold/facts.py - Business aggregates
src/ecom_platform/gold/aggregates.py - Realtime metrics
src/ecom_platform/gold/realtime.py - Gold tables
src/ecom_platform/gold - SQL views
sql/gold
Quality and operations
- Quality and quarantine
src/ecom_platform/common/quality.py - Control tables
src/ecom_platform/common/audit.py - Monitoring queries
monitoring/kql
Other components
- SQL sources
- Order API
- Web and mobile
- Power BI
Connections 31
- SQL sources → ADF orchestration: extracts changes
- Order API → ADF orchestration: extracts changes
- ADF orchestration → Landing files: writes extracts
- ADF orchestration → Batch job: starts job
- Landing files → Batch ingestion: reads files
- Batch job → Batch ingestion: runs batch
- Batch job → Silver batch: runs batch
- Batch job → Gold batch: runs batch
- Batch ingestion → Bronze tables: writes raw
- Batch ingestion → Quality and quarantine: quarantines records
- Web and mobile → Event Hubs: sends events
- Event Hubs → Stream ingestion: streams events
- Stream ingestion → Bronze tables: writes raw events
- Bronze tables → Silver batch: reads batch data
- Silver batch → Typed transforms: transforms entities
- Silver batch → Silver tables: merges tables
- Silver batch → Quality and quarantine: checks records
- Bronze tables → Clickstream silver: reads events
- Clickstream silver → Silver tables: merges events
- Clickstream silver → Quality and quarantine: checks records
- Silver tables → Gold batch: reads changes
- Gold batch → Facts and dimensions: builds tables
- Gold batch → Business aggregates: builds metrics
- Gold batch → Gold tables: writes gold
- Silver tables → Realtime metrics: reads events
- Realtime metrics → Gold tables: updates metrics
- Gold tables → Power BI: serves analytics
- Gold tables → SQL views: queried by
- Quality and quarantine → Control tables: records results
- Batch job → Control tables: audits runs
- Control tables → Monitoring queries: feeds monitoring