# shivatejabadugu/ecommerce-analytics-platform architecture

> Architecture diagram of the GitHub repository shivatejabadugu/ecommerce-analytics-platform, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shivatejabadugu/ecommerce-analytics-platform
- Repository: https://github.com/shivatejabadugu/ecommerce-analytics-platform
- Video tour (about a minute): https://gitdiagram.com/shivatejabadugu/ecommerce-analytics-platform/video

## Overview

This is an Azure/Databricks e-commerce analytics lakehouse. Its main workflow incrementally extracts order and reference data, processes batch and clickstream data through bronze, silver, and gold stages, then serves business metrics to Power BI. The graph also shows quality/audit state and monitoring. README documentation supports several integrations whose implementation was not sampled; their links are shown only where documented, and the unsampled monitoring and BI paths use existing directories.

Main entry points: [`src/ecom_platform/jobs/batch_job.py`](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/jobs/batch_job.py) (Batch job), [`src/ecom_platform/silver/batch.py`](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/batch.py) (Silver batch), [`src/ecom_platform/gold/batch.py`](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/batch.py) (Gold batch), [`adf/pipeline`](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/adf/pipeline) (ADF orchestration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Source ingestion"]
  node_adf["ADF orchestration"]
  node_landing[("Landing files")]
  node_eventhubs["Event Hubs"]
end

subgraph group_processing["Lakehouse processing"]
  node_batch_job["Batch job<br/>[batch_job.py]"]
  node_bronze_batch["Batch ingestion<br/>[batch_ingest.py]"]
  node_bronze_stream["Stream ingestion<br/>[stream_ingest.py]"]
  node_bronze[("Bronze tables")]
  node_silver_batch["Silver batch<br/>[batch.py]"]
  node_silver_stream["Clickstream silver<br/>[clickstream.py]"]
  node_silver_transforms["Typed transforms<br/>[transforms.py]"]
  node_silver[("Silver tables")]
end

subgraph group_serving["Analytics serving"]
  node_gold_batch["Gold batch<br/>[batch.py]"]
  node_gold_facts["Facts and dimensions<br/>[facts.py]"]
  node_gold_aggregates["Business aggregates<br/>[aggregates.py]"]
  node_gold_realtime["Realtime metrics<br/>[realtime.py]"]
  node_gold[("Gold tables")]
  node_sql_views["SQL views"]
end

subgraph group_operations["Quality and operations"]
  node_quality["Quality and quarantine<br/>[quality.py]"]
  node_control[("Control tables<br/>[audit.py]")]
  node_monitoring["Monitoring queries"]
end

node_sql_source(("SQL sources"))
node_order_api(("Order API"))
node_web_apps(("Web and mobile"))
node_powerbi(("Power BI"))

node_sql_source -->|"extracts changes"| node_adf
node_order_api -->|"extracts changes"| node_adf
node_adf -->|"writes extracts"| node_landing
node_adf -->|"starts job"| node_batch_job
node_landing -->|"reads files"| node_bronze_batch
node_batch_job -->|"runs batch"| node_bronze_batch
node_batch_job -->|"runs batch"| node_silver_batch
node_batch_job -->|"runs batch"| node_gold_batch
node_bronze_batch -->|"writes raw"| node_bronze
node_bronze_batch -->|"quarantines records"| node_quality
node_web_apps -->|"sends events"| node_eventhubs
node_eventhubs -->|"streams events"| node_bronze_stream
node_bronze_stream -->|"writes raw events"| node_bronze
node_bronze -->|"reads batch data"| node_silver_batch
node_silver_batch -->|"transforms entities"| node_silver_transforms
node_silver_batch -->|"merges tables"| node_silver
node_silver_batch -->|"checks records"| node_quality
node_bronze -->|"reads events"| node_silver_stream
node_silver_stream -->|"merges events"| node_silver
node_silver_stream -->|"checks records"| node_quality
node_silver -->|"reads changes"| node_gold_batch
node_gold_batch -->|"builds tables"| node_gold_facts
node_gold_batch -->|"builds metrics"| node_gold_aggregates
node_gold_batch -->|"writes gold"| node_gold
node_silver -->|"reads events"| node_gold_realtime
node_gold_realtime -->|"updates metrics"| node_gold
node_gold -->|"serves analytics"| node_powerbi
node_gold -->|"queried by"| node_sql_views
node_quality -->|"records results"| node_control
node_batch_job -->|"audits runs"| node_control
node_control -.->|"feeds monitoring"| node_monitoring

click node_adf "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/adf/pipeline"
click node_landing "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/adf/dataset"
click node_batch_job "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/jobs/batch_job.py"
click node_bronze_batch "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/bronze/batch_ingest.py"
click node_bronze_stream "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/bronze/stream_ingest.py"
click node_bronze "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/src/ecom_platform/bronze"
click node_silver_batch "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/batch.py"
click node_silver_stream "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/clickstream.py"
click node_silver_transforms "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/transforms.py"
click node_silver "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/src/ecom_platform/silver"
click node_gold_batch "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/batch.py"
click node_gold_facts "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/facts.py"
click node_gold_aggregates "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/aggregates.py"
click node_gold_realtime "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/realtime.py"
click node_gold "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/src/ecom_platform/gold"
click node_quality "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/common/quality.py"
click node_control "https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/common/audit.py"
click node_monitoring "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/monitoring/kql"
click node_sql_views "https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/sql/gold"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_adf,node_landing,node_eventhubs toneBlue
class node_batch_job,node_bronze_batch,node_bronze_stream,node_bronze,node_silver_batch,node_silver_stream,node_silver_transforms,node_silver toneAmber
class node_gold_batch,node_gold_facts,node_gold_aggregates,node_gold_realtime,node_gold,node_sql_views,node_order_api toneMint
class node_quality,node_control,node_monitoring toneRose
class node_sql_source,node_web_apps,node_powerbi toneIndigo
```

## Components

### Source ingestion

- [ADF orchestration](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/adf/pipeline): `adf/pipeline`
- [Landing files](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/adf/dataset): `adf/dataset`
- **Event Hubs**

### Lakehouse processing

- [Batch job](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/jobs/batch_job.py): `src/ecom_platform/jobs/batch_job.py`
- [Batch ingestion](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/bronze/batch_ingest.py): `src/ecom_platform/bronze/batch_ingest.py`
- [Stream ingestion](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/bronze/stream_ingest.py): `src/ecom_platform/bronze/stream_ingest.py`
- [Bronze tables](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/src/ecom_platform/bronze): `src/ecom_platform/bronze`
- [Silver batch](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/batch.py): `src/ecom_platform/silver/batch.py`
- [Clickstream silver](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/clickstream.py): `src/ecom_platform/silver/clickstream.py`
- [Typed transforms](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/silver/transforms.py): `src/ecom_platform/silver/transforms.py`
- [Silver tables](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/src/ecom_platform/silver): `src/ecom_platform/silver`

### Analytics serving

- [Gold batch](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/batch.py): `src/ecom_platform/gold/batch.py`
- [Facts and dimensions](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/facts.py): `src/ecom_platform/gold/facts.py`
- [Business aggregates](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/aggregates.py): `src/ecom_platform/gold/aggregates.py`
- [Realtime metrics](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/gold/realtime.py): `src/ecom_platform/gold/realtime.py`
- [Gold tables](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/src/ecom_platform/gold): `src/ecom_platform/gold`
- [SQL views](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/sql/gold): `sql/gold`

### Quality and operations

- [Quality and quarantine](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/common/quality.py): `src/ecom_platform/common/quality.py`
- [Control tables](https://github.com/shivatejabadugu/ecommerce-analytics-platform/blob/main/src/ecom_platform/common/audit.py): `src/ecom_platform/common/audit.py`
- [Monitoring queries](https://github.com/shivatejabadugu/ecommerce-analytics-platform/tree/main/monitoring/kql): `monitoring/kql`

### Other components

- **SQL sources**
- **Order API**
- **Web and mobile**
- **Power BI**

## Connections

- SQL sources → ADF orchestration: extracts changes
- Order API → ADF orchestration: extracts changes
- ADF orchestration → Landing files: writes extracts
- ADF orchestration → Batch job: starts job
- Landing files → Batch ingestion: reads files
- Batch job → Batch ingestion: runs batch
- Batch job → Silver batch: runs batch
- Batch job → Gold batch: runs batch
- Batch ingestion → Bronze tables: writes raw
- Batch ingestion → Quality and quarantine: quarantines records
- Web and mobile → Event Hubs: sends events
- Event Hubs → Stream ingestion: streams events
- Stream ingestion → Bronze tables: writes raw events
- Bronze tables → Silver batch: reads batch data
- Silver batch → Typed transforms: transforms entities
- Silver batch → Silver tables: merges tables
- Silver batch → Quality and quarantine: checks records
- Bronze tables → Clickstream silver: reads events
- Clickstream silver → Silver tables: merges events
- Clickstream silver → Quality and quarantine: checks records
- Silver tables → Gold batch: reads changes
- Gold batch → Facts and dimensions: builds tables
- Gold batch → Business aggregates: builds metrics
- Gold batch → Gold tables: writes gold
- Silver tables → Realtime metrics: reads events
- Realtime metrics → Gold tables: updates metrics
- Gold tables → Power BI: serves analytics
- Gold tables → SQL views: queried by
- Quality and quarantine → Control tables: records results
- Batch job → Control tables: audits runs
- Control tables → Monitoring queries: feeds monitoring