# nirmalkumar31/e-commerce-sales--azure-data-pipeline architecture

> Architecture diagram of the GitHub repository nirmalkumar31/e-commerce-sales--azure-data-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline
- Repository: https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline

## Overview

This repository documents an end-to-end incremental sales data warehouse pipeline: ingest sales data, track incremental loads, transform through Bronze and Silver layers, and produce Gold star-schema tables and visualizations. The tree contains CSV data, a stored-procedure artifact, and exported Databricks notebooks, but no ADF pipeline definitions. Relationships below follow README descriptions; unsampled implementation details are not inferred.

Main entry points: [`databricks workbooks/Silver_Notebook.html`](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Silver_Notebook.html) (Silver analysis), [`dataset/Sales_Data.csv`](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/dataset/Sales_Data.csv) (Sales data), [`Stored_Procedure SQL code`](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/Stored_Procedure%20SQL%20code) (Watermark procedure), [`databricks workbooks/Gold_Branch_Dimension.html`](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Branch_Dimension.html) (Branch dimension)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingestion["Ingestion and watermark"]
  node_sales_csv["Sales data<br/>[Sales_Data.csv]"]
  node_incremental_csv["Incremental sales"]
  node_adf["ADF pipeline"]
  node_sql_sales[("SQL sales table")]
  node_watermark[("Watermark table")]
  node_watermark_proc["Watermark procedure"]
end

subgraph group_processing["Databricks processing"]
  node_adls_bronze[("ADLS Bronze")]
  node_dbx_job["Databricks job"]
  node_bronze_silver["Bronze to Silver"]
end

subgraph group_warehouse["Gold warehouse"]
  node_gold_branch["Branch dimension"]
  node_gold_date["Date dimension"]
  node_gold_dealer["Dealer dimension"]
  node_gold_model["Model dimension"]
  node_gold_fact["Sales fact"]
  node_gold_delta[("Gold Delta tables")]
end

subgraph group_analytics["Analysis and insights"]
  node_silver_analysis["Silver analysis"]
  node_visualizations["Sales visualizations"]
end

node_github(("GitHub source"))

node_github -->|"provides source"| node_sales_csv
node_sales_csv -->|"initial copy"| node_sql_sales
node_incremental_csv -.->|"provides updates"| node_adf
node_adf -->|"reads watermark"| node_watermark
node_adf -->|"finds new rows"| node_sql_sales
node_adf -->|"copies new rows"| node_adls_bronze
node_adf -->|"runs after success"| node_watermark_proc
node_watermark_proc -->|"updates watermark"| node_watermark
node_adf -->|"triggers notebooks"| node_dbx_job
node_dbx_job -->|"executes notebook"| node_bronze_silver
node_bronze_silver -->|"loads Bronze"| node_adls_bronze
node_bronze_silver -->|"transforms to Silver"| node_silver_analysis
node_silver_analysis -->|"builds dimension"| node_gold_branch
node_silver_analysis -->|"builds dimension"| node_gold_date
node_silver_analysis -->|"builds dimension"| node_gold_dealer
node_silver_analysis -->|"builds dimension"| node_gold_model
node_silver_analysis -->|"builds fact"| node_gold_fact
node_gold_branch -->|"stores dimension"| node_gold_delta
node_gold_date -->|"stores dimension"| node_gold_delta
node_gold_dealer -->|"stores dimension"| node_gold_delta
node_gold_model -->|"stores dimension"| node_gold_delta
node_gold_fact -->|"stores fact"| node_gold_delta
node_gold_delta -->|"supports analysis"| node_visualizations

click node_sales_csv "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/dataset/Sales_Data.csv"
click node_incremental_csv "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/dataset/Incremental_Sales.csv"
click node_watermark_proc "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/Stored_Procedure%20SQL%20code"
click node_bronze_silver "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Silver_Notebook.html"
click node_silver_analysis "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Silver_Notebook.html"
click node_gold_branch "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Branch_Dimension.html"
click node_gold_date "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Date_Dimension.html"
click node_gold_dealer "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Dealer_Dimension.html"
click node_gold_model "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Model_Dimension.html"
click node_gold_fact "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Fact_Sales.html"
click node_visualizations "https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/tree/main/Visualizations%20and%20project%20resource%20details"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sales_csv,node_incremental_csv,node_adf,node_sql_sales,node_watermark,node_watermark_proc toneBlue
class node_adls_bronze,node_dbx_job,node_bronze_silver toneAmber
class node_gold_branch,node_gold_date,node_gold_dealer,node_gold_model,node_gold_fact,node_gold_delta toneMint
class node_silver_analysis,node_visualizations toneRose
class node_github toneIndigo
```

## Components

### Ingestion and watermark

- [Sales data](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/dataset/Sales_Data.csv): `dataset/Sales_Data.csv`
- [Incremental sales](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/dataset/Incremental_Sales.csv): `dataset/Incremental_Sales.csv`
- **ADF pipeline**
- **SQL sales table**
- **Watermark table**
- [Watermark procedure](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/Stored_Procedure%20SQL%20code): `Stored_Procedure SQL code`

### Databricks processing

- **ADLS Bronze**
- **Databricks job**
- [Bronze to Silver](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Silver_Notebook.html): `databricks workbooks/Silver_Notebook.html`

### Gold warehouse

- [Branch dimension](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Branch_Dimension.html): `databricks workbooks/Gold_Branch_Dimension.html`
- [Date dimension](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Date_Dimension.html): `databricks workbooks/Gold_Date_Dimension.html`
- [Dealer dimension](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Dealer_Dimension.html): `databricks workbooks/Gold_Dealer_Dimension.html`
- [Model dimension](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Model_Dimension.html): `databricks workbooks/Gold_Model_Dimension.html`
- [Sales fact](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Gold_Fact_Sales.html): `databricks workbooks/Gold_Fact_Sales.html`
- **Gold Delta tables**

### Analysis and insights

- [Silver analysis](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/blob/main/databricks%20workbooks/Silver_Notebook.html): `databricks workbooks/Silver_Notebook.html`
- [Sales visualizations](https://github.com/nirmalkumar31/e-commerce-sales--azure-data-pipeline/tree/main/Visualizations%20and%20project%20resource%20details): `Visualizations and project resource details`

### Other components

- **GitHub source**

## Connections

- GitHub source → Sales data: provides source (evidence: `README.md`)
- Sales data → SQL sales table: initial copy (evidence: `README.md`)
- Incremental sales → ADF pipeline: provides updates
- ADF pipeline → Watermark table: reads watermark (evidence: `README.md`)
- ADF pipeline → SQL sales table: finds new rows (evidence: `README.md`)
- ADF pipeline → ADLS Bronze: copies new rows (evidence: `README.md`)
- ADF pipeline → Watermark procedure: runs after success (evidence: `README.md`)
- Watermark procedure → Watermark table: updates watermark (evidence: `README.md`)
- ADF pipeline → Databricks job: triggers notebooks (evidence: `README.md`)
- Databricks job → Bronze to Silver: executes notebook (evidence: `README.md`)
- Bronze to Silver → ADLS Bronze: loads Bronze (evidence: `README.md`)
- Bronze to Silver → Silver analysis: transforms to Silver (evidence: `README.md`)
- Silver analysis → Branch dimension: builds dimension (evidence: `README.md`)
- Silver analysis → Date dimension: builds dimension (evidence: `README.md`)
- Silver analysis → Dealer dimension: builds dimension (evidence: `README.md`)
- Silver analysis → Model dimension: builds dimension (evidence: `README.md`)
- Silver analysis → Sales fact: builds fact (evidence: `README.md`)
- Branch dimension → Gold Delta tables: stores dimension (evidence: `README.md`)
- Date dimension → Gold Delta tables: stores dimension (evidence: `README.md`)
- Dealer dimension → Gold Delta tables: stores dimension (evidence: `README.md`)
- Model dimension → Gold Delta tables: stores dimension (evidence: `README.md`)
- Sales fact → Gold Delta tables: stores fact (evidence: `README.md`)
- Gold Delta tables → Sales visualizations: supports analysis (evidence: `README.md`)
