How nirmalkumar31/e-commerce-sales--azure-data-pipeline works
This repository documents an end-to-end incremental sales data warehouse pipeline: ingest sales data, track incremental loads, transform through Bronze and Silver layers, and produce Gold star-schema tables and visualizations. The tree contains CSV data, a stored-procedure artifact, and exported Databricks notebooks, but no ADF pipeline definitions. Relationships below follow README descriptions; unsampled implementation details are not inferred.
Main entry points: Silver_Notebook.html Silver analysis · Sales_Data.csv Sales data · Stored_Procedure SQL code Watermark procedure · Gold_Branch_Dimension.html Branch dimension
Components 18
Ingestion and watermark
- Sales data
dataset/Sales_Data.csv - Incremental sales
dataset/Incremental_Sales.csv - ADF pipeline
- SQL sales table
- Watermark table
- Watermark procedure
Stored_Procedure SQL code
Databricks processing
- ADLS Bronze
- Databricks job
- Bronze to Silver
databricks workbooks/Silver_Notebook.html
Gold warehouse
- Branch dimension
databricks workbooks/Gold_Branch_Dimension.html - Date dimension
databricks workbooks/Gold_Date_Dimension.html - Dealer dimension
databricks workbooks/Gold_Dealer_Dimension.html - Model dimension
databricks workbooks/Gold_Model_Dimension.html - Sales fact
databricks workbooks/Gold_Fact_Sales.html - Gold Delta tables
Analysis and insights
- Silver analysis
databricks workbooks/Silver_Notebook.html - Sales visualizations
Visualizations and project resource details
Other components
- GitHub source
Connections 23
- GitHub source → Sales data: provides source
README.md - Sales data → SQL sales table: initial copy
README.md - Incremental sales → ADF pipeline: provides updates
- ADF pipeline → Watermark table: reads watermark
README.md - ADF pipeline → SQL sales table: finds new rows
README.md - ADF pipeline → ADLS Bronze: copies new rows
README.md - ADF pipeline → Watermark procedure: runs after success
README.md - Watermark procedure → Watermark table: updates watermark
README.md - ADF pipeline → Databricks job: triggers notebooks
README.md - Databricks job → Bronze to Silver: executes notebook
README.md - Bronze to Silver → ADLS Bronze: loads Bronze
README.md - Bronze to Silver → Silver analysis: transforms to Silver
README.md - Silver analysis → Branch dimension: builds dimension
README.md - Silver analysis → Date dimension: builds dimension
README.md - Silver analysis → Dealer dimension: builds dimension
README.md - Silver analysis → Model dimension: builds dimension
README.md - Silver analysis → Sales fact: builds fact
README.md - Branch dimension → Gold Delta tables: stores dimension
README.md - Date dimension → Gold Delta tables: stores dimension
README.md - Dealer dimension → Gold Delta tables: stores dimension
README.md - Model dimension → Gold Delta tables: stores dimension
README.md - Sales fact → Gold Delta tables: stores fact
README.md - Gold Delta tables → Sales visualizations: supports analysis
README.md