How atulgawade12/projectcarsale works
Project type and purpose
• This is a Cloud-based data engineering pipeline implementing the Medallion Architecture (Bronze → Silver → Gold) for car sales data.
• It ingests CSV data from GitHub/raw storage into Azure SQL, lands raw data in Azure Data Lake Storage (ADLS) Bronze, transforms via Databricks into a Silver “big table,” models a star schema in Gold, and exposes results to Power BI.
File structure
• Notebooks/ : Databricks notebooks for each stage (Database_notebook, Silver_notebook, Gold_dim_*.ipynb, Gold_fact_sales.ipynb)
• Queries/ : SQL scripts for table creation, stored procedures, watermark logic
• raw_data/ : sample CSV input files (SalesData.csv, IncrementalSales.csv)
• powerBi_databricks.pbids : Power BI data connection file
• README.md : detailed architecture, tools, workflow
README for additional insights
• Confirms layers: Bronze (Parquet in ADLS), Silver (transformed Parquet/Delta), Gold (Delta star schema)
• Lists tools: Azure Data Factory (ADF), Azure SQL DB, ADLS Gen2, Databricks (+ Unity Catalog), Power BI
• Describes incremental load via watermark table and SCD Type 1
Essential elements
• Component names: “ADF SourcePrep,” “ADF Incremental,” “Databricks Silver_notebook,” etc.
• Data formats: CSV, Parquet, Delta
• Incremental logic: watermark table → lookup → copy → stored procedure
• SCD Type 1 upserts in Gold notebooks
• Power BI connection via Databricks SQL endpoint