How mrnidhi/etl-azure works
Project type and purpose
• This is an end-to-end Azure data pipeline project implementing the Medallion (Bronze→Silver→Gold) architecture.
• It ingests raw Olist CSV files (from GitHub and a sample SQL DB), lands them in ADLS Gen2, applies transformations in Databricks, and serves the cleansed/aggregated data to Azure Synapse and Power BI.
File structure
• Top-level folders:
– ADF/
· linkedService/ – JSON definitions for all source & sink connections (GitHub, HTTP, SQL DB, ADLS Gen2, Synapse).
· dataset/ – JSON definitions for CSV, JSON, SQL table datasets.
· pipeline/ – DataProcessingPipeLine.json describing the ETL orchestration.
· factory/ – factory metadata exports.
– DataBricks/
· PySpark scripts for Silver & Gold layers.
• CSV files at root are the raw source data for Bronze.
• SQLSynapsePath.json and the notebook show optional local-to-DB ingestion steps.
README for additional insights
• The README explicitly describes a Medallion pattern:
Bronze = raw files in ADLS
Silver = cleaned/standardized tables in ADLS
Gold = aggregated, business-ready tables in ADLS → Synapse
• Key technologies: Azure Data Factory, ADLS Gen2, Azure Databricks (PySpark), Azure Synapse Analytics, Power BI.