How raviteja000-maker/azure-etl-pipeline-with-databricks works
Project type and purpose
• This is a cloud-native, data-engineering pipeline built on Azure. Its primary goal is to ingest diverse raw data sources (HTTP-hosted CSVs, a MySQL table, and MongoDB), land them into an ADLS Gen2 “Bronze” zone, transform and enrich in Databricks into a “Silver” zone, and finally expose/query/write results via Synapse Analytics into a “Gold” zone.
File structure
• Top level:
• Databricks notebook (“Databricks Transformation.ipynb”) for Spark jobs
• “ForEachInput” JSON or parameter definition file for ADF
• “ProjectFlowSteps” folder containing the canonical pipeline image (ProjectFlowSteps/pipeline.png)
• “data” folder with all raw CSVs (olist_*.csv)
• This shows a clear separation: orchestration assets (ADF), raw data samples, and transformation code.
Key insights from README
• Ingestion in Azure Data Factory (ADF):
• HTTP-to-ADLS via a ForEach + Copy activity
• MySQL-to-ADLS via a direct Copy activity
• Later improvement: dynamic metadata via Lookup(HTTP JSON) feeding ForEach
• Bronze–Silver–Gold layered storage in ADLS Gen2
• Databricks: OAuth2 credential via Azure AD App Registration → read/write Bronze→Silver transformations, enrichment via MongoDB
• Synapse Analytics: external file formats, data sources, views and CETAS to Gold