How abhishekshah25/3-layer-medallion-data-pipeline works
Project type and purpose
• This is a data-engineering project implementing a three-layer Medallion Architecture pipeline on Azure.
• It uses Azure Data Factory for orchestration, Azure Databricks for Spark-based processing, and DBT for modular SQL transformations.
File structure
• Top-level:
• Procedure.pdf: setup/run instructions
• databricks_base_notebook.py: base code for Databricks notebooks
• medallion_dbt_spark/: DBT project
• medallion_dbt_spark/ contains:
• dbt_project.yml: project configuration
• models/staging/: “bronze” raw data staging layer
• models/marts/…/dim_*.sql, sales.sql: “gold” presentation-ready marts
• macros/, analyses/, seeds/, snapshots/: DBT features for reusable code, ad-hoc analysis, static data, change-data-capture snapshots
• No frontend or backend directories – it’s purely an ETL pipeline.
README for additional insights
• Confirms a 3-layer system (bronze/silver/gold) on Azure
• Highlights Azure Data Factory, Databricks, DBT components
• Emphasizes modularity, scalability, automated workflows, version control