# raviteja000-maker/azure-etl-pipeline-with-databricks architecture

> Architecture diagram of the GitHub repository raviteja000-maker/azure-etl-pipeline-with-databricks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raviteja000-maker/azure-etl-pipeline-with-databricks
- Repository: https://github.com/raviteja000-maker/azure-etl-pipeline-with-databricks

## Overview

##### Project type and purpose
- This is a cloud-native, data-engineering pipeline built on Azure. Its primary goal is to ingest diverse raw data sources (HTTP-hosted CSVs, a MySQL table, and MongoDB), land them into an ADLS Gen2 “Bronze” zone, transform and enrich in Databricks into a “Silver” zone, and finally expose/query/write results via Synapse Analytics into a “Gold” zone.

##### File structure
- Top level:
- Databricks notebook (“Databricks Transformation.ipynb”) for Spark jobs
- “ForEachInput” JSON or parameter definition file for ADF
- “ProjectFlowSteps” folder containing the canonical pipeline image (ProjectFlowSteps/pipeline.png)
- “data” folder with all raw CSVs (olist_*.csv)
- This shows a clear separation: orchestration assets (ADF), raw data samples, and transformation code.

##### Key insights from README
- Ingestion in Azure Data Factory (ADF):
- HTTP-to-ADLS via a ForEach + Copy activity
- MySQL-to-ADLS via a direct Copy activity
- Later improvement: dynamic metadata via Lookup(HTTP JSON) feeding ForEach
- Bronze–Silver–Gold layered storage in ADLS Gen2
- Databricks: OAuth2 credential via Azure AD App Registration → read/write Bronze→Silver transformations, enrichment via MongoDB
- Synapse Analytics: external file formats, data sources, views and CETAS to Gold

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    subgraph "Data Sources"
        classDef external fill:#f0f0f0,stroke:#888;
        GitHub["GitHub HTTP CSV Endpoint"]:::external
        Clever["Clever Cloud MySQL"]:::external
        MongoDB["MongoDB (Product Translations)"]:::external
        DataFolder["Raw CSV Samples (data/)"]:::external
    end

    %% Orchestration & Ingestion
    subgraph "Orchestration & Ingestion" 
        classDef orchestration fill:#cce5ff,stroke:#3399ff;
        Lookup["ADF: Lookup Activity"]:::orchestration
        ForEach["ADF: ForEach Activity"]:::orchestration
        CopyCSV["ADF: Copy Activity (HTTP→Bronze)"]:::orchestration
        CopyMySQL["ADF: Copy Activity (MySQL→Bronze)"]:::orchestration
        PipelineViz["ADF Pipeline Diagram"]:::orchestration
        ADFParams["ADF Parameters Definition (ForEachInput)"]:::orchestration
    end

    %% Storage
    subgraph "Storage (ADLS Gen2)"
        classDef storage fill:#d4edda,stroke:#28a745;
        Bronze["Bronze Container<br/>(raw CSVs)"]:::storage
        Silver["Silver Container<br/>(Parquet)"]:::storage
        Gold["Gold Container<br/>(External Tables)"]:::storage
    end

    %% Compute & Transformation
    subgraph "Compute & Transformation"
        classDef compute fill:#fff3cd,stroke:#ffcc00;
        Databricks["Databricks Notebook (Spark)"]:::compute
    end

    %% Analytics & Serving
    subgraph "Analytics & Serving"
        classDef analytics fill:#ffe5cc,stroke:#ff6600;
        Synapse["Synapse Analytics<br/>(Views & CETAS)"]:::analytics
    end

    %% Security & Configuration
    subgraph "Security & Configuration"
        classDef security fill:#e5ccff,stroke:#8000ff;
        ADApp["Azure AD App Registration"]:::security
        KeyVault["Key Vault / Databricks Secrets"]:::security
    end

    %% Connections
    GitHub -->|HTTP GET CSV| Lookup
    DataFolder -.->|sample files| Lookup
    Lookup -->|"metadata array"| ForEach
    ForEach -->|"iterates metadata"| CopyCSV
    Clever -->|JDBC Copy| CopyMySQL
    CopyCSV -->|write abfss://Bronze| Bronze
    CopyMySQL -->|write abfss://Bronze| Bronze
    Bronze -->|read Parquet| Databricks
    Databricks -->|write Parquet| Silver
    Silver -->|OPENROWSET| Synapse
    Synapse -->|CETAS / Views| Gold
    ADApp -->|credentials| KeyVault
    KeyVault -->|secrets| Databricks
    KeyVault -->|secrets| Synapse

    %% Click Events
    click DataFolder "https://github.com/raviteja000-maker/azure-etl-pipeline-with-databricks/tree/main/data/"
    click ADFParams "https://github.com/raviteja000-maker/azure-etl-pipeline-with-databricks/tree/main/ForEachInput"
    click PipelineViz "https://github.com/raviteja000-maker/azure-etl-pipeline-with-databricks/blob/main/ProjectFlowSteps/pipeline.png"
    click Databricks "https://github.com/raviteja000-maker/azure-etl-pipeline-with-databricks/blob/main/Databricks Transformation.ipynb"
    click README "https://github.com/raviteja000-maker/azure-etl-pipeline-with-databricks/blob/main/README.md"

    %% README Node
    README["Project Overview & Instructions (README.md)"]:::external

    %% Classes
    class GitHub,Clever,MongoDB,DataFolder external
    class Lookup,ForEach,CopyCSV,CopyMySQL,PipelineViz,ADFParams orchestration
    class Bronze,Silver,Gold storage
    class Databricks compute
    class Synapse analytics
    class ADApp,KeyVault security
```
