# jcarlosbatista/lakehouse architecture

> Architecture diagram of the GitHub repository jcarlosbatista/lakehouse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jcarlosbatista/lakehouse
- Repository: https://github.com/jcarlosbatista/lakehouse

## Overview

##### Project Type and Purpose
- This is a hands-on, lab-style data-lakehouse reference implementation built on Azure Synapse Analytics.
- Purpose: Demonstrate how to ingest, transform, and expose data using raw (“01-raw”) and curated (“03-curated”) zones, Synapse pipelines/SQL pools, and triggers, ending in visualization.

##### Key Architectural Components
a. Azure Resource Provisioning
- Azure Bicep / ARM templates (`template/azuredeploy.bicep`, `azuredeploy.json`)
- PowerShell bootstrap script (`lakehouselab.ps1`)
b. Storage Layers (Data Lake)
- Raw Zone (ADLS Gen2 containers under `output/moduleXX/01-raw`)
- Curated Zone (Delta Lake under `output/moduleXX/03-curated`)
c. Compute & Processing
- Azure Synapse Pipelines (copy activities, data flows)
- Synapse Spark or SQL pool for transformations (SCD Type 2 logic, fact/dimension loads)
d. Orchestration & Automation
- Event- or schedule-based triggers (Modules 01d & 02c)
e. Consumption & Visualization
- Logical Data Warehouse logical layer (Module03)
- Power BI or other tools for Data Visualization (Module04)

1. Deployment & Infrastructure
2. Ingestion (Raw Zone)
3. Transformation (Curated Zone)
4. Logical/Presentation Layer
5. Visualization/Reporting
b. Major Component:
- Modules by function: Module01 (Customers), Module02 (Orders), Module03 (Logical), Module04 (Visualization).
d. Automation:
e. Technologies & Artifacts:
- Bicep template files and PS script under Deployment layer.
- Under each pipeline, list pipeline names matching modules (e.g., “Module01a_Pipeline”).
- Under Storage, annotate path patterns (`/wwi/customers`, `/wwi/orders`).

##### Architectural Patterns & Principles
- Medallion (bronze → silver) pattern: raw (“bronze”) → curated (“silver”) → logical/semantic (“gold”).
- Incremental load strategies: CDC for dimension tables, high-watermark for fact tables.
- Delta Lake for ACID, versioning, time travel.
- Infrastructure as Code (Bicep/ARM + PowerShell).

##### Final Touches
- A “Modules” container subdivided into 1A, 1B, 1C, 1D, 2A, 2B, 2C, etc.
- Callouts for important `_delta_log` directories, demonstrating Delta Lake internals.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Deployment & Infrastructure"
        Bicep["Azure Bicep Template"]:::deployment
        ARM["ARM Template (azuredeploy.json)"]:::deployment
        PS["PowerShell Script"]:::deployment
    end

    subgraph "Raw Zone (01-raw)"
        CustRaw1["ADLS Gen2 /wwi/customers (Module01a)"]:::storage
        CustRaw2["ADLS Gen2 /wwi/customers (Module01d)"]:::storage
        OrdersRaw1["ADLS Gen2 /wwi/orders (Module02a)"]:::storage
        OrdersRaw2["ADLS Gen2 /wwi/orders (Module02c)"]:::storage
    end

    subgraph "Compute & Processing"
        P01a["Module01a_Pipeline"]:::compute
        P01b["Module01b_Pipeline"]:::compute
        P01c["Module01c_Pipeline"]:::compute
        P02a["Module02a_Pipeline"]:::compute
        P02b["Module02b_Pipeline"]:::compute
    end

    subgraph "Orchestration & Triggers"
        T01d["Module01d_Trigger"]:::orchestration
        T02c["Module02c_Trigger"]:::orchestration
    end

    subgraph "Curated Zone (03-curated)"
        CustCur1["Delta Lake /wwi/customers (Module01b)"]:::storage
        CustCur2["Delta Lake /wwi/customers (Module01c)"]:::storage
        CustCur3["Delta Lake /wwi/customers (Module01d)"]:::storage
        OrdersCur1["Delta Lake /wwi/orders (Module02b)"]:::storage
        OrdersCur2["Delta Lake /wwi/orders (Module02c)"]:::storage
    end

    subgraph "Logical/Presentation Layer"
        Logical["SQL Views & Semantic Layer (Module03)"]:::compute
    end

    subgraph "Visualization & Reporting"
        Viz["Power BI Dashboards (Module04)"]:::viz
    end

    %% Data Flow
    Bicep --> CustRaw1
    Bicep --> OrdersRaw1

    P01a --> CustRaw1
    P02a --> OrdersRaw1

    CustRaw1 --> P01b
    P01b --> CustCur1
    CustCur1 --> P01c
    P01c --> CustCur2
    CustCur2 --> Logical

    OrdersRaw1 --> P02b
    P02b --> OrdersCur1
    OrdersCur1 --> P02a
    P02a --> OrdersCur2
    OrdersCur2 --> Logical

    Logical --> Viz

    %% Triggers
    T01d -.-> P01a
    T01d -.-> P01b
    T01d -.-> P01c
    T02c -.-> P02a
    T02c -.-> P02b

    %% Click Events
    click Bicep "https://github.com/jcarlosbatista/lakehouse/blob/main/template/azuredeploy.bicep"
    click ARM "https://github.com/jcarlosbatista/lakehouse/blob/main/template/azuredeploy.json"
    click PS "https://github.com/jcarlosbatista/lakehouse/blob/main/template/lakehouselab.ps1"
    click CustRaw1 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module01a/01-raw/wwi/customers/"
    click CustRaw2 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module01d/01-raw/wwi/customers/"
    click OrdersRaw1 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module02a/01-raw/wwi/orders/"
    click OrdersRaw2 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module02c/01-raw/wwi/orders/"
    click CustCur1 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module01b/03-curated/wwi/customers/"
    click CustCur2 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module01c/03-curated/wwi/customers/"
    click CustCur3 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module01d/03-curated/wwi/customers/"
    click OrdersCur1 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module02b/03-curated/wwi/orders/"
    click OrdersCur2 "https://github.com/jcarlosbatista/lakehouse/tree/main/output/module02c/03-curated/wwi/orders/"
    click P01a "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module01a.md"
    click P01b "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module01b.md"
    click P01c "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module01c.md"
    click P02a "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module02a.md"
    click P02b "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module02b.md"
    click T01d "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module01d.md"
    click T02c "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module02c.md"
    click Logical "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module03.md"
    click Viz "https://github.com/jcarlosbatista/lakehouse/blob/main/modules/module04.md"
    click Viz "https://github.com/jcarlosbatista/lakehouse/tree/main/images/module04"

    %% Styles
    classDef storage fill:#cce5ff,stroke:#005cbf,color:#005cbf
    classDef compute fill:#ffe5b4,stroke:#ff8c00,color:#d2691e
    classDef orchestration fill:#d4edda,stroke:#28a745,color:#155724
    classDef deployment fill:#e2e3f3,stroke:#6f42c1,color:#4e2671
    classDef viz fill:#f8f9fa,stroke:#6c757d,color:#343a40
```
