How jcarlosbatista/lakehouse works
Project Type and Purpose
• This is a hands-on, lab-style data-lakehouse reference implementation built on Azure Synapse Analytics.
• Purpose: Demonstrate how to ingest, transform, and expose data using raw (“01-raw”) and curated (“03-curated”) zones, Synapse pipelines/SQL pools, and triggers, ending in visualization.
Key Architectural Components
a. Azure Resource Provisioning
• Azure Bicep / ARM templates (template/azuredeploy.bicep, azuredeploy.json)
• PowerShell bootstrap script (lakehouselab.ps1)
b. Storage Layers (Data Lake)
• Raw Zone (ADLS Gen2 containers under output/moduleXX/01-raw)
• Curated Zone (Delta Lake under output/moduleXX/03-curated)
c. Compute & Processing
• Azure Synapse Pipelines (copy activities, data flows)
• Synapse Spark or SQL pool for transformations (SCD Type 2 logic, fact/dimension loads)
d. Orchestration & Automation
• Event- or schedule-based triggers (Modules 01d & 02c)
e. Consumption & Visualization
• Logical Data Warehouse logical layer (Module03)
• Power BI or other tools for Data Visualization (Module04)
1. Deployment & Infrastructure
2. Ingestion (Raw Zone)
3. Transformation (Curated Zone)
4. Logical/Presentation Layer
5. Visualization/Reporting
b. Major Component:
• Modules by function: Module01 (Customers), Module02 (Orders), Module03 (Logical), Module04 (Visualization).
d. Automation:
e. Technologies & Artifacts:
• Bicep template files and PS script under Deployment layer.
• Under each pipeline, list pipeline names matching modules (e.g., “Module01a_Pipeline”).
• Under Storage, annotate path patterns (/wwi/customers, /wwi/orders).
Architectural Patterns & Principles
• Medallion (bronze → silver) pattern: raw (“bronze”) → curated (“silver”) → logical/semantic (“gold”).
• Incremental load strategies: CDC for dimension tables, high-watermark for fact tables.
• Delta Lake for ACID, versioning, time travel.
• Infrastructure as Code (Bicep/ARM + PowerShell).
Final Touches
• A “Modules” container subdivided into 1A, 1B, 1C, 1D, 2A, 2B, 2C, etc.
• Callouts for important _delta_log directories, demonstrating Delta Lake internals.