# yurilopesm/data-lake-revision architecture

> Architecture diagram of the GitHub repository yurilopesm/data-lake-revision, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yurilopesm/data-lake-revision
- Repository: https://github.com/yurilopesm/data-lake-revision

## Overview

##### Project Type and Purpose
- The project is a Data Lake/ETL practice repository built for data engineers. It shows how to provision cloud resources (ADLS) using Terraform, extract and load data using Python scripts, and transform data using Databricks notebooks.
- The practice scenario involves a SQL Server running in a Docker container, the extraction of data by Python scripts, storing data into Azure Data Lake (ADLS), and then processing it through successive transformation layers in Databricks (Bronze, Silver, Gold).

##### Analyzing the File Structure
- At the top-level, you have several configuration and environment files (e.g., .env.example, .terraform.lock.hcl) plus Terraform files (main.tf, provider.tf, output.tf, variables.tf) used to provision and configure Azure resources.
- The “python” folder contains ETL scripts (elt_sql_1_tabela.py, elt_sql_n_tabelas.py) and a pyproject.toml file for dependency management (using Poetry). These scripts are responsible for extracting data from a SQL Server database.
- The “databricks” folder holds multiple .dbc notebook files which represent different layers of a typical data lake pipeline (Bronze, Silver, Gold). Each notebook is tailored to a step in the data transformation process.
- The README outlines the end-to-end steps including launching a SQL Server container, configuring Terraform, setting up ADLS via Azure CLI, executing Python ETL scripts, and importing and running Databricks notebooks.

##### Main Components
a. Infrastructure Provisioning:
- Azure CLI and Environment: The .env file (copied from .env.example) provides necessary environment variables (like ADLS_ACCOUNT_NAME and ADLS_SAS_TOKEN).
b. Data Source:
- Dockerized SQL Server: The instructions specify creating a SQL Server container using Docker (with SSMS for querying). This acts as the primary data source.
c. Data Extraction and ETL:
- Python Scripts: Under the “python” folder, the extraction logic sits in scripts (elt_sql_1_tabela.py, elt_sql_n_tabelas.py). They are responsible for pulling data from SQL Server and likely pushing it into the Data Lake.
d. Data Lake and Transformation:
- Azure Data Lake Storage (ADLS): Provisioned via Terraform. This is where the raw (ingested) data is stored.
- Databricks Notebooks – Bronze/Silver/Gold: Each .dbc file in the “databricks” folder represents the transformation layers where:
- Bronze: Ingestion of raw landing data.
- Silver: Data cleaning and intermediate transformations.
- Gold: Final data aggregations or business-friendly views.

##### Relationships and Interactions
- The ETL step loads data into the ADLS that was pre-provisioned via Terraform.
- Next, the Databricks notebooks connect to the ADLS, process the data through the Bronze (raw ingestion), Silver (data cleaning/transformation), and Gold (refined data) stages.
- The overall infrastructure is controlled via Terraform (with configuration files and variables), and Azure resources are managed through Azure CLI commands.

##### Architectural Patterns and Design Principles
- Infrastructure as Code (IaC): The use of Terraform ensures that cloud resources are provisioned in a reproducible and maintainable manner.
- Layered Data Pipeline: The separation into Bronze, Silver, and Gold layers is a key architectural pattern in data lakes for ensuring clean, transformed data products.
- Modular Design: Separation of concerns across infrastructure provisioning (Terraform), data extraction (Python scripts), and data transformation (Databricks notebooks).

##### Relevant Technologies
- Docker – for running the SQL Server as an isolated service.
- Terraform – for provisioning and managing Azure cloud resources, particularly the ADLS account.
- Azure Cloud and Azure Data Lake Storage (ADLS) – cloud storage and resource management.
- Python (with Poetry for dependency management) – for ETL operations.
- Databricks – for large-scale data processing and layered transformations.
- Azure CLI – for managing and interfacing with Azure resources.
- SQL Server Management Studio (SSMS) – to connect to and manage the SQL Server container.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    A["Docker SQL Server (Docker Container)"]:::source

    %% Infrastructure Layer
    subgraph "Infrastructure Layer"
        T1["Provider Config (Terraform)"]:::infra
        T2["Terraform Main"]:::infra
        T3["Terraform Outputs"]:::infra
        T4["Terraform Variables"]:::infra
        E1["Env Variables (.env.example)"]:::infra
        E2["Terraform Lock (.terraform.lock.hcl)"]:::infra
        ADLS["Azure Data Lake Storage (ADLS) Provisioning via Terraform"]:::infra
    end

    %% ETL Layer
    subgraph "ETL Layer"
        P1["Python ETL - Single Table"]:::etl
        P2["Python ETL - Multi Table"]:::etl
        PM["Dependency Management (pyproject.toml)"]:::etl
    end

    %% Data Transformation Layer
    subgraph "Data Transformation Layer (Databricks)"
        D1["Databricks Bronze Notebook"]:::process
        D2["Databricks Silver Notebook"]:::process
        D3["Databricks Gold Notebook"]:::process
    end

    %% Relationships
    A -->|"extracts"| P1
    A -->|"extracts"| P2
    P1 -->|"loads"| ADLS
    P2 -->|"loads"| ADLS
    T1 -->|"provisions"| ADLS
    T2 -->|"provisions"| ADLS
    T3 -->|"provisions"| ADLS
    T4 -->|"provisions"| ADLS
    E1 -->|"env-config"| ADLS
    E2 -->|"lock-config"| ADLS
    ADLS -->|"ingests"| D1
    D1 -->|"transforms"| D2
    D2 -->|"transforms"| D3

    %% Click Events: Infrastructure Provisioning
    click T1 "https://github.com/yurilopesm/data-lake-revision/blob/master/provider.tf"
    click T2 "https://github.com/yurilopesm/data-lake-revision/blob/master/main.tf"
    click T3 "https://github.com/yurilopesm/data-lake-revision/blob/master/output.tf"
    click T4 "https://github.com/yurilopesm/data-lake-revision/blob/master/variables.tf"
    click E1 "https://github.com/yurilopesm/data-lake-revision/blob/master/.env.example"
    click E2 "https://github.com/yurilopesm/data-lake-revision/blob/master/.terraform.lock.hcl"

    %% Click Events: Data Extraction and ETL
    click P1 "https://github.com/yurilopesm/data-lake-revision/blob/master/python/elt_sql_1_tabela.py"
    click P2 "https://github.com/yurilopesm/data-lake-revision/blob/master/python/elt_sql_n_tabelas.py"
    click PM "https://github.com/yurilopesm/data-lake-revision/blob/master/python/pyproject.toml"

    %% Click Events: Data Transformation (Databricks Notebooks)
    click D1 "https://github.com/yurilopesm/data-lake-revision/blob/master/databricks/Atividade Pratica - Lakehouse - Bronze.dbc"
    click D2 "https://github.com/yurilopesm/data-lake-revision/blob/master/databricks/Atividade Pratica - Lakehouse - Silver.dbc"
    click D3 "https://github.com/yurilopesm/data-lake-revision/blob/master/databricks/Atividade Pratica - Lakehouse - Gold.dbc"

    %% Styles
    classDef source fill:#ffcccc,stroke:#cc0000,stroke-width:2px;
    classDef infra fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef etl fill:#ccccff,stroke:#000099,stroke-width:2px;
    classDef process fill:#ffffcc,stroke:#999900,stroke-width:2px;
```
