How yurilopesm/data-lake-revision works
Project Type and Purpose
• The project is a Data Lake/ETL practice repository built for data engineers. It shows how to provision cloud resources (ADLS) using Terraform, extract and load data using Python scripts, and transform data using Databricks notebooks.
• The practice scenario involves a SQL Server running in a Docker container, the extraction of data by Python scripts, storing data into Azure Data Lake (ADLS), and then processing it through successive transformation layers in Databricks (Bronze, Silver, Gold).
Analyzing the File Structure
• At the top-level, you have several configuration and environment files (e.g., .env.example, .terraform.lock.hcl) plus Terraform files (main.tf, provider.tf, output.tf, variables.tf) used to provision and configure Azure resources.
• The “python” folder contains ETL scripts (elt_sql_1_tabela.py, elt_sql_n_tabelas.py) and a pyproject.toml file for dependency management (using Poetry). These scripts are responsible for extracting data from a SQL Server database.
• The “databricks” folder holds multiple .dbc notebook files which represent different layers of a typical data lake pipeline (Bronze, Silver, Gold). Each notebook is tailored to a step in the data transformation process.
• The README outlines the end-to-end steps including launching a SQL Server container, configuring Terraform, setting up ADLS via Azure CLI, executing Python ETL scripts, and importing and running Databricks notebooks.
Main Components
a. Infrastructure Provisioning:
• Azure CLI and Environment: The .env file (copied from .env.example) provides necessary environment variables (like ADLS_ACCOUNT_NAME and ADLS_SAS_TOKEN).
b. Data Source:
• Dockerized SQL Server: The instructions specify creating a SQL Server container using Docker (with SSMS for querying). This acts as the primary data source.
c. Data Extraction and ETL:
• Python Scripts: Under the “python” folder, the extraction logic sits in scripts (elt_sql_1_tabela.py, elt_sql_n_tabelas.py). They are responsible for pulling data from SQL Server and likely pushing it into the Data Lake.
d. Data Lake and Transformation:
• Azure Data Lake Storage (ADLS): Provisioned via Terraform. This is where the raw (ingested) data is stored.
• Databricks Notebooks – Bronze/Silver/Gold: Each .dbc file in the “databricks” folder represents the transformation layers where:
• Bronze: Ingestion of raw landing data.
• Silver: Data cleaning and intermediate transformations.
• Gold: Final data aggregations or business-friendly views.
Relationships and Interactions
• The ETL step loads data into the ADLS that was pre-provisioned via Terraform.
• Next, the Databricks notebooks connect to the ADLS, process the data through the Bronze (raw ingestion), Silver (data cleaning/transformation), and Gold (refined data) stages.
• The overall infrastructure is controlled via Terraform (with configuration files and variables), and Azure resources are managed through Azure CLI commands.
Architectural Patterns and Design Principles
• Infrastructure as Code (IaC): The use of Terraform ensures that cloud resources are provisioned in a reproducible and maintainable manner.
• Layered Data Pipeline: The separation into Bronze, Silver, and Gold layers is a key architectural pattern in data lakes for ensuring clean, transformed data products.
• Modular Design: Separation of concerns across infrastructure provisioning (Terraform), data extraction (Python scripts), and data transformation (Databricks notebooks).
Relevant Technologies
• Docker – for running the SQL Server as an isolated service.
• Terraform – for provisioning and managing Azure cloud resources, particularly the ADLS account.
• Azure Cloud and Azure Data Lake Storage (ADLS) – cloud storage and resource management.
• Python (with Poetry for dependency management) – for ETL operations.
• Databricks – for large-scale data processing and layered transformations.
• Azure CLI – for managing and interfacing with Azure resources.
• SQL Server Management Studio (SSMS) – to connect to and manage the SQL Server container.