# jlsilva01/projeto-elt-satc architecture

> Architecture diagram of the GitHub repository jlsilva01/projeto-elt-satc, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jlsilva01/projeto-elt-satc
- Repository: https://github.com/jlsilva01/projeto-elt-satc

## Overview

##### Project Type and Purpose
- This is an open-source ELT tool written in Python 3.12.
- It connects to source databases (SQL Server, MongoDB) and writes data into Azure Data Lake Storage (ADLS).
- It uses Terraform (in `iac/adls`) to provision ADLS resources.

##### Main Components
A. Source Systems
- SQL Server (relational DB)
- (Optional) MongoDB
B. ELT Application (Python modules under `elt/`)
- `database/sql_server_service.py` – SQL Server connector & extract logic
- `azure_integration/adls_service.py` – ADLS upload logic
- `main.py` – orchestration (instantiates services, loops through tables/collections)
- Example scripts at project root & `examples/` for one-off runs
C. Configuration & Secrets
- `.env` file (loads with python-dotenv) – connection strings, SAS tokens
- `pyproject.toml` & `uv.lock` – dependency management via UV
D. Infrastructure as Code
- Terraform modules in `iac/adls/` – defines ADLS account, file system, directories
E. CI / Testing
- Tests under `test/` verify connectivity to SQL Server, ADLS, MongoDB

##### Relationships & Data Flow
1. Terraform provisions ADLS
→ (ADLS account, file system ready)
2. ELT Application reads `.env` → authenticates to SQL Server and/or MongoDB
3. SQL Server Service queries tables → pulls data (Pandas, SQLAlchemy, pyodbc)
4. ADLS Service writes Parquet/CSV files to Data Lake via Azure SDK (azure-storage-file-datalake)
5. (Optional) MongoDB Service could read & transform collections → load to ADLS as well
6. Tests run as smoke connectivity checks

##### Architectural Patterns & Design Principles
- Layered/Modular: separate “source connectors” (in `database/`), “sink connectors” (in `azure_integration/`), and orchestration (`main.py`).
- Adapter/Plugin: to add a new source or destination, simply drop a new service module into the appropriate folder and register it in `main.py`.
- Infrastructure-as-Code for reproducible resource provisioning.
- Environment-based configuration (12-factor app style).

##### Essential Elements
– Clear component labels and descriptions
· Data Sources
· Application Services
· Infrastructure
· Configuration

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Source Systems
    subgraph "Source Systems"
        direction TB
        SQLServer["SQL Server"]:::data
        MongoDB["MongoDB (Optional)"]:::data
    end

    %% Configuration
    subgraph "Configuration & Dependencies"
        direction TB
        Env[".env (python-dotenv)"]:::config
        ProjectCfg["pyproject.toml<br/>uv.lock"]:::config
    end

    %% Infrastructure as Code
    subgraph "Infrastructure as Code"
        direction TB
        Terraform["Terraform (iac/adls/)"]:::infra
    end

    %% ELT Application
    subgraph "ELT Application (Python 3.12 + UV)"
        direction TB
        Main["Main (Orchestration)"]:::app
        SQLService["SQLServerService<br/>(pyodbc, SQLAlchemy)"]:::app
        ADLSService["ADLSService<br/>(azure-storage-file-datalake)"]:::app
    end

    %% Sinks
    subgraph "Data Lake Storage"
        direction TB
        ADLS["ADLS<br/>(Data Lake Storage)"]:::data
    end

    %% Examples
    subgraph "Example Scripts"
        direction TB
        Ex1["elt_sql_1_tabela.py"]:::app
        Ex2["elt_sql_n_tabelas.py"]:::app
        Ex3["elt_mongodb_n_collections.py"]:::app
    end

    %% Tests
    subgraph "CI / Connectivity Tests"
        direction TB
        TestSQL["test_connection_sqlserver.py"]:::test
        TestADLS["test_connection_adls.py"]:::test
        TestMongo["test_connection_mongodb.py"]:::test
    end

    %% Relationships
    Terraform -->|"provisions ADLS"| ADLS
    Env -->|"provides config to"| Main
    ProjectCfg -->|"dependency info to"| Main

    SQLServer -->|extracts data via| SQLService
    MongoDB -.->|optional extract| SQLService

    Main -->|"calls"| SQLService
    Main -->|"calls"| ADLSService

    SQLService -->|"passes data to"| ADLSService
    ADLSService -->|"writes to"| ADLS

    Ex1 -->|invokes| Main
    Ex2 -->|invokes| Main
    Ex3 -->|invokes| Main

    TestSQL -->|tests| SQLServer
    TestADLS -->|tests| ADLS
    TestMongo -->|tests| MongoDB

    %% Click Events
    click SQLService "https://github.com/jlsilva01/projeto-elt-satc/blob/main/elt/database/sql_server_service.py"
    click ADLSService "https://github.com/jlsilva01/projeto-elt-satc/blob/main/elt/azure_integration/adls_service.py"
    click Main "https://github.com/jlsilva01/projeto-elt-satc/blob/main/elt/main.py"
    click Ex1 "https://github.com/jlsilva01/projeto-elt-satc/blob/main/examples/elt_sql_1_tabela.py"
    click Ex2 "https://github.com/jlsilva01/projeto-elt-satc/blob/main/examples/elt_sql_n_tabelas.py"
    click Ex3 "https://github.com/jlsilva01/projeto-elt-satc/blob/main/examples/elt_mongodb_n_collections.py"
    click Terraform "https://github.com/jlsilva01/projeto-elt-satc/tree/main/iac/adls/"
    click ProjectCfg "https://github.com/jlsilva01/projeto-elt-satc/blob/main/pyproject.toml"
    click TestSQL "https://github.com/jlsilva01/projeto-elt-satc/blob/main/test/test_connection_sqlserver.py"
    click TestADLS "https://github.com/jlsilva01/projeto-elt-satc/blob/main/test/test_connection_adls.py"
    click TestMongo "https://github.com/jlsilva01/projeto-elt-satc/blob/main/test/test_connection_mongodb.py"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef app fill:#98FB98,stroke:#333,stroke-width:1px
    classDef infra fill:#FFA500,stroke:#333,stroke-width:1px
    classDef config fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef test fill:#EEE8AA,stroke:#333,stroke-width:1px
```
