How jlsilva01/projeto-elt-satc works
Project Type and Purpose
• This is an open-source ELT tool written in Python 3.12.
• It connects to source databases (SQL Server, MongoDB) and writes data into Azure Data Lake Storage (ADLS).
• It uses Terraform (in iac/adls) to provision ADLS resources.
Main Components
A. Source Systems
• SQL Server (relational DB)
• (Optional) MongoDB
B. ELT Application (Python modules under elt/)
• database/sql_server_service.py – SQL Server connector & extract logic
• azure_integration/adls_service.py – ADLS upload logic
• main.py – orchestration (instantiates services, loops through tables/collections)
• Example scripts at project root & examples/ for one-off runs
C. Configuration & Secrets
• .env file (loads with python-dotenv) – connection strings, SAS tokens
• pyproject.toml & uv.lock – dependency management via UV
D. Infrastructure as Code
• Terraform modules in iac/adls/ – defines ADLS account, file system, directories
E. CI / Testing
• Tests under test/ verify connectivity to SQL Server, ADLS, MongoDB
Relationships & Data Flow
1. Terraform provisions ADLS
→ (ADLS account, file system ready)
2. ELT Application reads .env → authenticates to SQL Server and/or MongoDB
3. SQL Server Service queries tables → pulls data (Pandas, SQLAlchemy, pyodbc)
4. ADLS Service writes Parquet/CSV files to Data Lake via Azure SDK (azure-storage-file-datalake)
5. (Optional) MongoDB Service could read & transform collections → load to ADLS as well
6. Tests run as smoke connectivity checks
Architectural Patterns & Design Principles
• Layered/Modular: separate “source connectors” (in database/), “sink connectors” (in azure_integration/), and orchestration (main.py).
• Adapter/Plugin: to add a new source or destination, simply drop a new service module into the appropriate folder and register it in main.py.
• Infrastructure-as-Code for reproducible resource provisioning.
• Environment-based configuration (12-factor app style).
Essential Elements
– Clear component labels and descriptions
· Data Sources
· Application Services
· Infrastructure
· Configuration