# lucasilvap/etl_duckdb architecture

> Architecture diagram of the GitHub repository lucasilvap/etl_duckdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lucasilvap/etl_duckdb
- Repository: https://github.com/lucasilvap/etl_duckdb

## Overview

##### Project Overview
- This project is clearly an ETL (Extract, Transform, Load) tool that focuses on processing sales data from CSV files and likely loading the results into a DuckDB database.
- The project involves data extraction from CSV files (located in the pasta_gdown directory), processing via Python ETL pipelines (pipeline_00.py and pipeline_01.py), with configuration and dependency management handled via pyproject.toml.

##### Main Components
a. Data Sources:
- CSV Files: Files like vendas_05_01_2024.csv and vendas_06_01_2024.csv stored under the pasta_gdown directory.
b. ETL Pipelines:
- pipeline_00.py and pipeline_01.py: These scripts encapsulate the extraction, transformation, and possibly the loading logic. They might be split into different stages or represent alternate approaches.
c. Database / Data Storage:
- Although not explicitly shown as a file, the project title “ETL_DUCKDB” implies that the project uses DuckDB as the destination or analytical database.
d. Configuration/Dependency Management:
- pyproject.toml: Used to manage project dependencies and build configuration.

##### Relationships and Data Flow
- The CSV data files serve as the raw input.
- The ETL pipelines are responsible for reading these CSV files, performing any necessary transformations, and then loading the data into DuckDB.
- The flow can be illustrated as: CSV Files (Data Source) → ETL Pipelines (Processing/Transformation) → DuckDB (Data storage/Query engine).

##### Architectural Patterns / Design Principles
- The design separates concerns between data extraction (reading CSVs), data transformation (within pipeline scripts), and data loading (integrating with DuckDB).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Source"
        CSV["CSV Data Source (pasta_gdown)"]:::csvNode
    end

    subgraph "ETL Pipelines"
        P00["Pipeline: pipeline_00.py"]:::pipeline
        P01["Pipeline: pipeline_01.py"]:::pipeline
    end

    subgraph "Data Storage"
        DUCK["DuckDB Database"]:::db
    end

    CONFIG["Configuration (pyproject.toml)"]:::config

    CSV -->|"extracts"| P00
    CSV -->|"extracts"| P01
    P00 -->|"loads"| DUCK
    P01 -->|"loads"| DUCK
    CONFIG -->|"configures"| P00
    CONFIG -->|"configures"| P01

    click CSV "https://github.com/lucasilvap/etl_duckdb/tree/main/pasta_gdown/"
    click P00 "https://github.com/lucasilvap/etl_duckdb/blob/main/pipeline_00.py"
    click P01 "https://github.com/lucasilvap/etl_duckdb/blob/main/pipeline_01.py"
    click CONFIG "https://github.com/lucasilvap/etl_duckdb/blob/main/pyproject.toml"

    classDef csvNode fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef config fill:#ffcc99,stroke:#333,stroke-width:2px;
    classDef db fill:#ff9999,stroke:#333,stroke-width:2px;
```
