# aguiarandre/etl-pipelines architecture

> Architecture diagram of the GitHub repository aguiarandre/etl-pipelines, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aguiarandre/etl-pipelines
- Repository: https://github.com/aguiarandre/etl-pipelines

## Overview

This repository is a Cookiecutter template for creating a data ETL project, not a complete ETL implementation: its stage functions are placeholders. The intended workflow is to configure parameters and a database client, then run an orchestrator that prepares data and conditionally gathers, transforms, stores, and visualizes it. The README also defines raw, external, intermediate, and processed data areas. The graph shows documented orchestration and sampled calls; it does not infer data transfers between the placeholder stages.

Main entry points: [`{{ cookiecutter.repo_name }}/src/pipeline.py`](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/pipeline.py) (Pipeline runner), [`{{ cookiecutter.repo_name }}/src/client.py`](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/client.py) (Database client), [`{{ cookiecutter.repo_name }}/src/params.py`](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/params.py) (Parameters), [`{{ cookiecutter.repo_name }}/src/nodes/data_preparation.py`](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_preparation.py) (Data preparation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_setup["Runtime setup"]
  node_params["Parameters<br/>[params.py]"]
  node_client["Database client<br/>[client.py]"]
end

subgraph group_workflow["ETL workflow"]
  node_pipeline["Pipeline runner<br/>[pipeline.py]"]
  node_preparation["Data preparation"]
  node_gathering["Data gathering<br/>[data_gathering.py]"]
  node_transform["Data transform<br/>[data_transform.py]"]
  node_storage["Data storage<br/>[data_storage.py]"]
  node_visualization["Data visualization<br/>[data_viz.py]"]
end

subgraph group_data["Data layout"]
  node_raw[("Raw data<br/>[raw]")]
  node_external[("External data<br/>[external]")]
  node_intermediate[("Intermediate data<br/>[intermediate]")]
  node_processed[("Processed data<br/>[processed]")]
end

node_operator(("Project operator"))

node_operator -->|"runs"| node_pipeline
node_pipeline -->|"creates"| node_params
node_pipeline -->|"creates"| node_client
node_client -->|"reads settings"| node_params
node_pipeline -->|"runs"| node_preparation
node_pipeline -->|"checks and updates"| node_gathering
node_pipeline -->|"checks and updates"| node_transform
node_pipeline -->|"checks and updates"| node_storage
node_pipeline -->|"checks and updates"| node_visualization

click node_params "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/params.py"
click node_client "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/client.py"
click node_pipeline "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/pipeline.py"
click node_preparation "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_preparation.py"
click node_gathering "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_gathering.py"
click node_transform "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_transform.py"
click node_storage "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_storage.py"
click node_visualization "https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_viz.py"
click node_raw "https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/raw"
click node_external "https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/external"
click node_intermediate "https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/intermediate"
click node_processed "https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/processed"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_params,node_client toneBlue
class node_pipeline,node_preparation,node_gathering,node_transform,node_storage,node_visualization toneAmber
class node_raw,node_external,node_intermediate,node_processed toneMint
class node_operator toneIndigo
```

## Components

### Runtime setup

- [Parameters](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/params.py): `{{ cookiecutter.repo_name }}/src/params.py`
- [Database client](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/client.py): `{{ cookiecutter.repo_name }}/src/client.py`

### ETL workflow

- [Pipeline runner](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/pipeline.py): `{{ cookiecutter.repo_name }}/src/pipeline.py`
- [Data preparation](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_preparation.py): `{{ cookiecutter.repo_name }}/src/nodes/data_preparation.py`
- [Data gathering](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_gathering.py): `{{ cookiecutter.repo_name }}/src/nodes/data_gathering.py`
- [Data transform](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_transform.py): `{{ cookiecutter.repo_name }}/src/nodes/data_transform.py`
- [Data storage](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_storage.py): `{{ cookiecutter.repo_name }}/src/nodes/data_storage.py`
- [Data visualization](https://github.com/aguiarandre/etl-pipelines/blob/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/src/nodes/data_viz.py): `{{ cookiecutter.repo_name }}/src/nodes/data_viz.py`

### Data layout

- [Raw data](https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/raw): `{{ cookiecutter.repo_name }}/data/raw`
- [External data](https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/external): `{{ cookiecutter.repo_name }}/data/external`
- [Intermediate data](https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/intermediate): `{{ cookiecutter.repo_name }}/data/intermediate`
- [Processed data](https://github.com/aguiarandre/etl-pipelines/tree/master/%7B%7B%20cookiecutter.repo_name%20%7D%7D/data/processed): `{{ cookiecutter.repo_name }}/data/processed`

### Other components

- **Project operator**

## Connections

- Project operator → Pipeline runner: runs (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Pipeline runner → Parameters: creates (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Pipeline runner → Database client: creates (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Database client → Parameters: reads settings (evidence: `{{ cookiecutter.repo_name }}/src/client.py`)
- Pipeline runner → Data preparation: runs (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Pipeline runner → Data gathering: checks and updates (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Pipeline runner → Data transform: checks and updates (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Pipeline runner → Data storage: checks and updates (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)
- Pipeline runner → Data visualization: checks and updates (evidence: `{{ cookiecutter.repo_name }}/src/pipeline.py`)