# br3nnk/etl architecture

> Architecture diagram of the GitHub repository br3nnk/etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/br3nnk/etl
- Repository: https://github.com/br3nnk/etl

## Overview

This repository is an early ETL project: its standalone script fetches user and product records from a public API and saves each record as JSON. The `src/etl` package contains paginated extraction, field transformations, settings, and SQLAlchemy models, but its `main.py` and `load.py` are empty, so the sampled code does not connect those pieces into a database-loading workflow. The graph distinguishes the working script from these package capabilities; edges reflect only visible calls or requests.

Main entry points: [`ETL.py`](https://github.com/br3nnk/etl/blob/main/ETL.py) (Endpoint Loop), [`src/etl/extract.py`](https://github.com/br3nnk/etl/blob/main/src/etl/extract.py) (Paginated Extractor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_legacy["Standalone ETL"]
  node_endpoint_loop["Endpoint Loop<br/>[ETL.py]"]
  node_legacy_extract["API Extraction<br/>[ETL.py]"]
  node_json_loader["JSON File Loader<br/>[ETL.py]"]
  node_json_files["JSON Files"]
end

subgraph group_package["ETL Package"]
  node_settings["Settings<br/>[config.py]"]
  node_api_extractor["Paginated Extractor<br/>[extract.py]"]
  node_user_transform["User Transform<br/>[transform.py]"]
  node_product_transform["Product Transform<br/>[transform.py]"]
  node_load_module["Load Module<br/>[load.py]"]
  node_main_module["Package Entry<br/>[main.py]"]
end

subgraph group_schema["Data Schema"]
  node_orm_models["ORM Models<br/>[models.py]"]
end

node_public_api["DummyJSON API"]

node_endpoint_loop -->|"calls"| node_legacy_extract
node_endpoint_loop -->|"calls"| node_json_loader
node_legacy_extract -->|"requests"| node_public_api
node_json_loader -->|"writes"| node_json_files
node_api_extractor -->|"requests"| node_public_api

click node_endpoint_loop "https://github.com/br3nnk/etl/blob/main/ETL.py"
click node_legacy_extract "https://github.com/br3nnk/etl/blob/main/ETL.py"
click node_json_loader "https://github.com/br3nnk/etl/blob/main/ETL.py"
click node_settings "https://github.com/br3nnk/etl/blob/main/src/etl/config.py"
click node_api_extractor "https://github.com/br3nnk/etl/blob/main/src/etl/extract.py"
click node_user_transform "https://github.com/br3nnk/etl/blob/main/src/etl/transform.py"
click node_product_transform "https://github.com/br3nnk/etl/blob/main/src/etl/transform.py"
click node_load_module "https://github.com/br3nnk/etl/blob/main/src/etl/load.py"
click node_main_module "https://github.com/br3nnk/etl/blob/main/src/etl/main.py"
click node_orm_models "https://github.com/br3nnk/etl/blob/main/src/etl/models.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_endpoint_loop,node_legacy_extract,node_json_loader,node_json_files toneBlue
class node_settings,node_api_extractor,node_user_transform,node_product_transform,node_load_module,node_main_module toneAmber
class node_orm_models,node_public_api toneMint
```

## Components

### Standalone ETL

- [Endpoint Loop](https://github.com/br3nnk/etl/blob/main/ETL.py): `ETL.py`
- [API Extraction](https://github.com/br3nnk/etl/blob/main/ETL.py): `ETL.py`
- [JSON File Loader](https://github.com/br3nnk/etl/blob/main/ETL.py): `ETL.py`
- **JSON Files**

### ETL Package

- [Settings](https://github.com/br3nnk/etl/blob/main/src/etl/config.py): `src/etl/config.py`
- [Paginated Extractor](https://github.com/br3nnk/etl/blob/main/src/etl/extract.py): `src/etl/extract.py`
- [User Transform](https://github.com/br3nnk/etl/blob/main/src/etl/transform.py): `src/etl/transform.py`
- [Product Transform](https://github.com/br3nnk/etl/blob/main/src/etl/transform.py): `src/etl/transform.py`
- [Load Module](https://github.com/br3nnk/etl/blob/main/src/etl/load.py): `src/etl/load.py`
- [Package Entry](https://github.com/br3nnk/etl/blob/main/src/etl/main.py): `src/etl/main.py`

### Data Schema

- [ORM Models](https://github.com/br3nnk/etl/blob/main/src/etl/models.py): `src/etl/models.py`

### Other components

- **DummyJSON API**

## Connections

- Endpoint Loop → API Extraction: calls (evidence: `ETL.py`)
- Endpoint Loop → JSON File Loader: calls (evidence: `ETL.py`)
- API Extraction → DummyJSON API: requests (evidence: `ETL.py`)
- JSON File Loader → JSON Files: writes (evidence: `ETL.py`)
- Paginated Extractor → DummyJSON API: requests (evidence: `src/etl/extract.py`)