# pabeli/python-etl architecture

> Architecture diagram of the GitHub repository pabeli/python-etl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pabeli/python-etl
- Repository: https://github.com/pabeli/python-etl

## Overview

This repository is a small Spotify listening-history ETL application. The documented workflow starts at `main.py`: extract yesterday’s Spotify data, validate it, then load the needed records into a SQL database. `utils/extract.py` and `utils/validation.py` are documented capabilities but their implementations were not sampled, so their internal behavior and exact call wiring are not verified. The load implementation uses SQLAlchemy and writes a played-tracks table; the README documents the overall stages and destination.

Main entry points: [`main.py`](https://github.com/pabeli/python-etl/blob/master/main.py) (ETL entry point), [`utils/extract.py`](https://github.com/pabeli/python-etl/blob/master/utils/extract.py) (Spotify extraction), [`utils/load.py`](https://github.com/pabeli/python-etl/blob/master/utils/load.py) (Track loader), [`utils/validation.py`](https://github.com/pabeli/python-etl/blob/master/utils/validation.py) (Data validation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_etl["ETL workflow"]
  node_main["ETL entry point<br/>[main.py]"]
  node_extract["Spotify extraction<br/>[extract.py]"]
  node_validation["Data validation<br/>[validation.py]"]
  node_load["Track loader<br/>[load.py]"]
end

subgraph group_integrations["External systems"]
  node_spotify["Spotify API"]
  node_database[("SQL database<br/>[load.py]")]
end

node_user(("Operator"))

node_user -->|"runs"| node_main
node_main -->|"starts extraction"| node_extract
node_main -->|"checks data"| node_validation
node_main -->|"loads records"| node_load
node_extract -->|"downloads data"| node_spotify
node_load -->|"writes tracks"| node_database

click node_main "https://github.com/pabeli/python-etl/blob/master/main.py"
click node_extract "https://github.com/pabeli/python-etl/blob/master/utils/extract.py"
click node_validation "https://github.com/pabeli/python-etl/blob/master/utils/validation.py"
click node_load "https://github.com/pabeli/python-etl/blob/master/utils/load.py"
click node_database "https://github.com/pabeli/python-etl/blob/master/utils/load.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_extract,node_validation,node_load toneBlue
class node_spotify,node_database toneAmber
class node_user toneIndigo
```

## Components

### ETL workflow

- [ETL entry point](https://github.com/pabeli/python-etl/blob/master/main.py): `main.py`
- [Spotify extraction](https://github.com/pabeli/python-etl/blob/master/utils/extract.py): `utils/extract.py`
- [Data validation](https://github.com/pabeli/python-etl/blob/master/utils/validation.py): `utils/validation.py`
- [Track loader](https://github.com/pabeli/python-etl/blob/master/utils/load.py): `utils/load.py`

### External systems

- **Spotify API**
- [SQL database](https://github.com/pabeli/python-etl/blob/master/utils/load.py): `utils/load.py`

### Other components

- **Operator**

## Connections

- Operator → ETL entry point: runs (evidence: `README.md`)
- ETL entry point → Spotify extraction: starts extraction (evidence: `README.md`)
- ETL entry point → Data validation: checks data (evidence: `README.md`)
- ETL entry point → Track loader: loads records (evidence: `README.md`)
- Spotify extraction → Spotify API: downloads data (evidence: `README.md`)
- Track loader → SQL database: writes tracks (evidence: `README.md`)