# mxavier-dev/pokeapi-etl-pipeline architecture

> Architecture diagram of the GitHub repository mxavier-dev/pokeapi-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mxavier-dev/pokeapi-etl-pipeline
- Repository: https://github.com/mxavier-dev/pokeapi-etl-pipeline

## Overview

This repository is a PokéAPI ETL pipeline. The documented workflow is initiated by the pipeline runner, which fetches Pokémon, type, and ability data, saves raw snapshots, transforms the data into structured records, saves processed snapshots, and loads Pokémon and relationship data into MySQL. The README documents batching, UPSERTs, and the relational model; the sampled loader shows transaction handling and relationship inserts. Source excerpts establish the main calls, while some database details are documented rather than directly shown in the schema excerpt.

Main entry points: [`src/pipeline.py`](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/pipeline.py) (Pipeline runner), [`src/load.py`](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/load.py) (Database and file loader), [`src/extract.py`](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/extract.py) (Endpoint extractor), [`src/transform.py`](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/transform.py) (Data transformer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Orchestration"]
  node_runner["Pipeline runner<br/>[pipeline.py]"]
  node_logger["Pipeline logger<br/>[logger.py]"]
end

subgraph group_acquisition["Data acquisition"]
  node_extractor["Endpoint extractor<br/>[extract.py]"]
  node_detail_fetch["Detail fetch<br/>[extract.py]"]
end

subgraph group_transformation["Data transformation"]
  node_transformer["Data transformer<br/>[transform.py]"]
end

subgraph group_persistence["Persistence"]
  node_raw_files["Raw snapshots"]
  node_processed_files["Processed snapshots"]
  node_loader["Database and file loader<br/>[load.py]"]
  node_db_connection["MySQL connection<br/>[connection.py]"]
  node_db_config["Database settings<br/>[config.py]"]
  node_mysql[("MySQL database")]
  node_schema["Relational schema<br/>[schema.sql]"]
end

node_operator(("Pipeline operator"))
node_pokeapi(("PokéAPI"))

node_operator -->|"runs"| node_runner
node_runner -->|"requests data"| node_extractor
node_runner -->|"transforms data"| node_transformer
node_runner -->|"saves and loads"| node_loader
node_runner -->|"logs progress"| node_logger
node_extractor -->|"fetches pages"| node_pokeapi
node_transformer -->|"fetches details"| node_detail_fetch
node_detail_fetch -->|"fetches records"| node_pokeapi
node_loader -->|"writes snapshots"| node_raw_files
node_loader -->|"writes snapshots"| node_processed_files
node_loader -->|"opens connection"| node_db_connection
node_db_connection -->|"reads settings"| node_db_config
node_loader -->|"inserts records"| node_mysql
node_schema -->|"defines tables"| node_mysql

click node_runner "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/pipeline.py"
click node_logger "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/utils/logger.py"
click node_extractor "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/extract.py"
click node_detail_fetch "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/extract.py"
click node_transformer "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/transform.py"
click node_raw_files "https://github.com/mxavier-dev/pokeapi-etl-pipeline/tree/main/data/raw"
click node_processed_files "https://github.com/mxavier-dev/pokeapi-etl-pipeline/tree/main/data/processed"
click node_loader "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/load.py"
click node_db_connection "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/connect/connection.py"
click node_db_config "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/connect/config.py"
click node_schema "https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/db/schema.sql"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_logger toneBlue
class node_extractor,node_detail_fetch toneAmber
class node_transformer,node_pokeapi toneMint
class node_raw_files,node_processed_files,node_loader,node_db_connection,node_db_config,node_mysql,node_schema toneRose
class node_operator toneIndigo
```

## Components

### Orchestration

- [Pipeline runner](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/pipeline.py): `src/pipeline.py`
- [Pipeline logger](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/utils/logger.py): `src/utils/logger.py`

### Data acquisition

- [Endpoint extractor](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/extract.py): `src/extract.py`
- [Detail fetch](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/extract.py): `src/extract.py`

### Data transformation

- [Data transformer](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/transform.py): `src/transform.py`

### Persistence

- [Raw snapshots](https://github.com/mxavier-dev/pokeapi-etl-pipeline/tree/main/data/raw): `data/raw`
- [Processed snapshots](https://github.com/mxavier-dev/pokeapi-etl-pipeline/tree/main/data/processed): `data/processed`
- [Database and file loader](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/load.py): `src/load.py`
- [MySQL connection](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/connect/connection.py): `src/connect/connection.py`
- [Database settings](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/src/connect/config.py): `src/connect/config.py`
- **MySQL database**
- [Relational schema](https://github.com/mxavier-dev/pokeapi-etl-pipeline/blob/main/db/schema.sql): `db/schema.sql`

### Other components

- **Pipeline operator**
- **PokéAPI**

## Connections

- Pipeline operator → Pipeline runner: runs (evidence: `readme.md`)
- Pipeline runner → Endpoint extractor: requests data (evidence: `src/pipeline.py`)
- Pipeline runner → Data transformer: transforms data (evidence: `src/pipeline.py`)
- Pipeline runner → Database and file loader: saves and loads (evidence: `src/pipeline.py`)
- Pipeline runner → Pipeline logger: logs progress (evidence: `src/pipeline.py`)
- Endpoint extractor → PokéAPI: fetches pages (evidence: `src/extract.py`)
- Data transformer → Detail fetch: fetches details (evidence: `src/transform.py`)
- Detail fetch → PokéAPI: fetches records (evidence: `src/extract.py`)
- Database and file loader → Raw snapshots: writes snapshots (evidence: `src/load.py`)
- Database and file loader → Processed snapshots: writes snapshots (evidence: `src/load.py`)
- Database and file loader → MySQL connection: opens connection (evidence: `src/load.py`)
- MySQL connection → Database settings: reads settings (evidence: `src/connect/connection.py`)
- Database and file loader → MySQL database: inserts records (evidence: `readme.md`)
- Relational schema → MySQL database: defines tables (evidence: `readme.md`)
