How mxavier-dev/pokeapi-etl-pipeline works
This repository is a PokéAPI ETL pipeline. The documented workflow is initiated by the pipeline runner, which fetches Pokémon, type, and ability data, saves raw snapshots, transforms the data into structured records, saves processed snapshots, and loads Pokémon and relationship data into MySQL. The README documents batching, UPSERTs, and the relational model; the sampled loader shows transaction handling and relationship inserts. Source excerpts establish the main calls, while some database details are documented rather than directly shown in the schema excerpt.
Main entry points: pipeline.py Pipeline runner · load.py Database and file loader · extract.py Endpoint extractor · transform.py Data transformer
Components 14
Orchestration
- Pipeline runner
src/pipeline.py - Pipeline logger
src/utils/logger.py
Data acquisition
- Endpoint extractor
src/extract.py - Detail fetch
src/extract.py
Data transformation
- Data transformer
src/transform.py
Persistence
- Raw snapshots
data/raw - Processed snapshots
data/processed - Database and file loader
src/load.py - MySQL connection
src/connect/connection.py - Database settings
src/connect/config.py - MySQL database
- Relational schema
db/schema.sql
Other components
- Pipeline operator
- PokéAPI
Connections 14
- Pipeline operator → Pipeline runner: runs
readme.md - Pipeline runner → Endpoint extractor: requests data
src/pipeline.py - Pipeline runner → Data transformer: transforms data
src/pipeline.py - Pipeline runner → Database and file loader: saves and loads
src/pipeline.py - Pipeline runner → Pipeline logger: logs progress
src/pipeline.py - Endpoint extractor → PokéAPI: fetches pages
src/extract.py - Data transformer → Detail fetch: fetches details
src/transform.py - Detail fetch → PokéAPI: fetches records
src/extract.py - Database and file loader → Raw snapshots: writes snapshots
src/load.py - Database and file loader → Processed snapshots: writes snapshots
src/load.py - Database and file loader → MySQL connection: opens connection
src/load.py - MySQL connection → Database settings: reads settings
src/connect/connection.py - Database and file loader → MySQL database: inserts records
readme.md - Relational schema → MySQL database: defines tables
readme.md