How likhitasingireddyvaluecreed/pokemon-etl-pipeline works
This repository is a Pokémon API extraction and SQLite ETL pipeline. Its main workflow is orchestrated by main.py: fetch Pokémon, species, and move data; land raw records in Parquet; transform and validate DataFrames; load relational tables; then run SQL analyses and generate schema reports. The README describes DuckDB and DLT, but the sampled implementation uses SQLite; the graph reflects the implementation and treats the README’s broader capabilities as documented only where their relationship is explicit. Some links are omitted because source excerpts are incomplete.
Main entry points: main.py Pipeline runner · transform_runner.py Transform workflow · pokemon.db SQLite database · retrieve.py SQL analysis
Components 18
Pipeline orchestration
- Pipeline runner
main.py - Logging setup
logger.py - Pipeline configuration
config.py
Data ingestion
- PokeAPI
- API extraction
extract.py - Raw Parquet
load_raw.py - Raw dataset files
raw
Data processing
- Transform workflow
transform_runner.py - Pokémon transforms
transform.py - Species and move transforms
transform.py - Data validation
validation.py
Database and analysis
- Relational loading
load.py - SQLite database
pokemon.db - SQL analysis
schema/retrieve.py - Query output
retrieve.txt - Schema reporting
schema/schema.py - Schema reports
schema.txt
Other components
- Pipeline operator
Connections 18
- Pipeline operator → Pipeline runner: runs
README.md - Pipeline runner → Logging setup: configures
main.py - Pipeline runner → API extraction: invokes
main.py - API extraction → PokeAPI: requests data
extract.py - Pipeline runner → Raw Parquet: upserts records
main.py - Raw Parquet → Raw dataset files: writes Parquet
load_raw.py - Pipeline runner → Transform workflow: runs transforms
main.py - Transform workflow → Raw dataset files: reads Parquet
transform_runner.py - Transform workflow → Pokémon transforms: calls transforms
transform_runner.py - Transform workflow → Species and move transforms: calls transforms
transform_runner.py - Transform workflow → Data validation: validates data
transform_runner.py - Pipeline runner → Relational loading: loads datasets
main.py - Relational loading → SQLite database: writes tables
load.py - SQL analysis → SQLite database: queries
schema/retrieve.py - SQL analysis → Query output: saves results
schema/retrieve.py - Schema reporting → SQLite database: inspects schema
schema/schema.py - Schema reporting → Schema reports: writes reports
schema/schema.py - Pipeline runner → Pipeline configuration: uses settings
main.py