How likhitasingireddyvaluecreed/pokemon_pipeline2 works
This repository implements a Pokémon data ETL pipeline: it fetches Pokémon, species, and move records from an external API, persists raw snapshots, transforms and validates datasets, then merges processed tables into DuckDB. The main script coordinates extraction and loading; the transformation runner reads the raw Parquet files and calls the transformation and validation layers. Separate schema utilities inspect the DuckDB contents. The README is empty, and some source excerpts are partial, so uncertain wiring is omitted.
Main entry points: main.py Pipeline entry point · transform_runner.py Transformation runner · pokemon.duckdb Processed DuckDB · schema.py Schema generator
Components 17
Pipeline orchestration
- Pipeline entry point
pokemon/main.py
API extraction
- HTTP session
pokemon/extract/extract.py - Pokémon fetch
pokemon/extract/extract.py - Species fetch
pokemon/extract/extract.py - Move fetch
pokemon/extract/extract.py
Transform and validate
- Transformation runner
pokemon/transform/transform_runner.py - Dataset transforms
pokemon/transform/transform.py - Data validation
pokemon/validation/validation.py
Data persistence
- Raw Parquet upsert
pokemon/load/load_raw.py - Raw Parquet datasets
pokemon/raw - DuckDB table loader
pokemon/load/duckdb_load.py - Processed DuckDB
pokemon/pokemon.duckdb
Database inspection
- Schema generator
pokemon/schema/schema.py - Data retrieval
pokemon/schema/retrieve.py - Schema text output
pokemon/schema/schema.txt
Other components
- Pipeline operator
- PokéAPI
Connections 20
- Pipeline operator → Pipeline entry point: runs
- Pipeline entry point → HTTP session: creates
- Pipeline entry point → Pokémon fetch: fetches records
- Pipeline entry point → Species fetch: fetches species
- Pipeline entry point → Move fetch: fetches moves
- Pokémon fetch → PokéAPI: requests data
- Species fetch → PokéAPI: requests data
- Move fetch → PokéAPI: requests data
- Pipeline entry point → Raw Parquet upsert: upserts extracts
- Raw Parquet upsert → Raw Parquet datasets: writes Parquet
- Transformation runner → Raw Parquet datasets: reads Parquet
- Pipeline entry point → Transformation runner: runs transforms
- Transformation runner → Dataset transforms: builds datasets
- Transformation runner → Data validation: validates datasets
- Transformation runner → Pipeline entry point: returns datasets
- Pipeline entry point → DuckDB table loader: loads tables
- DuckDB table loader → Processed DuckDB: merges tables
- Schema generator → Processed DuckDB: inspects schema
- Schema generator → Schema text output: writes schema
- Data retrieval → Processed DuckDB: queries records