How likhitasingireddyvaluecreed/pokemon_pipeline works
This is a Pokémon data ETL application: main.py coordinates API extraction, raw Parquet upserts, transformation and validation, then batch loading into DuckDB. The separate retrieval utility queries processed tables for rankings and status counts, while schema generation documents the live database. The graph follows the documented flow and sampled calls; retrieval and schema generation appear to be separate entry points, and unsampled implementation details are not inferred.
Main entry points: main.py Pipeline runner · transform_runner.py Transform runner · extract.py API extraction · retrieve.py SQL retrieval
Components 15
Extraction and raw data
- Pipeline runner
main.py - API extraction
extract.py - Raw Parquet
load_raw.py
Transform and validation
- Transform runner
transform_runner.py - Data transformations
transform.py - Data validation
validation.py
Processed data storage
- DuckDB loader
duckdb_load.py - DLT pipeline
duckdb_load.py - Processed DuckDB
Query and schema output
- SQL retrieval
schema/retrieve.py - Query results
- Schema generator
schema/schema.py - Schema files
Other components
- PokeAPI
- Analyst
Connections 17
- Pipeline runner → API extraction: calls
- API extraction → PokeAPI: requests data
- API extraction → Pipeline runner: returns records
- Pipeline runner → Raw Parquet: upserts raw data
- Pipeline runner → Transform runner: runs
- Transform runner → Raw Parquet: reads Parquet
- Transform runner → Data transformations: transforms
- Transform runner → Data validation: validates
- Pipeline runner → DuckDB loader: loads dataframes
- DuckDB loader → DLT pipeline: runs resource
- DLT pipeline → Processed DuckDB: merges tables
- Analyst → SQL retrieval: runs queries
- SQL retrieval → Processed DuckDB: queries
- SQL retrieval → Query results: writes report
- Analyst → Schema generator: generates schema
- Schema generator → Processed DuckDB: inspects schema
- Schema generator → Schema files: writes files