# likhitasingireddyvaluecreed/pokemon_pipeline2 architecture

> Architecture diagram of the GitHub repository likhitasingireddyvaluecreed/pokemon_pipeline2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/likhitasingireddyvaluecreed/pokemon_pipeline2
- Repository: https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2

## Overview

This repository implements a Pokémon data ETL pipeline: it fetches Pokémon, species, and move records from an external API, persists raw snapshots, transforms and validates datasets, then merges processed tables into DuckDB. The main script coordinates extraction and loading; the transformation runner reads the raw Parquet files and calls the transformation and validation layers. Separate schema utilities inspect the DuckDB contents. The README is empty, and some source excerpts are partial, so uncertain wiring is omitted.

Main entry points: [`pokemon/main.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/main.py) (Pipeline entry point), [`pokemon/transform/transform_runner.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/transform/transform_runner.py) (Transformation runner), [`pokemon/pokemon.duckdb`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/pokemon.duckdb) (Processed DuckDB), [`pokemon/schema/schema.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/schema.py) (Schema generator)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_main["Pipeline entry point<br/>[main.py]"]
end

subgraph group_extraction["API extraction"]
  node_session["HTTP session<br/>[extract.py]"]
  node_pokemon_fetch["Pokémon fetch<br/>[extract.py]"]
  node_species_fetch["Species fetch<br/>[extract.py]"]
  node_move_fetch["Move fetch<br/>[extract.py]"]
end

subgraph group_processing["Transform and validate"]
  node_runner["Transformation runner"]
  node_transform["Dataset transforms<br/>[transform.py]"]
  node_validation["Data validation<br/>[validation.py]"]
end

subgraph group_persistence["Data persistence"]
  node_raw_upsert["Raw Parquet upsert<br/>[load_raw.py]"]
  node_raw_files["Raw Parquet datasets"]
  node_duck_load["DuckDB table loader<br/>[duckdb_load.py]"]
  node_duckdb[("Processed DuckDB<br/>[pokemon.duckdb]")]
end

subgraph group_inspection["Database inspection"]
  node_schema["Schema generator<br/>[schema.py]"]
  node_retrieve["Data retrieval<br/>[retrieve.py]"]
  node_schema_text["Schema text output<br/>[schema.txt]"]
end

node_operator(("Pipeline operator"))
node_api["PokéAPI"]

node_operator -->|"runs"| node_main
node_main -->|"creates"| node_session
node_main -->|"fetches records"| node_pokemon_fetch
node_main -->|"fetches species"| node_species_fetch
node_main -->|"fetches moves"| node_move_fetch
node_pokemon_fetch -->|"requests data"| node_api
node_species_fetch -->|"requests data"| node_api
node_move_fetch -->|"requests data"| node_api
node_main -->|"upserts extracts"| node_raw_upsert
node_raw_upsert -->|"writes Parquet"| node_raw_files
node_runner -->|"reads Parquet"| node_raw_files
node_main -->|"runs transforms"| node_runner
node_runner -->|"builds datasets"| node_transform
node_runner -->|"validates datasets"| node_validation
node_runner -->|"returns datasets"| node_main
node_main -->|"loads tables"| node_duck_load
node_duck_load -->|"merges tables"| node_duckdb
node_schema -->|"inspects schema"| node_duckdb
node_schema -->|"writes schema"| node_schema_text
node_retrieve -->|"queries records"| node_duckdb

click node_main "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/main.py"
click node_session "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py"
click node_pokemon_fetch "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py"
click node_species_fetch "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py"
click node_move_fetch "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py"
click node_raw_upsert "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/load/load_raw.py"
click node_raw_files "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/tree/main/pokemon/raw"
click node_runner "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/transform/transform_runner.py"
click node_transform "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/transform/transform.py"
click node_validation "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/validation/validation.py"
click node_duck_load "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/load/duckdb_load.py"
click node_duckdb "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/pokemon.duckdb"
click node_schema "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/schema.py"
click node_retrieve "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/retrieve.py"
click node_schema_text "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/schema.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main toneBlue
class node_session,node_pokemon_fetch,node_species_fetch,node_move_fetch toneAmber
class node_runner,node_transform,node_validation,node_api toneMint
class node_raw_upsert,node_raw_files,node_duck_load,node_duckdb toneRose
class node_schema,node_retrieve,node_schema_text,node_operator toneIndigo
```

## Components

### Pipeline orchestration

- [Pipeline entry point](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/main.py): `pokemon/main.py`

### API extraction

- [HTTP session](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py): `pokemon/extract/extract.py`
- [Pokémon fetch](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py): `pokemon/extract/extract.py`
- [Species fetch](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py): `pokemon/extract/extract.py`
- [Move fetch](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/extract/extract.py): `pokemon/extract/extract.py`

### Transform and validate

- [Transformation runner](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/transform/transform_runner.py): `pokemon/transform/transform_runner.py`
- [Dataset transforms](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/transform/transform.py): `pokemon/transform/transform.py`
- [Data validation](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/validation/validation.py): `pokemon/validation/validation.py`

### Data persistence

- [Raw Parquet upsert](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/load/load_raw.py): `pokemon/load/load_raw.py`
- [Raw Parquet datasets](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/tree/main/pokemon/raw): `pokemon/raw`
- [DuckDB table loader](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/load/duckdb_load.py): `pokemon/load/duckdb_load.py`
- [Processed DuckDB](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/pokemon.duckdb): `pokemon/pokemon.duckdb`

### Database inspection

- [Schema generator](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/schema.py): `pokemon/schema/schema.py`
- [Data retrieval](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/retrieve.py): `pokemon/schema/retrieve.py`
- [Schema text output](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline2/blob/main/pokemon/schema/schema.txt): `pokemon/schema/schema.txt`

### Other components

- **Pipeline operator**
- **PokéAPI**

## Connections

- Pipeline operator → Pipeline entry point: runs
- Pipeline entry point → HTTP session: creates
- Pipeline entry point → Pokémon fetch: fetches records
- Pipeline entry point → Species fetch: fetches species
- Pipeline entry point → Move fetch: fetches moves
- Pokémon fetch → PokéAPI: requests data
- Species fetch → PokéAPI: requests data
- Move fetch → PokéAPI: requests data
- Pipeline entry point → Raw Parquet upsert: upserts extracts
- Raw Parquet upsert → Raw Parquet datasets: writes Parquet
- Transformation runner → Raw Parquet datasets: reads Parquet
- Pipeline entry point → Transformation runner: runs transforms
- Transformation runner → Dataset transforms: builds datasets
- Transformation runner → Data validation: validates datasets
- Transformation runner → Pipeline entry point: returns datasets
- Pipeline entry point → DuckDB table loader: loads tables
- DuckDB table loader → Processed DuckDB: merges tables
- Schema generator → Processed DuckDB: inspects schema
- Schema generator → Schema text output: writes schema
- Data retrieval → Processed DuckDB: queries records