# likhitasingireddyvaluecreed/pokemon_pipeline architecture

> Architecture diagram of the GitHub repository likhitasingireddyvaluecreed/pokemon_pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/likhitasingireddyvaluecreed/pokemon_pipeline
- Repository: https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline

## Overview

This is a Pokémon data ETL application: `main.py` coordinates API extraction, raw Parquet upserts, transformation and validation, then batch loading into DuckDB. The separate retrieval utility queries processed tables for rankings and status counts, while schema generation documents the live database. The graph follows the documented flow and sampled calls; retrieval and schema generation appear to be separate entry points, and unsampled implementation details are not inferred.

Main entry points: [`main.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/main.py) (Pipeline runner), [`transform_runner.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/transform_runner.py) (Transform runner), [`extract.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/extract.py) (API extraction), [`schema/retrieve.py`](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/schema/retrieve.py) (SQL retrieval)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Extraction and raw data"]
  node_runner["Pipeline runner<br/>[main.py]"]
  node_extract["API extraction<br/>[extract.py]"]
  node_raw[("Raw Parquet<br/>[load_raw.py]")]
end

subgraph group_process["Transform and validation"]
  node_transform_runner["Transform runner"]
  node_transform["Data transformations<br/>[transform.py]"]
  node_validation["Data validation<br/>[validation.py]"]
end

subgraph group_storage["Processed data storage"]
  node_loader["DuckDB loader<br/>[duckdb_load.py]"]
  node_dlt["DLT pipeline<br/>[duckdb_load.py]"]
  node_db[("Processed DuckDB")]
end

subgraph group_consumption["Query and schema output"]
  node_retrieve["SQL retrieval<br/>[retrieve.py]"]
  node_results["Query results"]
  node_schema_gen["Schema generator<br/>[schema.py]"]
  node_schema_docs["Schema files"]
end

node_api{{"PokeAPI"}}
node_user(("Analyst"))

node_runner -->|"calls"| node_extract
node_extract -->|"requests data"| node_api
node_extract -->|"returns records"| node_runner
node_runner -->|"upserts raw data"| node_raw
node_runner -->|"runs"| node_transform_runner
node_transform_runner -->|"reads Parquet"| node_raw
node_transform_runner -->|"transforms"| node_transform
node_transform_runner -->|"validates"| node_validation
node_runner -->|"loads dataframes"| node_loader
node_loader -->|"runs resource"| node_dlt
node_dlt -->|"merges tables"| node_db
node_user -->|"runs queries"| node_retrieve
node_retrieve -->|"queries"| node_db
node_retrieve -->|"writes report"| node_results
node_user -->|"generates schema"| node_schema_gen
node_schema_gen -->|"inspects schema"| node_db
node_schema_gen -->|"writes files"| node_schema_docs

click node_runner "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/main.py"
click node_extract "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/extract.py"
click node_raw "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/load_raw.py"
click node_transform_runner "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/transform_runner.py"
click node_transform "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/transform.py"
click node_validation "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/validation.py"
click node_loader "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/duckdb_load.py"
click node_dlt "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/duckdb_load.py"
click node_retrieve "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/schema/retrieve.py"
click node_schema_gen "https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/schema/schema.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_extract,node_raw toneBlue
class node_transform_runner,node_transform,node_validation toneAmber
class node_loader,node_dlt,node_db,node_api toneMint
class node_retrieve,node_results,node_schema_gen,node_schema_docs toneRose
class node_user toneIndigo
```

## Components

### Extraction and raw data

- [Pipeline runner](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/main.py): `main.py`
- [API extraction](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/extract.py): `extract.py`
- [Raw Parquet](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/load_raw.py): `load_raw.py`

### Transform and validation

- [Transform runner](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/transform_runner.py): `transform_runner.py`
- [Data transformations](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/transform.py): `transform.py`
- [Data validation](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/validation.py): `validation.py`

### Processed data storage

- [DuckDB loader](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/duckdb_load.py): `duckdb_load.py`
- [DLT pipeline](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/duckdb_load.py): `duckdb_load.py`
- **Processed DuckDB**

### Query and schema output

- [SQL retrieval](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/schema/retrieve.py): `schema/retrieve.py`
- **Query results**
- [Schema generator](https://github.com/likhitasingireddyvaluecreed/pokemon_pipeline/blob/main/schema/schema.py): `schema/schema.py`
- **Schema files**

### Other components

- **PokeAPI**
- **Analyst**

## Connections

- Pipeline runner → API extraction: calls
- API extraction → PokeAPI: requests data
- API extraction → Pipeline runner: returns records
- Pipeline runner → Raw Parquet: upserts raw data
- Pipeline runner → Transform runner: runs
- Transform runner → Raw Parquet: reads Parquet
- Transform runner → Data transformations: transforms
- Transform runner → Data validation: validates
- Pipeline runner → DuckDB loader: loads dataframes
- DuckDB loader → DLT pipeline: runs resource
- DLT pipeline → Processed DuckDB: merges tables
- Analyst → SQL retrieval: runs queries
- SQL retrieval → Processed DuckDB: queries
- SQL retrieval → Query results: writes report
- Analyst → Schema generator: generates schema
- Schema generator → Processed DuckDB: inspects schema
- Schema generator → Schema files: writes files