# likhitasingireddyvaluecreed/pokemon-etl-pipeline architecture

> Architecture diagram of the GitHub repository likhitasingireddyvaluecreed/pokemon-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline
- Repository: https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline

## Overview

This repository is a Pokémon API extraction and SQLite ETL pipeline. Its main workflow is orchestrated by main.py: fetch Pokémon, species, and move data; land raw records in Parquet; transform and validate DataFrames; load relational tables; then run SQL analyses and generate schema reports. The README describes DuckDB and DLT, but the sampled implementation uses SQLite; the graph reflects the implementation and treats the README’s broader capabilities as documented only where their relationship is explicit. Some links are omitted because source excerpts are incomplete.

Main entry points: [`main.py`](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/main.py) (Pipeline runner), [`transform_runner.py`](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform_runner.py) (Transform workflow), [`pokemon.db`](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/pokemon.db) (SQLite database), [`schema/retrieve.py`](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema/retrieve.py) (SQL analysis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline orchestration"]
  node_main["Pipeline runner<br/>[main.py]"]
  node_logging["Logging setup<br/>[logger.py]"]
  node_config["Pipeline configuration<br/>[config.py]"]
end

subgraph group_ingestion["Data ingestion"]
  node_pokeapi{{"PokeAPI"}}
  node_extract["API extraction<br/>[extract.py]"]
  node_rawfiles["Raw Parquet<br/>[load_raw.py]"]
  node_rawdata[("Raw dataset files")]
end

subgraph group_processing["Data processing"]
  node_transform_runner["Transform workflow"]
  node_pokemon_transform["Pokémon transforms<br/>[transform.py]"]
  node_species_moves["Species and move transforms<br/>[transform.py]"]
  node_validation["Data validation<br/>[validation.py]"]
end

subgraph group_database["Database and analysis"]
  node_loader["Relational loading<br/>[load.py]"]
  node_sqlite[("SQLite database<br/>[pokemon.db]")]
  node_retrieval["SQL analysis<br/>[retrieve.py]"]
  node_report["Query output<br/>[retrieve.txt]"]
  node_schema_generation["Schema reporting<br/>[schema.py]"]
  node_schema_files["Schema reports<br/>[schema.txt]"]
end

node_operator(("Pipeline operator"))

node_operator -->|"runs"| node_main
node_main -->|"configures"| node_logging
node_main -->|"invokes"| node_extract
node_extract -->|"requests data"| node_pokeapi
node_main -->|"upserts records"| node_rawfiles
node_rawfiles -->|"writes Parquet"| node_rawdata
node_main -->|"runs transforms"| node_transform_runner
node_transform_runner -->|"reads Parquet"| node_rawdata
node_transform_runner -->|"calls transforms"| node_pokemon_transform
node_transform_runner -->|"calls transforms"| node_species_moves
node_transform_runner -->|"validates data"| node_validation
node_main -->|"loads datasets"| node_loader
node_loader -->|"writes tables"| node_sqlite
node_retrieval -->|"queries"| node_sqlite
node_retrieval -->|"saves results"| node_report
node_schema_generation -->|"inspects schema"| node_sqlite
node_schema_generation -->|"writes reports"| node_schema_files
node_main -->|"uses settings"| node_config

click node_main "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/main.py"
click node_logging "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/logger.py"
click node_config "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/config.py"
click node_extract "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/extract.py"
click node_rawfiles "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/load_raw.py"
click node_rawdata "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/tree/main/raw"
click node_transform_runner "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform_runner.py"
click node_pokemon_transform "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform.py"
click node_species_moves "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform.py"
click node_validation "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/validation.py"
click node_loader "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/load.py"
click node_sqlite "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/pokemon.db"
click node_retrieval "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema/retrieve.py"
click node_report "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/retrieve.txt"
click node_schema_generation "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema/schema.py"
click node_schema_files "https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema.txt"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_logging,node_config toneBlue
class node_pokeapi,node_extract,node_rawfiles,node_rawdata toneAmber
class node_transform_runner,node_pokemon_transform,node_species_moves,node_validation toneMint
class node_loader,node_sqlite,node_retrieval,node_report,node_schema_generation,node_schema_files toneRose
class node_operator toneIndigo
```

## Components

### Pipeline orchestration

- [Pipeline runner](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/main.py): `main.py`
- [Logging setup](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/logger.py): `logger.py`
- [Pipeline configuration](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/config.py): `config.py`

### Data ingestion

- **PokeAPI**
- [API extraction](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/extract.py): `extract.py`
- [Raw Parquet](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/load_raw.py): `load_raw.py`
- [Raw dataset files](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/tree/main/raw): `raw`

### Data processing

- [Transform workflow](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform_runner.py): `transform_runner.py`
- [Pokémon transforms](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform.py): `transform.py`
- [Species and move transforms](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/transform.py): `transform.py`
- [Data validation](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/validation.py): `validation.py`

### Database and analysis

- [Relational loading](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/load.py): `load.py`
- [SQLite database](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/pokemon.db): `pokemon.db`
- [SQL analysis](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema/retrieve.py): `schema/retrieve.py`
- [Query output](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/retrieve.txt): `retrieve.txt`
- [Schema reporting](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema/schema.py): `schema/schema.py`
- [Schema reports](https://github.com/likhitasingireddyvaluecreed/pokemon-etl-pipeline/blob/main/schema.txt): `schema.txt`

### Other components

- **Pipeline operator**

## Connections

- Pipeline operator → Pipeline runner: runs (evidence: `README.md`)
- Pipeline runner → Logging setup: configures (evidence: `main.py`)
- Pipeline runner → API extraction: invokes (evidence: `main.py`)
- API extraction → PokeAPI: requests data (evidence: `extract.py`)
- Pipeline runner → Raw Parquet: upserts records (evidence: `main.py`)
- Raw Parquet → Raw dataset files: writes Parquet (evidence: `load_raw.py`)
- Pipeline runner → Transform workflow: runs transforms (evidence: `main.py`)
- Transform workflow → Raw dataset files: reads Parquet (evidence: `transform_runner.py`)
- Transform workflow → Pokémon transforms: calls transforms (evidence: `transform_runner.py`)
- Transform workflow → Species and move transforms: calls transforms (evidence: `transform_runner.py`)
- Transform workflow → Data validation: validates data (evidence: `transform_runner.py`)
- Pipeline runner → Relational loading: loads datasets (evidence: `main.py`)
- Relational loading → SQLite database: writes tables (evidence: `load.py`)
- SQL analysis → SQLite database: queries (evidence: `schema/retrieve.py`)
- SQL analysis → Query output: saves results (evidence: `schema/retrieve.py`)
- Schema reporting → SQLite database: inspects schema (evidence: `schema/schema.py`)
- Schema reporting → Schema reports: writes reports (evidence: `schema/schema.py`)
- Pipeline runner → Pipeline configuration: uses settings (evidence: `main.py`)
