# hemanthvaluecreed/entertainment_etl_pipleline_2 architecture

> Architecture diagram of the GitHub repository hemanthvaluecreed/entertainment_etl_pipleline_2, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hemanthvaluecreed/entertainment_etl_pipleline_2
- Repository: https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2

## Overview

This is a batch ETL pipeline for TVMaze television-show data. Running `main.py` requests paginated API records, preserves each page as raw JSON, transforms and normalizes the records into processed CSV datasets, loads those datasets into PostgreSQL, and runs database validation queries. Transformation includes cleaning, type conversion, derived fields, and relational datasets for genres, networks, web channels, and schedule days. The excerpts show the main orchestration and core extraction, loading, and validation behavior; transformation excerpts are partial, so the graph avoids asserting unshown internal call details.

Main entry points: [`main.py`](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/main.py) (Pipeline Runner), [`extract.py`](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/extract.py) (Paginated Extractor), [`transform.py`](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/transform.py) (Transformer), [`load.py`](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/load.py) (Database Loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Pipeline Orchestration"]
  node_main["Pipeline Runner<br/>[main.py]"]
end

subgraph group_collection["Data Collection"]
  node_extract["Paginated Extractor<br/>[extract.py]"]
  node_raw["Raw JSON Pages"]
end

subgraph group_preparation["Data Preparation"]
  node_transform["Transformer<br/>[transform.py]"]
  node_processed["Processed CSVs"]
end

subgraph group_persistence["Persistence and Checks"]
  node_load["Database Loader<br/>[load.py]"]
  node_postgres[("PostgreSQL Database<br/>[schema.sql]")]
  node_validate["Database Validator<br/>[validate.py]"]
end

node_operator(("Pipeline Operator"))
node_tvmaze{{"TVMaze API"}}

node_operator -->|"runs"| node_main
node_main -->|"calls"| node_extract
node_main -->|"calls"| node_transform
node_main -->|"calls"| node_load
node_main -->|"calls"| node_validate
node_extract -->|"requests pages"| node_tvmaze
node_extract -->|"writes pages"| node_raw
node_transform -->|"reads pages"| node_raw
node_transform -->|"writes datasets"| node_processed
node_load -->|"reads datasets"| node_processed
node_load -->|"loads tables"| node_postgres
node_validate -->|"queries tables"| node_postgres

click node_main "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/main.py"
click node_extract "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/extract.py"
click node_raw "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/tree/main/data/raw"
click node_transform "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/transform.py"
click node_processed "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/tree/main/data/processed"
click node_load "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/load.py"
click node_postgres "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/sql/schema.sql"
click node_validate "https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/validate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main toneBlue
class node_extract,node_raw toneAmber
class node_transform,node_processed,node_tvmaze toneMint
class node_load,node_postgres,node_validate toneRose
class node_operator toneIndigo
```

## Components

### Pipeline Orchestration

- [Pipeline Runner](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/main.py): `main.py`

### Data Collection

- [Paginated Extractor](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/extract.py): `extract.py`
- [Raw JSON Pages](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/tree/main/data/raw): `data/raw`

### Data Preparation

- [Transformer](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/transform.py): `transform.py`
- [Processed CSVs](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/tree/main/data/processed): `data/processed`

### Persistence and Checks

- [Database Loader](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/load.py): `load.py`
- [PostgreSQL Database](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/sql/schema.sql): `sql/schema.sql`
- [Database Validator](https://github.com/hemanthvaluecreed/entertainment_etl_pipleline_2/blob/main/validate.py): `validate.py`

### Other components

- **Pipeline Operator**
- **TVMaze API**

## Connections

- Pipeline Operator → Pipeline Runner: runs
- Pipeline Runner → Paginated Extractor: calls
- Pipeline Runner → Transformer: calls
- Pipeline Runner → Database Loader: calls
- Pipeline Runner → Database Validator: calls
- Paginated Extractor → TVMaze API: requests pages
- Paginated Extractor → Raw JSON Pages: writes pages
- Transformer → Raw JSON Pages: reads pages
- Transformer → Processed CSVs: writes datasets
- Database Loader → Processed CSVs: reads datasets
- Database Loader → PostgreSQL Database: loads tables
- Database Validator → PostgreSQL Database: queries tables