How hemanthvaluecreed/entertainment_etl_pipleline_2 works
This is a batch ETL pipeline for TVMaze television-show data. Running main.py requests paginated API records, preserves each page as raw JSON, transforms and normalizes the records into processed CSV datasets, loads those datasets into PostgreSQL, and runs database validation queries. Transformation includes cleaning, type conversion, derived fields, and relational datasets for genres, networks, web channels, and schedule days. The excerpts show the main orchestration and core extraction, loading, and validation behavior; transformation excerpts are partial, so the graph avoids asserting unshown internal call details.
Main entry points: main.py Pipeline Runner · extract.py Paginated Extractor · transform.py Transformer · load.py Database Loader
Components 10
Pipeline Orchestration
- Pipeline Runner
main.py
Data Collection
- Paginated Extractor
extract.py - Raw JSON Pages
data/raw
Data Preparation
- Transformer
transform.py - Processed CSVs
data/processed
Persistence and Checks
- Database Loader
load.py - PostgreSQL Database
sql/schema.sql - Database Validator
validate.py
Other components
- Pipeline Operator
- TVMaze API
Connections 12
- Pipeline Operator → Pipeline Runner: runs
- Pipeline Runner → Paginated Extractor: calls
- Pipeline Runner → Transformer: calls
- Pipeline Runner → Database Loader: calls
- Pipeline Runner → Database Validator: calls
- Paginated Extractor → TVMaze API: requests pages
- Paginated Extractor → Raw JSON Pages: writes pages
- Transformer → Raw JSON Pages: reads pages
- Transformer → Processed CSVs: writes datasets
- Database Loader → Processed CSVs: reads datasets
- Database Loader → PostgreSQL Database: loads tables
- Database Validator → PostgreSQL Database: queries tables