How amrajhar/worldbankapiexercise works
This repository is a three-notebook Databricks World Bank population pipeline: it fetches paginated API records, loads Bronze Delta, cleans and upserts Silver data, then produces country statistics and regional rankings in Gold tables. The graph follows documented orchestration and table handoffs; excerpts do not show the complete Bronze or Gold orchestration and write/optimization details, so only documented relationships are included where necessary.
Main entry points: 04_Gold_WorldBank_Aggregate.py Regional rankings · 02_Bronze_WorldBank_Extract.py Population API client · 03_Silver_WorldBank_Transform.py Flatten and deduplicate
Components 13
Bronze ingestion
- Population API client
02_Bronze_WorldBank_Extract.py - Record loader
02_Bronze_WorldBank_Extract.py - Bronze Delta table
Silver transformation
- Flatten and deduplicate
03_Silver_WorldBank_Transform.py - Silver schema casting
03_Silver_WorldBank_Transform.py - Silver Delta table
- Schema evolution
03_Silver_WorldBank_Transform.py
Gold analytics
- Country statistics
04_Gold_WorldBank_Aggregate.py - Regional rankings
04_Gold_WorldBank_Aggregate.py - Gold Delta tables
- Gold optimization
04_Gold_WorldBank_Aggregate.py
Other components
- World Bank API
- Country-region API
Connections 13
- Population API client → World Bank API: fetches pages
02_Bronze_WorldBank_Extract.py - Record loader → Bronze Delta table: writes records
02_Bronze_WorldBank_Extract.py - Population API client → Record loader: passes records
02_Bronze_WorldBank_Extract.py - Bronze Delta table → Flatten and deduplicate: reads table
03_Silver_WorldBank_Transform.py - Flatten and deduplicate → Silver schema casting: cleans rows
03_Silver_WorldBank_Transform.py - Silver schema casting → Silver Delta table: upserts rows
03_Silver_WorldBank_Transform.py - Schema evolution → Silver Delta table: alters and merges
03_Silver_WorldBank_Transform.py - Silver Delta table → Country statistics: reads population
04_Gold_WorldBank_Aggregate.py - Silver Delta table → Regional rankings: reads population
04_Gold_WorldBank_Aggregate.py - Regional rankings → Country-region API: fetches regions
04_Gold_WorldBank_Aggregate.py - Country statistics → Gold Delta tables: writes statistics
- Regional rankings → Gold Delta tables: writes rankings
- Gold optimization → Gold Delta tables: optimizes table