# amrajhar/worldbankapiexercise architecture

> Architecture diagram of the GitHub repository amrajhar/worldbankapiexercise, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amrajhar/worldbankapiexercise
- Repository: https://github.com/amrajhar/worldbankapiexercise

## Overview

This repository is a three-notebook Databricks World Bank population pipeline: it fetches paginated API records, loads Bronze Delta, cleans and upserts Silver data, then produces country statistics and regional rankings in Gold tables. The graph follows documented orchestration and table handoffs; excerpts do not show the complete Bronze or Gold orchestration and write/optimization details, so only documented relationships are included where necessary.

Main entry points: [`04_Gold_WorldBank_Aggregate.py`](https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py) (Regional rankings), [`02_Bronze_WorldBank_Extract.py`](https://github.com/amrajhar/worldbankapiexercise/blob/main/02_Bronze_WorldBank_Extract.py) (Population API client), [`03_Silver_WorldBank_Transform.py`](https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py) (Flatten and deduplicate)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Bronze ingestion"]
  node_api_client["Population API client"]
  node_record_loader["Record loader"]
  node_bronze_table[("Bronze Delta table")]
end

subgraph group_transform["Silver transformation"]
  node_flatten_clean["Flatten and deduplicate"]
  node_schema["Silver schema casting"]
  node_silver_table[("Silver Delta table")]
  node_schema_evolution["Schema evolution"]
end

subgraph group_analytics["Gold analytics"]
  node_country_stats["Country statistics"]
  node_regional_rankings["Regional rankings"]
  node_gold_tables[("Gold Delta tables")]
  node_gold_optimization["Gold optimization"]
end

node_worldbank(("World Bank API"))
node_region_api(("Country-region API"))

node_api_client -->|"fetches pages"| node_worldbank
node_record_loader -->|"writes records"| node_bronze_table
node_api_client -.->|"passes records"| node_record_loader
node_bronze_table -->|"reads table"| node_flatten_clean
node_flatten_clean -->|"cleans rows"| node_schema
node_schema -->|"upserts rows"| node_silver_table
node_schema_evolution -->|"alters and merges"| node_silver_table
node_silver_table -->|"reads population"| node_country_stats
node_silver_table -->|"reads population"| node_regional_rankings
node_regional_rankings -->|"fetches regions"| node_region_api
node_country_stats -.->|"writes statistics"| node_gold_tables
node_regional_rankings -.->|"writes rankings"| node_gold_tables
node_gold_optimization -.->|"optimizes table"| node_gold_tables

click node_api_client "https://github.com/amrajhar/worldbankapiexercise/blob/main/02_Bronze_WorldBank_Extract.py"
click node_record_loader "https://github.com/amrajhar/worldbankapiexercise/blob/main/02_Bronze_WorldBank_Extract.py"
click node_flatten_clean "https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py"
click node_schema "https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py"
click node_schema_evolution "https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py"
click node_country_stats "https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py"
click node_regional_rankings "https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py"
click node_gold_optimization "https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_client,node_record_loader,node_bronze_table toneBlue
class node_flatten_clean,node_schema,node_silver_table,node_schema_evolution toneAmber
class node_country_stats,node_regional_rankings,node_gold_tables,node_gold_optimization,node_worldbank,node_region_api toneMint
```

## Components

### Bronze ingestion

- [Population API client](https://github.com/amrajhar/worldbankapiexercise/blob/main/02_Bronze_WorldBank_Extract.py): `02_Bronze_WorldBank_Extract.py`
- [Record loader](https://github.com/amrajhar/worldbankapiexercise/blob/main/02_Bronze_WorldBank_Extract.py): `02_Bronze_WorldBank_Extract.py`
- **Bronze Delta table**

### Silver transformation

- [Flatten and deduplicate](https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py): `03_Silver_WorldBank_Transform.py`
- [Silver schema casting](https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py): `03_Silver_WorldBank_Transform.py`
- **Silver Delta table**
- [Schema evolution](https://github.com/amrajhar/worldbankapiexercise/blob/main/03_Silver_WorldBank_Transform.py): `03_Silver_WorldBank_Transform.py`

### Gold analytics

- [Country statistics](https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py): `04_Gold_WorldBank_Aggregate.py`
- [Regional rankings](https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py): `04_Gold_WorldBank_Aggregate.py`
- **Gold Delta tables**
- [Gold optimization](https://github.com/amrajhar/worldbankapiexercise/blob/main/04_Gold_WorldBank_Aggregate.py): `04_Gold_WorldBank_Aggregate.py`

### Other components

- **World Bank API**
- **Country-region API**

## Connections

- Population API client → World Bank API: fetches pages (evidence: `02_Bronze_WorldBank_Extract.py`)
- Record loader → Bronze Delta table: writes records (evidence: `02_Bronze_WorldBank_Extract.py`)
- Population API client → Record loader: passes records (evidence: `02_Bronze_WorldBank_Extract.py`)
- Bronze Delta table → Flatten and deduplicate: reads table (evidence: `03_Silver_WorldBank_Transform.py`)
- Flatten and deduplicate → Silver schema casting: cleans rows (evidence: `03_Silver_WorldBank_Transform.py`)
- Silver schema casting → Silver Delta table: upserts rows (evidence: `03_Silver_WorldBank_Transform.py`)
- Schema evolution → Silver Delta table: alters and merges (evidence: `03_Silver_WorldBank_Transform.py`)
- Silver Delta table → Country statistics: reads population (evidence: `04_Gold_WorldBank_Aggregate.py`)
- Silver Delta table → Regional rankings: reads population (evidence: `04_Gold_WorldBank_Aggregate.py`)
- Regional rankings → Country-region API: fetches regions (evidence: `04_Gold_WorldBank_Aggregate.py`)
- Country statistics → Gold Delta tables: writes statistics
- Regional rankings → Gold Delta tables: writes rankings
- Gold optimization → Gold Delta tables: optimizes table
