How fie21/cambodia-trade-etl-pipeline works
This repository is a Cambodia GDCE trade-and-transport data warehouse and analytics platform. Its principal workflow schedules monthly extraction across four statistical pillars, normalizes snapshots, loads raw audit payloads and analytical facts into PostgreSQL, then serves dashboard analytics and forecasts. The graph retains the four pillar-specific scrapers and the dashboards, while grouping shared transformation and loading logic. Some warehouse and dashboard connections are based on README descriptions; unsampled implementation details are not inferred.
Main entry points: transport_stats_dag.py Monthly DAG · raw Raw snapshots · transport.py Transport scraper · country.py Country scraper
Components 16
Scheduling
- Monthly DAG
dags/transport_stats_dag.py
GDCE Extraction
- Transport scraper
src/scrapers/transport.py - Country scraper
src/scrapers/country.py - SITC scraper
src/scrapers/sitc.py - HS scraper
src/scrapers/hs.py - HTTP session
src/scrapers/common.py
Data Preparation
- Raw snapshots
data/raw - Normalize facts
src/transform.py - Processed CSVs
data/processed
Warehouse
- Warehouse loader
src/load.py - PostgreSQL warehouse
sql/init_warehouse.sql
Analytics
- Business dashboard
dashboard/app_business.py - Engineering dashboard
dashboard/app_data_engineering.py - Forecast models
src/forecast.py - Dashboard datasets
dashboard/data
Other components
- GDCE API
Connections 28
- Monthly DAG → Transport scraper: invokes scrape
dags/transport_stats_dag.py - Monthly DAG → Country scraper: invokes scrape
dags/transport_stats_dag.py - Monthly DAG → SITC scraper: invokes scrape
dags/transport_stats_dag.py - Monthly DAG → HS scraper: invokes scrape
dags/transport_stats_dag.py - Transport scraper → GDCE API: requests data
src/scrapers/transport.py - Country scraper → GDCE API: requests data
src/scrapers/country.py - SITC scraper → GDCE API: requests data
src/scrapers/sitc.py - HS scraper → GDCE API: requests data
src/scrapers/hs.py - Transport scraper → HTTP session: creates session
src/scrapers/transport.py - Country scraper → HTTP session: creates session
src/scrapers/country.py - SITC scraper → HTTP session: creates session
src/scrapers/sitc.py - HS scraper → HTTP session: creates session
src/scrapers/hs.py - Transport scraper → Raw snapshots: writes snapshot
src/scrapers/transport.py - Country scraper → Raw snapshots: writes snapshot
src/scrapers/country.py - SITC scraper → Raw snapshots: writes snapshot
src/scrapers/sitc.py - HS scraper → Raw snapshots: writes snapshot
src/scrapers/hs.py - Monthly DAG → Normalize facts: invokes transforms
dags/transport_stats_dag.py - Normalize facts → Raw snapshots: reads JSON
src/transform.py - Normalize facts → Processed CSVs: writes CSV
src/transform.py - Monthly DAG → Warehouse loader: invokes loaders
dags/transport_stats_dag.py - Warehouse loader → Raw snapshots: stages payloads
src/load.py - Warehouse loader → Processed CSVs: loads CSVs
src/load.py - Warehouse loader → PostgreSQL warehouse: upserts facts
src/load.py - Business dashboard → Forecast models: uses models
dashboard/app_business.py - PostgreSQL warehouse → Business dashboard: serves analytics
README.md - PostgreSQL warehouse → Engineering dashboard: serves monitoring
README.md - Business dashboard → Dashboard datasets: uses datasets
README.md - Business dashboard → Engineering dashboard: offers dashboards
README.md