# fie21/cambodia-trade-etl-pipeline architecture

> Architecture diagram of the GitHub repository fie21/cambodia-trade-etl-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fie21/cambodia-trade-etl-pipeline
- Repository: https://github.com/fie21/cambodia-trade-etl-pipeline

## Overview

This repository is a Cambodia GDCE trade-and-transport data warehouse and analytics platform. Its principal workflow schedules monthly extraction across four statistical pillars, normalizes snapshots, loads raw audit payloads and analytical facts into PostgreSQL, then serves dashboard analytics and forecasts. The graph retains the four pillar-specific scrapers and the dashboards, while grouping shared transformation and loading logic. Some warehouse and dashboard connections are based on README descriptions; unsampled implementation details are not inferred.

Main entry points: [`dags/transport_stats_dag.py`](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dags/transport_stats_dag.py) (Monthly DAG), [`data/raw`](https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/data/raw) (Raw snapshots), [`src/scrapers/transport.py`](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/transport.py) (Transport scraper), [`src/scrapers/country.py`](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/country.py) (Country scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Scheduling"]
  node_airflow["Monthly DAG"]
end

subgraph group_extraction["GDCE Extraction"]
  node_transport["Transport scraper<br/>[transport.py]"]
  node_country["Country scraper<br/>[country.py]"]
  node_sitc["SITC scraper<br/>[sitc.py]"]
  node_hs["HS scraper<br/>[hs.py]"]
  node_http["HTTP session<br/>[common.py]"]
end

subgraph group_data["Data Preparation"]
  node_raw["Raw snapshots"]
  node_transform["Normalize facts<br/>[transform.py]"]
  node_processed["Processed CSVs"]
end

subgraph group_warehouse["Warehouse"]
  node_loader["Warehouse loader<br/>[load.py]"]
  node_postgres[("PostgreSQL warehouse<br/>[init_warehouse.sql]")]
end

subgraph group_analytics["Analytics"]
  node_business["Business dashboard<br/>[app_business.py]"]
  node_engineering["Engineering dashboard"]
  node_forecast["Forecast models<br/>[forecast.py]"]
  node_parquet["Dashboard datasets"]
end

node_gdce{{"GDCE API"}}

node_airflow -->|"invokes scrape"| node_transport
node_airflow -->|"invokes scrape"| node_country
node_airflow -->|"invokes scrape"| node_sitc
node_airflow -->|"invokes scrape"| node_hs
node_transport -->|"requests data"| node_gdce
node_country -->|"requests data"| node_gdce
node_sitc -->|"requests data"| node_gdce
node_hs -->|"requests data"| node_gdce
node_transport -->|"creates session"| node_http
node_country -->|"creates session"| node_http
node_sitc -->|"creates session"| node_http
node_hs -->|"creates session"| node_http
node_transport -->|"writes snapshot"| node_raw
node_country -->|"writes snapshot"| node_raw
node_sitc -->|"writes snapshot"| node_raw
node_hs -->|"writes snapshot"| node_raw
node_airflow -->|"invokes transforms"| node_transform
node_transform -->|"reads JSON"| node_raw
node_transform -->|"writes CSV"| node_processed
node_airflow -->|"invokes loaders"| node_loader
node_loader -->|"stages payloads"| node_raw
node_loader -->|"loads CSVs"| node_processed
node_loader -->|"upserts facts"| node_postgres
node_business -->|"uses models"| node_forecast
node_postgres -.->|"serves analytics"| node_business
node_postgres -.->|"serves monitoring"| node_engineering
node_business -.->|"uses datasets"| node_parquet
node_business -.->|"offers dashboards"| node_engineering

click node_airflow "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dags/transport_stats_dag.py"
click node_transport "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/transport.py"
click node_country "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/country.py"
click node_sitc "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/sitc.py"
click node_hs "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/hs.py"
click node_http "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/common.py"
click node_raw "https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/data/raw"
click node_transform "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/transform.py"
click node_processed "https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/data/processed"
click node_loader "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/load.py"
click node_postgres "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/sql/init_warehouse.sql"
click node_business "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dashboard/app_business.py"
click node_engineering "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dashboard/app_data_engineering.py"
click node_forecast "https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/forecast.py"
click node_parquet "https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/dashboard/data"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_airflow toneBlue
class node_transport,node_country,node_sitc,node_hs,node_http toneAmber
class node_raw,node_transform,node_processed,node_gdce toneMint
class node_loader,node_postgres toneRose
class node_business,node_engineering,node_forecast,node_parquet toneIndigo
```

## Components

### Scheduling

- [Monthly DAG](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dags/transport_stats_dag.py): `dags/transport_stats_dag.py`

### GDCE Extraction

- [Transport scraper](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/transport.py): `src/scrapers/transport.py`
- [Country scraper](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/country.py): `src/scrapers/country.py`
- [SITC scraper](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/sitc.py): `src/scrapers/sitc.py`
- [HS scraper](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/hs.py): `src/scrapers/hs.py`
- [HTTP session](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/scrapers/common.py): `src/scrapers/common.py`

### Data Preparation

- [Raw snapshots](https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/data/raw): `data/raw`
- [Normalize facts](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/transform.py): `src/transform.py`
- [Processed CSVs](https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/data/processed): `data/processed`

### Warehouse

- [Warehouse loader](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/load.py): `src/load.py`
- [PostgreSQL warehouse](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/sql/init_warehouse.sql): `sql/init_warehouse.sql`

### Analytics

- [Business dashboard](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dashboard/app_business.py): `dashboard/app_business.py`
- [Engineering dashboard](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/dashboard/app_data_engineering.py): `dashboard/app_data_engineering.py`
- [Forecast models](https://github.com/fie21/cambodia-trade-etl-pipeline/blob/main/src/forecast.py): `src/forecast.py`
- [Dashboard datasets](https://github.com/fie21/cambodia-trade-etl-pipeline/tree/main/dashboard/data): `dashboard/data`

### Other components

- **GDCE API**

## Connections

- Monthly DAG → Transport scraper: invokes scrape (evidence: `dags/transport_stats_dag.py`)
- Monthly DAG → Country scraper: invokes scrape (evidence: `dags/transport_stats_dag.py`)
- Monthly DAG → SITC scraper: invokes scrape (evidence: `dags/transport_stats_dag.py`)
- Monthly DAG → HS scraper: invokes scrape (evidence: `dags/transport_stats_dag.py`)
- Transport scraper → GDCE API: requests data (evidence: `src/scrapers/transport.py`)
- Country scraper → GDCE API: requests data (evidence: `src/scrapers/country.py`)
- SITC scraper → GDCE API: requests data (evidence: `src/scrapers/sitc.py`)
- HS scraper → GDCE API: requests data (evidence: `src/scrapers/hs.py`)
- Transport scraper → HTTP session: creates session (evidence: `src/scrapers/transport.py`)
- Country scraper → HTTP session: creates session (evidence: `src/scrapers/country.py`)
- SITC scraper → HTTP session: creates session (evidence: `src/scrapers/sitc.py`)
- HS scraper → HTTP session: creates session (evidence: `src/scrapers/hs.py`)
- Transport scraper → Raw snapshots: writes snapshot (evidence: `src/scrapers/transport.py`)
- Country scraper → Raw snapshots: writes snapshot (evidence: `src/scrapers/country.py`)
- SITC scraper → Raw snapshots: writes snapshot (evidence: `src/scrapers/sitc.py`)
- HS scraper → Raw snapshots: writes snapshot (evidence: `src/scrapers/hs.py`)
- Monthly DAG → Normalize facts: invokes transforms (evidence: `dags/transport_stats_dag.py`)
- Normalize facts → Raw snapshots: reads JSON (evidence: `src/transform.py`)
- Normalize facts → Processed CSVs: writes CSV (evidence: `src/transform.py`)
- Monthly DAG → Warehouse loader: invokes loaders (evidence: `dags/transport_stats_dag.py`)
- Warehouse loader → Raw snapshots: stages payloads (evidence: `src/load.py`)
- Warehouse loader → Processed CSVs: loads CSVs (evidence: `src/load.py`)
- Warehouse loader → PostgreSQL warehouse: upserts facts (evidence: `src/load.py`)
- Business dashboard → Forecast models: uses models (evidence: `dashboard/app_business.py`)
- PostgreSQL warehouse → Business dashboard: serves analytics (evidence: `README.md`)
- PostgreSQL warehouse → Engineering dashboard: serves monitoring (evidence: `README.md`)
- Business dashboard → Dashboard datasets: uses datasets (evidence: `README.md`)
- Business dashboard → Engineering dashboard: offers dashboards (evidence: `README.md`)