# aivietnam-aio-phamtien/outbreak-signal-de architecture

> Architecture diagram of the GitHub repository aivietnam-aio-phamtien/outbreak-signal-de, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aivietnam-aio-phamtien/outbreak-signal-de
- Repository: https://github.com/aivietnam-aio-phamtien/outbreak-signal-de

## Overview

OutbreakSignal DE is a dengue early-warning data-ingestion system for Southeast Asia. Its main workflow is to fetch official source data into an immutable Landing area, then use PySpark to validate and write queryable Delta Bronze tables with lineage. The source-specific jobs cover case counts, news, administrative boundaries and population, Thai and Philippine case data, and Singapore outbreak clusters. Shared configuration, HTTP, Spark, validation, paths, logging and run metadata support ingestion. Evidence is limited to the README and supplied excerpts/index; links for unsampled jobs rely on documented workflow or index references.

Main entry points: [`ingestion/opendengue.py`](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/opendengue.py) (OpenDengue), [`ingestion/who_gho.py`](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/who_gho.py) (WHO GHO), [`ingestion/news_rss.py`](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/news_rss.py) (News RSS), [`scripts/run_batch.py`](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/scripts/run_batch.py) (Batch runner)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_orchestration["Batch orchestration"]
  node_batch["Batch runner<br/>[run_batch.py]"]
end

subgraph group_sources["Source ingestion"]
  node_opendengue["OpenDengue<br/>[opendengue.py]"]
  node_who["WHO GHO<br/>[who_gho.py]"]
  node_news["News RSS<br/>[news_rss.py]"]
  node_hdx["HDX COD<br/>[hdx_cod.py]"]
  node_trends["TRENDS Thailand<br/>[trends_th.py]"]
  node_phdoh["Philippines DOH<br/>[ph_doh.py]"]
  node_nea["Singapore NEA<br/>[sg_nea.py]"]
end

subgraph group_storage["Bronze storage"]
  node_landing[("Raw Landing")]
  node_bronze[("Delta Bronze")]
  node_runmeta["Run metadata"]
end

subgraph group_shared["Shared ingestion"]
  node_config["Source config<br/>[config.py]"]
  node_http["HTTP retry<br/>[http.py]"]
  node_spark["Spark session<br/>[spark_session.py]"]
  node_validate["Ingestion validation<br/>[validation.py]"]
  node_bronzelogic["Bronze writer<br/>[bronze.py]"]
  node_paths["Path utilities<br/>[paths.py]"]
  node_metadata["Metadata tracking<br/>[metadata.py]"]
  node_logging["Logging<br/>[logging.py]"]
  node_hdxhelper["HDX downloads<br/>[hdx.py]"]
  node_excel["Excel reader<br/>[excel.py]"]
end

node_operator(("Operator"))
node_external(("Official data sources"))

node_operator -->|"runs"| node_batch
node_batch -->|"dispatches"| node_opendengue
node_batch -->|"dispatches"| node_who
node_batch -->|"dispatches"| node_news
node_batch -->|"dispatches"| node_hdx
node_batch -->|"dispatches"| node_trends
node_batch -->|"dispatches"| node_phdoh
node_batch -->|"dispatches"| node_nea
node_opendengue -->|"fetches"| node_external
node_who -->|"fetches"| node_external
node_news -->|"fetches"| node_external
node_hdx -->|"fetches"| node_external
node_trends -->|"fetches"| node_external
node_phdoh -->|"fetches"| node_external
node_nea -->|"fetches"| node_external
node_opendengue -->|"stores raw files"| node_landing
node_news -->|"stores raw files"| node_landing
node_who -->|"stores raw files"| node_landing
node_hdx -->|"stores raw files"| node_landing
node_trends -->|"stores raw files"| node_landing
node_phdoh -->|"stores raw files"| node_landing
node_nea -->|"stores raw files"| node_landing
node_opendengue -->|"writes Delta"| node_bronze
node_news -->|"writes Delta"| node_bronze
node_who -->|"writes Delta"| node_bronze
node_hdx -->|"writes Delta"| node_bronze
node_trends -->|"writes Delta"| node_bronze
node_phdoh -->|"writes Delta"| node_bronze
node_nea -->|"writes Delta"| node_bronze
node_opendengue -->|"uses"| node_http
node_news -->|"uses"| node_http
node_who -->|"uses"| node_http
node_hdx -->|"uses"| node_http
node_opendengue -->|"loads settings"| node_config
node_who -->|"loads settings"| node_config
node_news -->|"loads settings"| node_config
node_hdx -->|"loads settings"| node_config
node_opendengue -->|"builds session"| node_spark
node_who -->|"builds session"| node_spark
node_news -->|"builds session"| node_spark
node_hdx -->|"builds session"| node_spark
node_opendengue -->|"validates input"| node_validate
node_who -->|"validates input"| node_validate
node_news -->|"validates input"| node_validate
node_hdx -->|"validates input"| node_validate
node_opendengue -->|"calls writer"| node_bronzelogic
node_who -->|"calls writer"| node_bronzelogic
node_news -->|"calls writer"| node_bronzelogic

click node_batch "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/scripts/run_batch.py"
click node_opendengue "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/opendengue.py"
click node_who "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/who_gho.py"
click node_news "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/news_rss.py"
click node_hdx "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/hdx_cod.py"
click node_trends "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/trends_th.py"
click node_phdoh "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/ph_doh.py"
click node_nea "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/sg_nea.py"
click node_config "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/config.py"
click node_http "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/http.py"
click node_spark "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/spark_session.py"
click node_validate "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/validation.py"
click node_bronzelogic "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/bronze.py"
click node_paths "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/paths.py"
click node_metadata "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/metadata.py"
click node_logging "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/logging.py"
click node_hdxhelper "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/hdx.py"
click node_excel "https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/excel.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_batch toneBlue
class node_opendengue,node_who,node_news,node_hdx,node_trends,node_phdoh,node_nea toneAmber
class node_landing,node_bronze,node_runmeta toneMint
class node_config,node_http,node_spark,node_validate,node_bronzelogic,node_paths,node_metadata,node_logging,node_hdxhelper,node_excel toneRose
class node_operator,node_external toneIndigo
```

## Components

### Batch orchestration

- [Batch runner](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/scripts/run_batch.py): `scripts/run_batch.py`

### Source ingestion

- [OpenDengue](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/opendengue.py): `ingestion/opendengue.py`
- [WHO GHO](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/who_gho.py): `ingestion/who_gho.py`
- [News RSS](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/news_rss.py): `ingestion/news_rss.py`
- [HDX COD](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/hdx_cod.py): `ingestion/hdx_cod.py`
- [TRENDS Thailand](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/trends_th.py): `ingestion/trends_th.py`
- [Philippines DOH](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/ph_doh.py): `ingestion/ph_doh.py`
- [Singapore NEA](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/sg_nea.py): `ingestion/sg_nea.py`

### Bronze storage

- **Raw Landing**
- **Delta Bronze**
- **Run metadata**

### Shared ingestion

- [Source config](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/config.py): `ingestion/common/config.py`
- [HTTP retry](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/http.py): `ingestion/common/http.py`
- [Spark session](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/spark_session.py): `ingestion/common/spark_session.py`
- [Ingestion validation](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/validation.py): `ingestion/common/validation.py`
- [Bronze writer](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/bronze.py): `ingestion/common/bronze.py`
- [Path utilities](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/paths.py): `ingestion/common/paths.py`
- [Metadata tracking](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/metadata.py): `ingestion/common/metadata.py`
- [Logging](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/logging.py): `ingestion/common/logging.py`
- [HDX downloads](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/hdx.py): `ingestion/common/hdx.py`
- [Excel reader](https://github.com/aivietnam-aio-phamtien/outbreak-signal-de/blob/main/ingestion/common/excel.py): `ingestion/common/excel.py`

### Other components

- **Operator**
- **Official data sources**

## Connections

- Operator → Batch runner: runs (evidence: `README.md`)
- Batch runner → OpenDengue: dispatches (evidence: `scripts/run_batch.py`)
- Batch runner → WHO GHO: dispatches (evidence: `scripts/run_batch.py`)
- Batch runner → News RSS: dispatches (evidence: `scripts/run_batch.py`)
- Batch runner → HDX COD: dispatches (evidence: `scripts/run_batch.py`)
- Batch runner → TRENDS Thailand: dispatches (evidence: `scripts/run_batch.py`)
- Batch runner → Philippines DOH: dispatches (evidence: `scripts/run_batch.py`)
- Batch runner → Singapore NEA: dispatches (evidence: `scripts/run_batch.py`)
- OpenDengue → Official data sources: fetches (evidence: `README.md`)
- WHO GHO → Official data sources: fetches (evidence: `README.md`)
- News RSS → Official data sources: fetches (evidence: `README.md`)
- HDX COD → Official data sources: fetches (evidence: `README.md`)
- TRENDS Thailand → Official data sources: fetches (evidence: `README.md`)
- Philippines DOH → Official data sources: fetches (evidence: `README.md`)
- Singapore NEA → Official data sources: fetches (evidence: `README.md`)
- OpenDengue → Raw Landing: stores raw files (evidence: `ingestion/opendengue.py`)
- News RSS → Raw Landing: stores raw files (evidence: `ingestion/news_rss.py`)
- WHO GHO → Raw Landing: stores raw files (evidence: `ingestion/who_gho.py`)
- HDX COD → Raw Landing: stores raw files (evidence: `README.md`)
- TRENDS Thailand → Raw Landing: stores raw files (evidence: `README.md`)
- Philippines DOH → Raw Landing: stores raw files (evidence: `README.md`)
- Singapore NEA → Raw Landing: stores raw files (evidence: `README.md`)
- OpenDengue → Delta Bronze: writes Delta (evidence: `ingestion/opendengue.py`)
- News RSS → Delta Bronze: writes Delta (evidence: `README.md`)
- WHO GHO → Delta Bronze: writes Delta (evidence: `ingestion/who_gho.py`)
- HDX COD → Delta Bronze: writes Delta (evidence: `ingestion/hdx_cod.py`)
- TRENDS Thailand → Delta Bronze: writes Delta (evidence: `README.md`)
- Philippines DOH → Delta Bronze: writes Delta (evidence: `README.md`)
- Singapore NEA → Delta Bronze: writes Delta (evidence: `README.md`)
- OpenDengue → HTTP retry: uses (evidence: `ingestion/opendengue.py`)
- News RSS → HTTP retry: uses (evidence: `ingestion/news_rss.py`)
- WHO GHO → HTTP retry: uses (evidence: `ingestion/who_gho.py`)
- HDX COD → HTTP retry: uses (evidence: `ingestion/hdx_cod.py`)
- OpenDengue → Source config: loads settings (evidence: `ingestion/opendengue.py`)
- WHO GHO → Source config: loads settings (evidence: `ingestion/who_gho.py`)
- News RSS → Source config: loads settings (evidence: `ingestion/news_rss.py`)
- HDX COD → Source config: loads settings (evidence: `ingestion/hdx_cod.py`)
- OpenDengue → Spark session: builds session (evidence: `ingestion/opendengue.py`)
- WHO GHO → Spark session: builds session (evidence: `ingestion/who_gho.py`)
- News RSS → Spark session: builds session (evidence: `ingestion/news_rss.py`)
- HDX COD → Spark session: builds session (evidence: `ingestion/hdx_cod.py`)
- OpenDengue → Ingestion validation: validates input (evidence: `ingestion/opendengue.py`)
- WHO GHO → Ingestion validation: validates input (evidence: `ingestion/who_gho.py`)
- News RSS → Ingestion validation: validates input (evidence: `ingestion/news_rss.py`)
- HDX COD → Ingestion validation: validates input (evidence: `ingestion/hdx_cod.py`)
- OpenDengue → Bronze writer: calls writer (evidence: `ingestion/opendengue.py`)
- WHO GHO → Bronze writer: calls writer (evidence: `ingestion/who_gho.py`)
- News RSS → Bronze writer: calls writer (evidence: `ingestion/news_rss.py`)
