How aivietnam-aio-phamtien/outbreak-signal-de works
OutbreakSignal DE is a dengue early-warning data-ingestion system for Southeast Asia. Its main workflow is to fetch official source data into an immutable Landing area, then use PySpark to validate and write queryable Delta Bronze tables with lineage. The source-specific jobs cover case counts, news, administrative boundaries and population, Thai and Philippine case data, and Singapore outbreak clusters. Shared configuration, HTTP, Spark, validation, paths, logging and run metadata support ingestion. Evidence is limited to the README and supplied excerpts/index; links for unsampled jobs rely on documented workflow or index references.
Main entry points: opendengue.py OpenDengue · who_gho.py WHO GHO · news_rss.py News RSS · run_batch.py Batch runner
Components 23
Batch orchestration
- Batch runner
scripts/run_batch.py
Source ingestion
- OpenDengue
ingestion/opendengue.py - WHO GHO
ingestion/who_gho.py - News RSS
ingestion/news_rss.py - HDX COD
ingestion/hdx_cod.py - TRENDS Thailand
ingestion/trends_th.py - Philippines DOH
ingestion/ph_doh.py - Singapore NEA
ingestion/sg_nea.py
Bronze storage
- Raw Landing
- Delta Bronze
- Run metadata
Shared ingestion
- Source config
ingestion/common/config.py - HTTP retry
ingestion/common/http.py - Spark session
ingestion/common/spark_session.py - Ingestion validation
ingestion/common/validation.py - Bronze writer
ingestion/common/bronze.py - Path utilities
ingestion/common/paths.py - Metadata tracking
ingestion/common/metadata.py - Logging
ingestion/common/logging.py - HDX downloads
ingestion/common/hdx.py - Excel reader
ingestion/common/excel.py
Other components
- Operator
- Official data sources
Connections 48
- Operator → Batch runner: runs
README.md - Batch runner → OpenDengue: dispatches
scripts/run_batch.py - Batch runner → WHO GHO: dispatches
scripts/run_batch.py - Batch runner → News RSS: dispatches
scripts/run_batch.py - Batch runner → HDX COD: dispatches
scripts/run_batch.py - Batch runner → TRENDS Thailand: dispatches
scripts/run_batch.py - Batch runner → Philippines DOH: dispatches
scripts/run_batch.py - Batch runner → Singapore NEA: dispatches
scripts/run_batch.py - OpenDengue → Official data sources: fetches
README.md - WHO GHO → Official data sources: fetches
README.md - News RSS → Official data sources: fetches
README.md - HDX COD → Official data sources: fetches
README.md - TRENDS Thailand → Official data sources: fetches
README.md - Philippines DOH → Official data sources: fetches
README.md - Singapore NEA → Official data sources: fetches
README.md - OpenDengue → Raw Landing: stores raw files
ingestion/opendengue.py - News RSS → Raw Landing: stores raw files
ingestion/news_rss.py - WHO GHO → Raw Landing: stores raw files
ingestion/who_gho.py - HDX COD → Raw Landing: stores raw files
README.md - TRENDS Thailand → Raw Landing: stores raw files
README.md - Philippines DOH → Raw Landing: stores raw files
README.md - Singapore NEA → Raw Landing: stores raw files
README.md - OpenDengue → Delta Bronze: writes Delta
ingestion/opendengue.py - News RSS → Delta Bronze: writes Delta
README.md - WHO GHO → Delta Bronze: writes Delta
ingestion/who_gho.py - HDX COD → Delta Bronze: writes Delta
ingestion/hdx_cod.py - TRENDS Thailand → Delta Bronze: writes Delta
README.md - Philippines DOH → Delta Bronze: writes Delta
README.md - Singapore NEA → Delta Bronze: writes Delta
README.md - OpenDengue → HTTP retry: uses
ingestion/opendengue.py - News RSS → HTTP retry: uses
ingestion/news_rss.py - WHO GHO → HTTP retry: uses
ingestion/who_gho.py - HDX COD → HTTP retry: uses
ingestion/hdx_cod.py - OpenDengue → Source config: loads settings
ingestion/opendengue.py - WHO GHO → Source config: loads settings
ingestion/who_gho.py - News RSS → Source config: loads settings
ingestion/news_rss.py - HDX COD → Source config: loads settings
ingestion/hdx_cod.py - OpenDengue → Spark session: builds session
ingestion/opendengue.py - WHO GHO → Spark session: builds session
ingestion/who_gho.py - News RSS → Spark session: builds session
ingestion/news_rss.py - HDX COD → Spark session: builds session
ingestion/hdx_cod.py - OpenDengue → Ingestion validation: validates input
ingestion/opendengue.py - WHO GHO → Ingestion validation: validates input
ingestion/who_gho.py - News RSS → Ingestion validation: validates input
ingestion/news_rss.py - HDX COD → Ingestion validation: validates input
ingestion/hdx_cod.py - OpenDengue → Bronze writer: calls writer
ingestion/opendengue.py - WHO GHO → Bronze writer: calls writer
ingestion/who_gho.py - News RSS → Bronze writer: calls writer
ingestion/news_rss.py