# billycemerson/indo-gpr-index architecture

> Architecture diagram of the GitHub repository billycemerson/indo-gpr-index, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/billycemerson/indo-gpr-index
- Repository: https://github.com/billycemerson/indo-gpr-index

## Overview

1. **What this repo is**
   - This is a scheduled Python data pipeline for Indonesia’s Geopolitical Risk (GPR) index.
   - The system scrapes political/news articles from multiple Indonesian media sites, stores raw data, transforms it with dbt, and exports the final daily index to Google Sheets.

2. **End-to-end data flow**
   - The pipeline is: **scrape news → write raw JSON → load into DuckDB → dbt staging → dbt mart → Google Sheets export**.
   - The key intermediate persistence point is the **raw news table** in DuckDB, which separates collection from transformation.

3. **Scraping subsystem**
   - `src/scraper/` is the collection layer.
   - `base_parser.py` defines the parser contract, and `src/scraper/parsers/` contains source-specific parsers for **Antara, Detik, Kompas, Tempo, and Tribunnews**.
   - `main_scraper.py` orchestrates all parsers so the pipeline can harvest multiple sources in a single run.
   - The scraper is source-aware: each parser targets specific categories or site layouts, then normalizes articles into a common structure.

4. **Raw ingestion boundary**
   - `src/load_to_db.py` is the handoff from scraping to analytics storage.
   - `src/database/connection.py` centralizes the database connection, and the repo’s structure strongly suggests **DuckDB** as the local analytical store.
   - This layer is responsible for getting scraped article records into a structured table without doing business logic transformations.

5. **Transformation layer with dbt**
   - `dbt_project/` owns the analytical modeling boundary.
   - `models/staging/stg_scraped_news.sql` performs **deduplication and cleanup** of raw scraped data.
   - `models/marts/mart_gpr_daily.sql` computes the daily GPR output.
   - `models/staging/sources.yml` defines the raw source inputs, and `dbt_project.yml` plus `profiles.yml` define how dbt connects to the database.

6. **GPR calculation logic**
   - The mart calculates article-level keyword matches across five dimensions: **war threat, peace threat, military buildup, war act, and terror act**.
   - `macros/match_keywords.sql` centralizes the keyword matching logic so the same rule set can be reused consistently in dbt models.
   - The final output includes component indexes and composites such as **`gpr_threats_index`**, **`gpr_acts_index`**, and **`total_gpr_index`**.

7. **Export boundary**
   - `src/export_table.py` is the final delivery step.
   - `src/exports/gsheet_client.py` handles the Google Sheets integration, using a service-account credential file.
   - This boundary is deliberately narrow: it consumes the final mart and pushes it outward, without re-reading source articles or re-computing metrics.

8. **Orchestration and execution**
   - `src/pipelines/run_pipeline.py` and the top-level `main_pipeline.py` coordinate the full workflow.
   - The repo supports both full-run execution and stepwise execution for debugging or local development.
   - `scripts/try_parser.py` is a focused utility for testing an individual parser before running the entire pipeline.

9. **Runtime configuration and environments**
   - `config/settings.py` and `APP_ENV` control environment-specific behavior.
   - The repo distinguishes **dev** and **prod** data paths, which affects where scraped and intermediate files are written.
   - This separation is important because the pipeline has both local testing workflows and scheduled production execution.

10. **Scheduling and deployment**
   - `.github/workflows/pipeline.yml` runs the pipeline on a daily schedule, and the README states the target time is **08:00 WIB**.
   - GitHub Actions is the operational scheduler and CI/CD entry point.
   - The workflow relies on a Google Sheets secret passed as `GSHEET_KEY_JSON`.

11. **Testing strategy**
   - `tests/test_parsers.py` validates source parsing behavior, which is critical because source layouts change often.
   - `tests/test_load_to_db.py` and `tests/test_export.py` cover the ingestion and export boundaries.
   - `conftest.py` files support test setup, indicating the repo treats each stage as a separately testable subsystem.

12. **Practical architecture shape**
   - The cleanest diagram for this repo is a **three-layer pipeline**:
     - **Collection**: source-specific scrapers and parsers
     - **Analytics storage and modeling**: DuckDB + dbt staging/mart
     - **Delivery**: Google Sheets export
   - The main boundary to emphasize is that scraping is source-dependent and brittle, while the dbt models are the stable analytical core.

Main entry points: [`src/scraper/parsers`](https://github.com/billycemerson/indo-gpr-index/tree/main/src/scraper/parsers) (Source parsers), [`src/scraper/main_scraper.py`](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/main_scraper.py) (Scraper run), [`dbt_project`](https://github.com/billycemerson/indo-gpr-index/tree/main/dbt_project) (dbt project), [`src/pipelines/run_pipeline.py`](https://github.com/billycemerson/indo-gpr-index/blob/main/src/pipelines/run_pipeline.py) (Run pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ops["Ops"]
  node_main_pipeline["Main pipeline<br/>entrypoint<br/>[main_pipeline.py]"]
  node_run_pipeline["Run pipeline<br/>orchestrator<br/>[run_pipeline.py]"]
  node_pipeline_workflow["GitHub Actions<br/>scheduler<br/>[pipeline.yml]"]
  node_settings["Settings<br/>config<br/>[settings.py]"]
  node_try_parser["Try parser<br/>[try_parser.py]"]
end

subgraph group_collection["Collection"]
  node_main_scraper["Scraper run<br/>orchestrator<br/>[main_scraper.py]"]
  node_base_parser["Base parser<br/>contract<br/>[base_parser.py]"]
  node_parsers["Source parsers<br/>adapters"]
  node_antara_parser["Antara<br/>parser<br/>[antara.py]"]
  node_detik_parser["Detik<br/>parser<br/>[detik.py]"]
  node_kompas_parser["Kompas<br/>parser<br/>[kompas.py]"]
  node_tempo_parser["Tempo<br/>parser<br/>[tempo.py]"]
  node_tribun_parser["Tribunnews<br/>parser<br/>[tribunnews.py]"]
end

subgraph group_analytics["Analytics"]
  node_load_to_db["Load to DB<br/>ingestion<br/>[load_to_db.py]"]
  node_db_connection[("DB connection<br/>duckdb<br/>[connection.py]")]
  node_dbt_project["dbt project<br/>modeling"]
  node_stg_scraped_news["Staging news<br/>dbt staging"]
  node_match_keywords["Keyword match<br/>dbt macro<br/>[match_keywords.sql]"]
  node_mart_gpr_daily["Daily GPR<br/>dbt mart<br/>[mart_gpr_daily.sql]"]
  node_source_defs["Source defs<br/>dbt sources<br/>[sources.yml]"]
end

subgraph group_delivery["Delivery"]
  node_export_table["Export table<br/>delivery<br/>[export_table.py]"]
  node_gsheet_client["Google Sheets<br/>publisher<br/>[gsheet_client.py]"]
end

node_pipeline_workflow -->|"schedules"| node_main_pipeline
node_main_pipeline -->|"delegates"| node_run_pipeline
node_run_pipeline -->|"reads config"| node_settings
node_run_pipeline -->|"collects"| node_main_scraper
node_main_scraper -.->|"implements"| node_base_parser
node_main_scraper -->|"loads"| node_parsers
node_parsers -->|"includes"| node_antara_parser
node_parsers -->|"includes"| node_detik_parser
node_parsers -->|"includes"| node_kompas_parser
node_parsers -->|"includes"| node_tempo_parser
node_parsers -->|"includes"| node_tribun_parser
node_main_scraper -->|"emits raw"| node_load_to_db
node_load_to_db -->|"writes via"| node_db_connection
node_db_connection -->|"feeds"| node_dbt_project
node_dbt_project -->|"defines"| node_source_defs
node_dbt_project -->|"builds"| node_stg_scraped_news
node_dbt_project -->|"reuses"| node_match_keywords
node_stg_scraped_news -->|"transforms into"| node_mart_gpr_daily
node_match_keywords -->|"powers"| node_mart_gpr_daily
node_mart_gpr_daily -->|"publishes"| node_export_table
node_export_table -->|"pushes to"| node_gsheet_client
node_try_parser -.->|"tests"| node_parsers

click node_main_pipeline "https://github.com/billycemerson/indo-gpr-index/blob/main/main_pipeline.py"
click node_run_pipeline "https://github.com/billycemerson/indo-gpr-index/blob/main/src/pipelines/run_pipeline.py"
click node_pipeline_workflow "https://github.com/billycemerson/indo-gpr-index/blob/main/.github/workflows/pipeline.yml"
click node_settings "https://github.com/billycemerson/indo-gpr-index/blob/main/config/settings.py"
click node_main_scraper "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/main_scraper.py"
click node_base_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/base_parser.py"
click node_parsers "https://github.com/billycemerson/indo-gpr-index/tree/main/src/scraper/parsers"
click node_antara_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/antara.py"
click node_detik_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/detik.py"
click node_kompas_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/kompas.py"
click node_tempo_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/tempo.py"
click node_tribun_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/tribunnews.py"
click node_load_to_db "https://github.com/billycemerson/indo-gpr-index/blob/main/src/load_to_db.py"
click node_db_connection "https://github.com/billycemerson/indo-gpr-index/blob/main/src/database/connection.py"
click node_dbt_project "https://github.com/billycemerson/indo-gpr-index/tree/main/dbt_project"
click node_stg_scraped_news "https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/models/staging/stg_scraped_news.sql"
click node_match_keywords "https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/macros/match_keywords.sql"
click node_mart_gpr_daily "https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/models/marts/mart_gpr_daily.sql"
click node_source_defs "https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/models/staging/sources.yml"
click node_export_table "https://github.com/billycemerson/indo-gpr-index/blob/main/src/export_table.py"
click node_gsheet_client "https://github.com/billycemerson/indo-gpr-index/blob/main/src/exports/gsheet_client.py"
click node_try_parser "https://github.com/billycemerson/indo-gpr-index/blob/main/scripts/try_parser.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_pipeline,node_run_pipeline,node_pipeline_workflow,node_settings,node_try_parser toneBlue
class node_main_scraper,node_base_parser,node_parsers,node_antara_parser,node_detik_parser,node_kompas_parser,node_tempo_parser,node_tribun_parser toneAmber
class node_load_to_db,node_db_connection,node_dbt_project,node_stg_scraped_news,node_match_keywords,node_mart_gpr_daily,node_source_defs toneMint
class node_export_table,node_gsheet_client toneRose
```

## Components

### Ops

Scheduling, entrypoints, and runtime config

- [Main pipeline](https://github.com/billycemerson/indo-gpr-index/blob/main/main_pipeline.py): entrypoint · Top-level workflow entrypoint for full or stepped runs · `main_pipeline.py`
- [Run pipeline](https://github.com/billycemerson/indo-gpr-index/blob/main/src/pipelines/run_pipeline.py): orchestrator · Coordinates scrape, load, transform, and export stages · `src/pipelines/run_pipeline.py`
- [GitHub Actions](https://github.com/billycemerson/indo-gpr-index/blob/main/.github/workflows/pipeline.yml): scheduler · Daily scheduled execution in CI/CD · `.github/workflows/pipeline.yml`
- [Settings](https://github.com/billycemerson/indo-gpr-index/blob/main/config/settings.py): config · Environment-specific runtime configuration and paths · `config/settings.py`
- [Try parser](https://github.com/billycemerson/indo-gpr-index/blob/main/scripts/try_parser.py): utility · Focused utility for testing an individual parser · `scripts/try_parser.py`

### Collection

Source-specific scraping and normalization

- [Scraper run](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/main_scraper.py): orchestrator · Runs all source parsers in one collection pass · `src/scraper/main_scraper.py`
- [Base parser](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/base_parser.py): contract · Common parser interface and shared scraping contract · `src/scraper/base_parser.py`
- [Source parsers](https://github.com/billycemerson/indo-gpr-index/tree/main/src/scraper/parsers): adapters · Source-specific parsers for Indonesian news sites · `src/scraper/parsers`
- [Antara](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/antara.py): parser · Antara article parser · `src/scraper/parsers/antara.py`
- [Detik](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/detik.py): parser · Detik article parser · `src/scraper/parsers/detik.py`
- [Kompas](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/kompas.py): parser · Kompas article parser · `src/scraper/parsers/kompas.py`
- [Tempo](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/tempo.py): parser · Tempo article parser · `src/scraper/parsers/tempo.py`
- [Tribunnews](https://github.com/billycemerson/indo-gpr-index/blob/main/src/scraper/parsers/tribunnews.py): parser · Tribunnews article parser · `src/scraper/parsers/tribunnews.py`

### Analytics

Raw ingestion and dbt modeling

- [Load to DB](https://github.com/billycemerson/indo-gpr-index/blob/main/src/load_to_db.py): ingestion · Handoff from scraped records into analytical storage · `src/load_to_db.py`
- [DB connection](https://github.com/billycemerson/indo-gpr-index/blob/main/src/database/connection.py): duckdb · Centralized local analytics database connection · `src/database/connection.py`
- [dbt project](https://github.com/billycemerson/indo-gpr-index/tree/main/dbt_project): modeling · Analytical transformation boundary and project configuration · `dbt_project`
- [Staging news](https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/models/staging/stg_scraped_news.sql): dbt staging · Cleans and deduplicates raw scraped news · `dbt_project/models/staging/stg_scraped_news.sql`
- [Keyword match](https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/macros/match_keywords.sql): dbt macro · Reusable keyword matching logic for GPR signals · `dbt_project/macros/match_keywords.sql`
- [Daily GPR](https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/models/marts/mart_gpr_daily.sql): dbt mart · Computes daily GPR components and composite indices · `dbt_project/models/marts/mart_gpr_daily.sql`
- [Source defs](https://github.com/billycemerson/indo-gpr-index/blob/main/dbt_project/models/staging/sources.yml): dbt sources · Declares raw source inputs for dbt models · `dbt_project/models/staging/sources.yml`

### Delivery

External publishing

- [Export table](https://github.com/billycemerson/indo-gpr-index/blob/main/src/export_table.py): delivery · Final delivery step from mart to external output · `src/export_table.py`
- [Google Sheets](https://github.com/billycemerson/indo-gpr-index/blob/main/src/exports/gsheet_client.py): publisher · Google Sheets integration using service-account credentials · `src/exports/gsheet_client.py`

## Connections

- GitHub Actions → Main pipeline: schedules · GitHub Actions triggers the pipeline entrypoint
- Main pipeline → Run pipeline: delegates · Top-level entrypoint hands off execution to the pipeline runner
- Run pipeline → Settings: reads config · Pipeline runtime uses environment and path settings
- Run pipeline → Scraper run: collects · Orchestrator starts scraping across all sources
- Scraper run → Base parser: implements · Source parsers conform to the shared parser contract
- Scraper run → Source parsers: loads · Scraper coordinates the source-specific parser set
- Source parsers → Antara: includes · Antara parser module
- Source parsers → Detik: includes · Detik parser module
- Source parsers → Kompas: includes · Kompas parser module
- Source parsers → Tempo: includes · Tempo parser module
- Source parsers → Tribunnews: includes · Tribunnews parser module
- Scraper run → Load to DB: emits raw · Scraped article records are handed to ingestion
- Load to DB → DB connection: writes via · Ingestion persists data through the shared database connection
- DB connection → dbt project: feeds · DuckDB-backed raw table is the input for dbt models
- dbt project → Source defs: defines · dbt sources describe the raw inputs
- dbt project → Staging news: builds · Staging model cleans and deduplicates scraped news
- dbt project → Keyword match: reuses · Macro centralizes keyword matching logic
- Staging news → Daily GPR: transforms into · Cleaned staging data is aggregated into the daily mart
- Keyword match → Daily GPR: powers · Macro supports the GPR dimension calculations
- Daily GPR → Export table: publishes · Final mart is exported after transformation
- Export table → Google Sheets: pushes to · Delivery layer writes results into Google Sheets
- Try parser → Source parsers: tests · Utility exercises individual source parsers