billycemerson/indo-gpr-index

1. What this repo is

- This is a scheduled Python data pipeline for Indonesia’s Geopolitical Risk (GPR) index.

- The system scrapes political/news articles from multiple Indonesian media sites, stores raw data, transforms it with dbt, and exports the final daily index to Google Sheets.

2. End-to-end data flow

- The pipeline is: scrape news → write raw JSON → load into DuckDB → dbt staging → dbt mart → Google Sheets export.

- The key intermediate persistence point is the raw news table in DuckDB, which separates collection from transformation.

3. Scraping subsystem

- src/scraper/ is the collection layer.

- base_parser.py defines the parser contract, and src/scraper/parsers/ contains source-specific parsers for Antara, Detik, Kompas, Tempo, and Tribunnews.

- main_scraper.py orchestrates all parsers so the pipeline can harvest multiple sources in a single run.

- The scraper is source-aware: each parser targets specific categories or site layouts, then normalizes articles into a common structure.

4. Raw ingestion boundary

- src/load_to_db.py is the handoff from scraping to analytics storage.

- src/database/connection.py centralizes the database connection, and the repo’s structure strongly suggests DuckDB as the local analytical store.

- This layer is responsible for getting scraped article records into a structured table without doing business logic transformations.

5. Transformation layer with dbt

- dbt_project/ owns the analytical modeling boundary.

- models/staging/stg_scraped_news.sql performs deduplication and cleanup of raw scraped data.

- models/marts/mart_gpr_daily.sql computes the daily GPR output.

- models/staging/sources.yml defines the raw source inputs, and dbt_project.yml plus profiles.yml define how dbt connects to the database.

6. GPR calculation logic

- The mart calculates article-level keyword matches across five dimensions: war threat, peace threat, military buildup, war act, and terror act.

- macros/match_keywords.sql centralizes the keyword matching logic so the same rule set can be reused consistently in dbt models.

- The final output includes component indexes and composites such as gpr_threats_index, gpr_acts_index, and total_gpr_index.

7. Export boundary

- src/export_table.py is the final delivery step.

- src/exports/gsheet_client.py handles the Google Sheets integration, using a service-account credential file.

- This boundary is deliberately narrow: it consumes the final mart and pushes it outward, without re-reading source articles or re-computing metrics.

8. Orchestration and execution

- src/pipelines/run_pipeline.py and the top-level main_pipeline.py coordinate the full workflow.

- The repo supports both full-run execution and stepwise execution for debugging or local development.

- scripts/try_parser.py is a focused utility for testing an individual parser before running the entire pipeline.

9. Runtime configuration and environments

- config/settings.py and APP_ENV control environment-specific behavior.

- The repo distinguishes dev and prod data paths, which affects where scraped and intermediate files are written.

- This separation is important because the pipeline has both local testing workflows and scheduled production execution.

10. Scheduling and deployment

- .github/workflows/pipeline.yml runs the pipeline on a daily schedule, and the README states the target time is 08:00 WIB.

- GitHub Actions is the operational scheduler and CI/CD entry point.

- The workflow relies on a Google Sheets secret passed as GSHEET_KEY_JSON.

11. Testing strategy

- tests/test_parsers.py validates source parsing behavior, which is critical because source layouts change often.

- tests/test_load_to_db.py and tests/test_export.py cover the ingestion and export boundaries.

- conftest.py files support test setup, indicating the repo treats each stage as a separately testable subsystem.

12. Practical architecture shape

- The cleanest diagram for this repo is a three-layer pipeline:

- Collection: source-specific scrapers and parsers

- Analytics storage and modeling: DuckDB + dbt staging/mart

- Delivery: Google Sheets export

- The main boundary to emphasize is that scraping is source-dependent and brittle, while the dbt models are the stable analytical core.