# whoishusni/berita_scrapper_api architecture

> Architecture diagram of the GitHub repository whoishusni/berita_scrapper_api, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/whoishusni/berita_scrapper_api
- Repository: https://github.com/whoishusni/berita_scrapper_api

## Overview

This repository scrapes trending Indonesian news from four portals, normalizes and aggregates articles, and exposes cached results through a FastAPI service. The workflow starts either from the CLI or an API request: source-specific scrapers collect articles, then results are saved as JSON and Excel (the API refresh path writes JSON). API routes read the JSON cache, filtering news for clients; an empty cache triggers a scrape. The README documents all four portal integrations and API capabilities; some scraper internals were not sampled, so uncertain internal call links are omitted.

Main entry points: [`scraper.py`](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py) (Scrape aggregator), [`api.py`](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py) (Cache loader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["News collection"]
  node_cli_scrape["CLI scrape<br/>[scraper.py]"]
  node_scrape_all["Scrape aggregator<br/>[scraper.py]"]
  node_kompas_scraper["Kompas scraper<br/>[scraper.py]"]
  node_detik_scraper["Detik scraper<br/>[scraper.py]"]
  node_kumparan_scraper["Kumparan scraper<br/>[scraper.py]"]
  node_narasi_scraper["Narasi scraper<br/>[scraper.py]"]
  node_article_builder["Article builder<br/>[scraper.py]"]
end

subgraph group_persistence["Output and cache"]
  node_json_store[("JSON cache<br/>[scraper.py]")]
  node_excel_export["Excel export<br/>[scraper.py]"]
end

subgraph group_service["HTTP service"]
  node_fastapi["FastAPI app<br/>[api.py]"]
  node_cache_loader["Cache loader<br/>[api.py]"]
end

subgraph group_query["News queries"]
  node_source_endpoint["Sources endpoint<br/>[api.py]"]
  node_news_endpoints["News endpoints<br/>[api.py]"]
  node_scrape_endpoint["Scrape endpoint<br/>[api.py]"]
  node_article_filter["Article filter<br/>[api.py]"]
end

node_cli(("CLI user"))
node_api_client(("API client"))
node_kompas{{"Kompas.com"}}
node_detik{{"Detik.com"}}
node_kumparan{{"Kumparan.com"}}
node_narasi{{"Narasi.tv"}}

node_cli -->|"runs"| node_cli_scrape
node_cli_scrape -->|"invokes"| node_scrape_all
node_scrape_all -->|"dispatches"| node_kompas_scraper
node_scrape_all -->|"dispatches"| node_detik_scraper
node_scrape_all -->|"dispatches"| node_kumparan_scraper
node_scrape_all -->|"dispatches"| node_narasi_scraper
node_kompas_scraper -->|"fetches"| node_kompas
node_detik_scraper -->|"fetches"| node_detik
node_kumparan_scraper -->|"fetches"| node_kumparan
node_narasi_scraper -->|"fetches"| node_narasi
node_kompas_scraper -->|"builds articles"| node_article_builder
node_detik_scraper -->|"builds articles"| node_article_builder
node_kumparan_scraper -->|"builds articles"| node_article_builder
node_narasi_scraper -->|"builds articles"| node_article_builder
node_scrape_all -->|"writes"| node_json_store
node_scrape_all -->|"exports"| node_excel_export
node_api_client -->|"requests"| node_fastapi
node_fastapi -->|"routes"| node_source_endpoint
node_fastapi -->|"routes"| node_news_endpoints
node_fastapi -->|"routes"| node_scrape_endpoint
node_source_endpoint -->|"loads cache"| node_cache_loader
node_news_endpoints -->|"loads cache"| node_cache_loader
node_cache_loader -->|"reads"| node_json_store
node_cache_loader -->|"scrapes if empty"| node_scrape_all
node_cache_loader -->|"writes refreshed cache"| node_json_store
node_news_endpoints -->|"filters via"| node_article_filter
node_scrape_endpoint -->|"triggers"| node_scrape_all
node_scrape_endpoint -->|"writes"| node_json_store

click node_cli_scrape "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_scrape_all "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_kompas_scraper "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_detik_scraper "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_kumparan_scraper "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_narasi_scraper "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_article_builder "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_json_store "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_excel_export "https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py"
click node_fastapi "https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py"
click node_cache_loader "https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py"
click node_source_endpoint "https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py"
click node_news_endpoints "https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py"
click node_scrape_endpoint "https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py"
click node_article_filter "https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli_scrape,node_scrape_all,node_kompas_scraper,node_detik_scraper,node_kumparan_scraper,node_narasi_scraper,node_article_builder,node_cli,node_api_client toneBlue
class node_json_store,node_excel_export toneAmber
class node_fastapi,node_cache_loader toneMint
class node_source_endpoint,node_news_endpoints,node_scrape_endpoint,node_article_filter toneRose
class node_kompas,node_detik,node_kumparan,node_narasi toneIndigo
```

## Components

### News collection

- [CLI scrape](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Scrape aggregator](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Kompas scraper](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Detik scraper](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Kumparan scraper](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Narasi scraper](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Article builder](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`

### Output and cache

- [JSON cache](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`
- [Excel export](https://github.com/whoishusni/berita_scrapper_api/blob/master/scraper.py): `scraper.py`

### HTTP service

- [FastAPI app](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py): `api.py`
- [Cache loader](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py): `api.py`

### News queries

- [Sources endpoint](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py): `api.py`
- [News endpoints](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py): `api.py`
- [Scrape endpoint](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py): `api.py`
- [Article filter](https://github.com/whoishusni/berita_scrapper_api/blob/master/api.py): `api.py`

### Other components

- **CLI user**
- **API client**
- **Kompas.com**
- **Detik.com**
- **Kumparan.com**
- **Narasi.tv**

## Connections

- CLI user → CLI scrape: runs
- CLI scrape → Scrape aggregator: invokes
- Scrape aggregator → Kompas scraper: dispatches
- Scrape aggregator → Detik scraper: dispatches
- Scrape aggregator → Kumparan scraper: dispatches
- Scrape aggregator → Narasi scraper: dispatches
- Kompas scraper → Kompas.com: fetches
- Detik scraper → Detik.com: fetches
- Kumparan scraper → Kumparan.com: fetches
- Narasi scraper → Narasi.tv: fetches
- Kompas scraper → Article builder: builds articles
- Detik scraper → Article builder: builds articles
- Kumparan scraper → Article builder: builds articles
- Narasi scraper → Article builder: builds articles
- Scrape aggregator → JSON cache: writes
- Scrape aggregator → Excel export: exports
- API client → FastAPI app: requests
- FastAPI app → Sources endpoint: routes
- FastAPI app → News endpoints: routes
- FastAPI app → Scrape endpoint: routes
- Sources endpoint → Cache loader: loads cache
- News endpoints → Cache loader: loads cache
- Cache loader → JSON cache: reads
- Cache loader → Scrape aggregator: scrapes if empty
- Cache loader → JSON cache: writes refreshed cache
- News endpoints → Article filter: filters via
- Scrape endpoint → Scrape aggregator: triggers
- Scrape endpoint → JSON cache: writes