How whoishusni/berita_scrapper_api works
This repository scrapes trending Indonesian news from four portals, normalizes and aggregates articles, and exposes cached results through a FastAPI service. The workflow starts either from the CLI or an API request: source-specific scrapers collect articles, then results are saved as JSON and Excel (the API refresh path writes JSON). API routes read the JSON cache, filtering news for clients; an empty cache triggers a scrape. The README documents all four portal integrations and API capabilities; some scraper internals were not sampled, so uncertain internal call links are omitted.
Main entry points: scraper.py Scrape aggregator · api.py Cache loader
Components 21
News collection
- CLI scrape
scraper.py - Scrape aggregator
scraper.py - Kompas scraper
scraper.py - Detik scraper
scraper.py - Kumparan scraper
scraper.py - Narasi scraper
scraper.py - Article builder
scraper.py
Output and cache
- JSON cache
scraper.py - Excel export
scraper.py
HTTP service
- FastAPI app
api.py - Cache loader
api.py
News queries
- Sources endpoint
api.py - News endpoints
api.py - Scrape endpoint
api.py - Article filter
api.py
Other components
- CLI user
- API client
- Kompas.com
- Detik.com
- Kumparan.com
- Narasi.tv
Connections 28
- CLI user → CLI scrape: runs
- CLI scrape → Scrape aggregator: invokes
- Scrape aggregator → Kompas scraper: dispatches
- Scrape aggregator → Detik scraper: dispatches
- Scrape aggregator → Kumparan scraper: dispatches
- Scrape aggregator → Narasi scraper: dispatches
- Kompas scraper → Kompas.com: fetches
- Detik scraper → Detik.com: fetches
- Kumparan scraper → Kumparan.com: fetches
- Narasi scraper → Narasi.tv: fetches
- Kompas scraper → Article builder: builds articles
- Detik scraper → Article builder: builds articles
- Kumparan scraper → Article builder: builds articles
- Narasi scraper → Article builder: builds articles
- Scrape aggregator → JSON cache: writes
- Scrape aggregator → Excel export: exports
- API client → FastAPI app: requests
- FastAPI app → Sources endpoint: routes
- FastAPI app → News endpoints: routes
- FastAPI app → Scrape endpoint: routes
- Sources endpoint → Cache loader: loads cache
- News endpoints → Cache loader: loads cache
- Cache loader → JSON cache: reads
- Cache loader → Scrape aggregator: scrapes if empty
- Cache loader → JSON cache: writes refreshed cache
- News endpoints → Article filter: filters via
- Scrape endpoint → Scrape aggregator: triggers
- Scrape endpoint → JSON cache: writes