How tslashd/vik-scraper works
This is a VikPZ water-outage scraper that collects annual article pages, extracts outage details, and stores records in MySQL, using OpenAI when local parsing is insufficient. The main workflow is visible in scraper.py; the README is empty, and some scraper.py excerpts are omitted, so only relationships shown in the excerpts or source index are drawn.
Main entry points: scraper.py Page scraper · db.py Database adapter · ai.py AI extractor · logs Local logs
Components 8
Collection
- Annual export
scraper.py - Page scraper
scraper.py
Data extraction
- AI extractor
ai.py
Persistence
- Database adapter
db.py - Local logs
logs
Other components
- VikPZ website
- OpenAI API
- MySQL database
Connections 8
- Annual export → Page scraper: creates scraper
scraper.py - Page scraper → VikPZ website: fetches pages
scraper.py - Page scraper → AI extractor: requests extraction
scraper.py - AI extractor → OpenAI API: sends chat request
ai.py - Annual export → Database adapter: loads existing data
scraper.py - Page scraper → Database adapter: writes records
scraper.py - Database adapter → MySQL database: executes queries
db.py - Page scraper → Local logs: dumps extraction data
scraper.py