# tslashd/vik-scraper architecture

> Architecture diagram of the GitHub repository tslashd/vik-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tslashd/vik-scraper
- Repository: https://github.com/tslashd/vik-scraper

## Overview

This is a VikPZ water-outage scraper that collects annual article pages, extracts outage details, and stores records in MySQL, using OpenAI when local parsing is insufficient. The main workflow is visible in scraper.py; the README is empty, and some scraper.py excerpts are omitted, so only relationships shown in the excerpts or source index are drawn.

Main entry points: [`scraper.py`](https://github.com/tslashd/vik-scraper/blob/main/scraper.py) (Page scraper), [`db.py`](https://github.com/tslashd/vik-scraper/blob/main/db.py) (Database adapter), [`ai.py`](https://github.com/tslashd/vik-scraper/blob/main/ai.py) (AI extractor), [`logs`](https://github.com/tslashd/vik-scraper/tree/main/logs) (Local logs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection"]
  node_annual_run["Annual export<br/>[scraper.py]"]
  node_scraper["Page scraper<br/>[scraper.py]"]
end

subgraph group_extraction["Data extraction"]
  node_ai_extractor["AI extractor<br/>[ai.py]"]
end

subgraph group_storage["Persistence"]
  node_database_adapter["Database adapter<br/>[db.py]"]
  node_local_logs["Local logs"]
end

node_vikpz["VikPZ website"]
node_openai["OpenAI API"]
node_mysql[("MySQL database")]

node_annual_run -->|"creates scraper"| node_scraper
node_scraper -->|"fetches pages"| node_vikpz
node_scraper -->|"requests extraction"| node_ai_extractor
node_ai_extractor -->|"sends chat request"| node_openai
node_annual_run -->|"loads existing data"| node_database_adapter
node_scraper -->|"writes records"| node_database_adapter
node_database_adapter -->|"executes queries"| node_mysql
node_scraper -->|"dumps extraction data"| node_local_logs

click node_annual_run "https://github.com/tslashd/vik-scraper/blob/main/scraper.py"
click node_scraper "https://github.com/tslashd/vik-scraper/blob/main/scraper.py"
click node_ai_extractor "https://github.com/tslashd/vik-scraper/blob/main/ai.py"
click node_database_adapter "https://github.com/tslashd/vik-scraper/blob/main/db.py"
click node_local_logs "https://github.com/tslashd/vik-scraper/tree/main/logs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_annual_run,node_scraper toneBlue
class node_ai_extractor,node_mysql toneAmber
class node_database_adapter,node_local_logs,node_openai toneMint
class node_vikpz toneIndigo
```

## Components

### Collection

- [Annual export](https://github.com/tslashd/vik-scraper/blob/main/scraper.py): `scraper.py`
- [Page scraper](https://github.com/tslashd/vik-scraper/blob/main/scraper.py): `scraper.py`

### Data extraction

- [AI extractor](https://github.com/tslashd/vik-scraper/blob/main/ai.py): `ai.py`

### Persistence

- [Database adapter](https://github.com/tslashd/vik-scraper/blob/main/db.py): `db.py`
- [Local logs](https://github.com/tslashd/vik-scraper/tree/main/logs): `logs`

### Other components

- **VikPZ website**
- **OpenAI API**
- **MySQL database**

## Connections

- Annual export → Page scraper: creates scraper (evidence: `scraper.py`)
- Page scraper → VikPZ website: fetches pages (evidence: `scraper.py`)
- Page scraper → AI extractor: requests extraction (evidence: `scraper.py`)
- AI extractor → OpenAI API: sends chat request (evidence: `ai.py`)
- Annual export → Database adapter: loads existing data (evidence: `scraper.py`)
- Page scraper → Database adapter: writes records (evidence: `scraper.py`)
- Database adapter → MySQL database: executes queries (evidence: `db.py`)
- Page scraper → Local logs: dumps extraction data (evidence: `scraper.py`)