# rkt-1597/civicbot architecture

> Architecture diagram of the GitHub repository rkt-1597/civicbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rkt-1597/civicbot
- Repository: https://github.com/rkt-1597/civicbot

## Overview

CivicBot is a Cambridge municipal-services guide built around crawling council websites, preparing and analysing their content, and answering citizen questions with source links. The principal workflow runs from a robots-policy check through crawling, cleaning, fact extraction, and indexing to retrieval-backed answers in the Streamlit app. Separate analysis and change-tracking capabilities feed the dashboard. The graph uses README pipeline statements and the supplied source excerpts/index; unsampled implementation details are represented only where the README documents them.

Main entry points: [`chat/ask.py`](https://github.com/rkt-1597/civicbot/blob/main/chat/ask.py) (Question answering), [`crawler/run_crawl.py`](https://github.com/rkt-1597/civicbot/blob/main/crawler/run_crawl.py) (Crawl runner), [`app.py`](https://github.com/rkt-1597/civicbot/blob/main/app.py) (Streamlit application), [`crawler/civic_spider.py`](https://github.com/rkt-1597/civicbot/blob/main/crawler/civic_spider.py) (Civic spider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Collection & updates"]
  node_robots["Robots check<br/>[robots_check.py]"]
  node_crawl_runner["Crawl runner<br/>[run_crawl.py]"]
  node_spider["Civic spider<br/>[civic_spider.py]"]
  node_change_detect["Change detection<br/>[change_detect.py]"]
  node_config["Shared configuration<br/>[config.py]"]
end

subgraph group_content["Content preparation"]
  node_clean["Text cleaning<br/>[clean.py]"]
  node_extract["Fact extraction<br/>[extract.py]"]
end

subgraph group_analysis["Site analysis"]
  node_clusters["Topic clustering<br/>[cluster_pages.py]"]
  node_link_analysis["Link analysis<br/>[link_graph.py]"]
  node_deadlines["Deadline timeline"]
end

subgraph group_chat["Search &amp; answers"]
  node_chat_index["Hybrid index builder<br/>[index.py]"]
  node_chat_ask["Question answering<br/>[ask.py]"]
  node_evaluation["Retrieval evaluation<br/>[evaluate.py]"]
end

subgraph group_app["Application"]
  node_snapshot["Demo snapshot<br/>[make_snapshot.py]"]
  node_app["Streamlit application<br/>[app.py]"]
  node_snapshot_data["Snapshot data"]
end

node_citizen(("Citizen"))
node_council_sites(("Council sites"))
node_llm_services{{"Gemini / Ollama"}}

node_robots -->|"checks policy"| node_council_sites
node_robots -->|"permits crawl"| node_crawl_runner
node_crawl_runner -->|"starts crawl"| node_spider
node_spider -->|"fetches pages"| node_council_sites
node_spider -->|"writes crawl pages"| node_clean
node_clean -->|"provides clean text"| node_extract
node_extract -->|"feeds analysis"| node_clusters
node_clusters -->|"provides topic labels"| node_link_analysis
node_link_analysis -->|"precedes timeline"| node_deadlines
node_clean -->|"reads cleaned pages"| node_chat_index
node_clusters -->|"reads cluster labels"| node_chat_index
node_chat_index -->|"supplies search index"| node_chat_ask
node_chat_ask -->|"returns sourced answer"| node_citizen
node_chat_ask -.->|"optionally generates"| node_llm_services
node_chat_ask -->|"evaluates retrieval"| node_evaluation
node_crawl_runner -->|"compares crawl"| node_change_detect
node_change_detect -->|"updates snapshot inputs"| node_snapshot
node_snapshot -->|"saves outputs"| node_snapshot_data
node_snapshot_data -->|"loads demo data"| node_app
node_citizen -->|"asks question"| node_app
node_app -->|"calls answering"| node_chat_ask
node_app -->|"reads dashboard data"| node_snapshot_data
node_config -->|"configures crawl"| node_crawl_runner
node_config -->|"configures models"| node_chat_ask

click node_robots "https://github.com/rkt-1597/civicbot/blob/main/crawler/robots_check.py"
click node_crawl_runner "https://github.com/rkt-1597/civicbot/blob/main/crawler/run_crawl.py"
click node_spider "https://github.com/rkt-1597/civicbot/blob/main/crawler/civic_spider.py"
click node_change_detect "https://github.com/rkt-1597/civicbot/blob/main/crawler/change_detect.py"
click node_clean "https://github.com/rkt-1597/civicbot/blob/main/preprocess/clean.py"
click node_extract "https://github.com/rkt-1597/civicbot/blob/main/preprocess/extract.py"
click node_clusters "https://github.com/rkt-1597/civicbot/blob/main/analysis/cluster_pages.py"
click node_link_analysis "https://github.com/rkt-1597/civicbot/blob/main/analysis/link_graph.py"
click node_deadlines "https://github.com/rkt-1597/civicbot/blob/main/analysis/deadline_timeline.py"
click node_chat_index "https://github.com/rkt-1597/civicbot/blob/main/chat/index.py"
click node_chat_ask "https://github.com/rkt-1597/civicbot/blob/main/chat/ask.py"
click node_evaluation "https://github.com/rkt-1597/civicbot/blob/main/chat/evaluate.py"
click node_snapshot "https://github.com/rkt-1597/civicbot/blob/main/make_snapshot.py"
click node_app "https://github.com/rkt-1597/civicbot/blob/main/app.py"
click node_snapshot_data "https://github.com/rkt-1597/civicbot/tree/main/data/snapshot"
click node_config "https://github.com/rkt-1597/civicbot/blob/main/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_robots,node_crawl_runner,node_spider,node_change_detect,node_config toneBlue
class node_clean,node_extract toneAmber
class node_clusters,node_link_analysis,node_deadlines toneMint
class node_chat_index,node_chat_ask,node_evaluation toneRose
class node_snapshot,node_app,node_snapshot_data,node_citizen,node_council_sites,node_llm_services toneIndigo
```

## Components

### Collection & updates

- [Robots check](https://github.com/rkt-1597/civicbot/blob/main/crawler/robots_check.py): `crawler/robots_check.py`
- [Crawl runner](https://github.com/rkt-1597/civicbot/blob/main/crawler/run_crawl.py): `crawler/run_crawl.py`
- [Civic spider](https://github.com/rkt-1597/civicbot/blob/main/crawler/civic_spider.py): `crawler/civic_spider.py`
- [Change detection](https://github.com/rkt-1597/civicbot/blob/main/crawler/change_detect.py): `crawler/change_detect.py`
- [Shared configuration](https://github.com/rkt-1597/civicbot/blob/main/config.py): `config.py`

### Content preparation

- [Text cleaning](https://github.com/rkt-1597/civicbot/blob/main/preprocess/clean.py): `preprocess/clean.py`
- [Fact extraction](https://github.com/rkt-1597/civicbot/blob/main/preprocess/extract.py): `preprocess/extract.py`

### Site analysis

- [Topic clustering](https://github.com/rkt-1597/civicbot/blob/main/analysis/cluster_pages.py): `analysis/cluster_pages.py`
- [Link analysis](https://github.com/rkt-1597/civicbot/blob/main/analysis/link_graph.py): `analysis/link_graph.py`
- [Deadline timeline](https://github.com/rkt-1597/civicbot/blob/main/analysis/deadline_timeline.py): `analysis/deadline_timeline.py`

### Search & answers

- [Hybrid index builder](https://github.com/rkt-1597/civicbot/blob/main/chat/index.py): `chat/index.py`
- [Question answering](https://github.com/rkt-1597/civicbot/blob/main/chat/ask.py): `chat/ask.py`
- [Retrieval evaluation](https://github.com/rkt-1597/civicbot/blob/main/chat/evaluate.py): `chat/evaluate.py`

### Application

- [Demo snapshot](https://github.com/rkt-1597/civicbot/blob/main/make_snapshot.py): `make_snapshot.py`
- [Streamlit application](https://github.com/rkt-1597/civicbot/blob/main/app.py): `app.py`
- [Snapshot data](https://github.com/rkt-1597/civicbot/tree/main/data/snapshot): `data/snapshot`

### Other components

- **Citizen**
- **Council sites**
- **Gemini / Ollama**

## Connections

- Robots check → Council sites: checks policy (evidence: `crawler/robots_check.py`)
- Robots check → Crawl runner: permits crawl (evidence: `README.md`)
- Crawl runner → Civic spider: starts crawl (evidence: `crawler/run_crawl.py`)
- Civic spider → Council sites: fetches pages (evidence: `README.md`)
- Civic spider → Text cleaning: writes crawl pages (evidence: `README.md`)
- Text cleaning → Fact extraction: provides clean text (evidence: `README.md`)
- Fact extraction → Topic clustering: feeds analysis (evidence: `README.md`)
- Topic clustering → Link analysis: provides topic labels (evidence: `README.md`)
- Link analysis → Deadline timeline: precedes timeline (evidence: `README.md`)
- Text cleaning → Hybrid index builder: reads cleaned pages (evidence: `chat/index.py`)
- Topic clustering → Hybrid index builder: reads cluster labels (evidence: `chat/index.py`)
- Hybrid index builder → Question answering: supplies search index (evidence: `README.md`)
- Question answering → Citizen: returns sourced answer (evidence: `README.md`)
- Question answering → Gemini / Ollama: optionally generates (evidence: `README.md`)
- Question answering → Retrieval evaluation: evaluates retrieval (evidence: `README.md`)
- Crawl runner → Change detection: compares crawl (evidence: `README.md`)
- Change detection → Demo snapshot: updates snapshot inputs (evidence: `README.md`)
- Demo snapshot → Snapshot data: saves outputs (evidence: `README.md`)
- Snapshot data → Streamlit application: loads demo data (evidence: `README.md`)
- Citizen → Streamlit application: asks question (evidence: `README.md`)
- Streamlit application → Question answering: calls answering (evidence: `app.py`)
- Streamlit application → Snapshot data: reads dashboard data (evidence: `README.md`)
- Shared configuration → Crawl runner: configures crawl (evidence: `crawler/run_crawl.py`)
- Shared configuration → Question answering: configures models (evidence: `chat/ask.py`)