# bilbobackends/isitvegan architecture

> Architecture diagram of the GitHub repository bilbobackends/isitvegan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bilbobackends/isitvegan
- Repository: https://github.com/bilbobackends/isitvegan

## Overview

1. **System shape**
   - This repo is a two-plane application: a **local Python pipeline + Flask admin API** for discovery/scraping/classification, and a **consumer React frontend** that is also exported as a **fully static site** for GitHub Pages.
   - The public site never talks to external services or holds credentials; only the local backend does.

2. **Persistent store**
   - The central state is a **SQLite database** (`db.py`) that holds restaurants, menu sources, dish records, classification history, crawl profiles, menu versions, dish changes, hidden/archived flags, and job state.
   - The architecture is intentionally history-oriented: menu snapshots and classification runs are preserved so the UI can show change over time instead of only the latest state.

3. **Restaurant discovery**
   - Discovery starts with **Google Places API** via `places_client.py` and `discover.py`, which finds candidate restaurants in a configured area and upserts them into SQLite.
   - `add_restaurants.py` is the confirm-first path for named additions: it resolves text queries to Places candidates, lets the operator pick the exact match, then optionally runs enrichment, scraping, and classification.

4. **Menu ingestion pipeline**
   - `ingest.py` drives scraping of restaurant websites and stores raw menu text as restaurant-level sources.
   - `scraper.py` is the main crawler: it follows menu-like links, probes `/menu`, rejects marketing pages using `menu_score.py`, and persists each accepted page.
   - The scraper has layered fallbacks: structured JSON-LD/menu state mining (`structured_menu.py`), PDF extraction (`pdf_menu.py`), cheap LLM link navigation (`llm_nav.py`), and Playwright-based headless browsing (`headless.py`) for JS-rendered or virtualized menus.

5. **Menu versioning and crawl memory**
   - Every distinct menu capture is written as an immutable **menu version**, while repeated identical crawls are ignored.
   - Successful scrapes also update **crawl profiles** with validated URLs, transport type, size/score, and fingerprints so later recrawls can try the best-known route first.
   - This makes the pipeline adaptive: it prefers previously successful menu paths but can re-discover when a site changes.

6. **Dish classification pipeline**
   - `classify.py` and `classifier.py` convert scraped menu text into structured dish records and vegan verdicts.
   - Classification is **change-aware**: unchanged menus are skipped; changed menus run in delta mode; `--full` forces re-extraction.
   - The model output includes verdict, confidence, evidence, dietary attributes, meal/sides role, ingredients, calories when explicitly stated, and other normalized fields used by search and filtering.

7. **Classification transport and fallbacks**
   - `classification_providers.py` implements a provider chain with three backends: **Claude subscription CLI**, **Codex subscription CLI**, and the **Anthropic API**.
   - The default “auto” mode uses subscription-based providers first and only falls back further when needed; usage-limit cooldowns prevent repeatedly hitting a constrained provider.
   - `classification_exchange.py` supports manual import/export of classification job results when needed.

8. **Enrichment, limits, and quality checks**
   - `enrich.py` adds Google-derived metadata such as vegetarian flag, ratings, rating counts, hours, and other restaurant context used in the consumer UI.
   - `usage_limits.py` powers the Admin subscription-usage panel.
   - `menu_audit.py` scans stored menus for likely false positives or incomplete scrapes and surfaces one-click rescrape actions in Admin.
   - `venue_filter.py` is the visibility boundary that keeps non-food venues, hidden rows, and archived listings out of consumer-facing views.

9. **Backend API and job execution**
   - `api.py` is the Flask JSON API for the local dashboard: it serves SQLite-backed data, triggers discovery/ingest/classify actions, exposes menu/version history, and tracks job progress.
   - Bulk runs and single-item actions execute in the backend process; progress is reconnectable from the browser, but job state is process-local, so restarting the backend stops in-flight work.

10. **Frontend and publishing split**
   - `frontend/` is a **React + Vite** app with two major surfaces: the consumer **Explore/Saved** experience and the local-only **Admin** dashboard.
   - In dev, Vite proxies `/api/*` to the Flask backend; in production, `publish_static.py` exports consumer JSON snapshots into `frontend/public/data/`, and GitHub Pages serves the built static site via `.github/workflows/deploy-pages.yml`.
   - The static build strips admin-only fields and excludes hidden/archived/non-food records, so the published site is safe to host with no backend at all.

11. **Testing and fixtures**
   - `tests/` contains network-free regression tests for the scraper, classification logic, menu history, deletion/visibility rules, and API job behavior.
   - `fixtures/` provides mock discovery data for offline or deterministic runs, which supports local development without live API calls.

Main entry points: [`db.py`](https://github.com/bilbobackends/isitvegan/blob/master/db.py) (SQLite DB), [`api.py`](https://github.com/bilbobackends/isitvegan/blob/master/api.py) (Flask API), [`scraper.py`](https://github.com/bilbobackends/isitvegan/blob/master/scraper.py) (Scraper), [`classify.py`](https://github.com/bilbobackends/isitvegan/blob/master/classify.py) (Classify)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_backend["Backend pipeline"]
  node_config["Config<br/>settings<br/>[config.py]"]
  node_places["Places API<br/>external source<br/>[places_client.py]"]
  node_discover["Discover<br/>[discover.py]"]
  node_add["Add restaurants<br/>operator entrypoint<br/>[add_restaurants.py]"]
  node_ingest["Ingest<br/>scrape driver<br/>[ingest.py]"]
  node_scraper["Scraper<br/>crawler<br/>[scraper.py]"]
  node_score["Menu score<br/>page filter<br/>[menu_score.py]"]
  node_structured["Structured parse<br/>menu extraction<br/>[structured_menu.py]"]
  node_pdf["PDF menu<br/>menu extraction<br/>[pdf_menu.py]"]
  node_llmnav["LLM nav<br/>link navigation<br/>[llm_nav.py]"]
  node_headless["Headless browse<br/>browser crawler<br/>[headless.py]"]
  node_classify["Classify<br/>classification driver<br/>[classify.py]"]
  node_classifier["Dish classifier<br/>extraction engine<br/>[classifier.py]"]
  node_providers["Providers<br/>model chain"]
  node_exchange["Job exchange<br/>import/export"]
  node_enrich["Enrich<br/>[enrich.py]"]
  node_audit["Menu audit<br/>quality check<br/>[menu_audit.py]"]
  node_limits["Usage limits<br/>quota tracking<br/>[usage_limits.py]"]
  node_api["Flask API<br/>admin api<br/>[api.py]"]
end

subgraph group_frontend["Frontend"]
  node_app["App shell<br/>react app<br/>[App.jsx]"]
  node_explore["Explore hub<br/>consumer ui<br/>[ExploreHub.jsx]"]
  node_admin["Admin panel<br/>admin ui<br/>[Admin.jsx]"]
end

subgraph group_data["State and data"]
  node_db[("SQLite DB<br/>state store<br/>[db.py]")]
  node_filter["Venue filter<br/>visibility gate<br/>[venue_filter.py]"]
  node_static["Static data<br/>published snapshots"]
end

subgraph group_delivery["Publish"]
  node_publish["Publish static<br/>exporter<br/>[publish_static.py]"]
end

node_config -->|"API keys"| node_places
node_config -->|"runtime"| node_api
node_places -->|"candidates"| node_discover
node_places -->|"search"| node_add
node_discover -->|"upsert"| node_db
node_add -->|"optional"| node_enrich
node_add -->|"optional"| node_ingest
node_add -->|"optional"| node_classify
node_ingest -->|"drives"| node_scraper
node_scraper -->|"filters"| node_score
node_scraper -->|"parse"| node_structured
node_scraper -->|"fallback"| node_pdf
node_scraper -->|"fallback"| node_llmnav
node_scraper -->|"fallback"| node_headless
node_scraper -->|"store"| node_db
node_structured -->|"menu data"| node_db
node_pdf -->|"menu data"| node_db
node_llmnav -->|"route memory"| node_db
node_headless -->|"captures"| node_db
node_classify -->|"drives"| node_classifier
node_classifier -->|"model calls"| node_providers
node_classifier -->|"writebacks"| node_db
node_classify -->|"tracks"| node_limits
node_exchange -->|"import/export"| node_classify
node_enrich -->|"metadata"| node_places
node_enrich -->|"update"| node_db
node_audit -->|"scan"| node_db
node_audit -->|"rescrape actions"| node_api
node_filter -->|"visibility"| node_db
node_api -->|"serve data"| node_db
node_api -->|"trigger"| node_discover
node_api -->|"trigger"| node_ingest
node_api -->|"trigger"| node_classify
node_api -->|"trigger"| node_audit
node_publish -->|"exports"| node_static
node_publish -->|"applies"| node_filter
node_static -->|"loads"| node_app
node_api -->|"serves"| node_admin
node_static -->|"consumer data"| node_explore
node_api -->|"dev proxy"| node_explore
node_app -->|"routes"| node_explore
node_app -->|"routes"| node_admin

click node_config "https://github.com/bilbobackends/isitvegan/blob/master/config.py"
click node_db "https://github.com/bilbobackends/isitvegan/blob/master/db.py"
click node_places "https://github.com/bilbobackends/isitvegan/blob/master/places_client.py"
click node_discover "https://github.com/bilbobackends/isitvegan/blob/master/discover.py"
click node_add "https://github.com/bilbobackends/isitvegan/blob/master/add_restaurants.py"
click node_ingest "https://github.com/bilbobackends/isitvegan/blob/master/ingest.py"
click node_scraper "https://github.com/bilbobackends/isitvegan/blob/master/scraper.py"
click node_score "https://github.com/bilbobackends/isitvegan/blob/master/menu_score.py"
click node_structured "https://github.com/bilbobackends/isitvegan/blob/master/structured_menu.py"
click node_pdf "https://github.com/bilbobackends/isitvegan/blob/master/pdf_menu.py"
click node_llmnav "https://github.com/bilbobackends/isitvegan/blob/master/llm_nav.py"
click node_headless "https://github.com/bilbobackends/isitvegan/blob/master/headless.py"
click node_classify "https://github.com/bilbobackends/isitvegan/blob/master/classify.py"
click node_classifier "https://github.com/bilbobackends/isitvegan/blob/master/classifier.py"
click node_providers "https://github.com/bilbobackends/isitvegan/blob/master/classification_providers.py"
click node_exchange "https://github.com/bilbobackends/isitvegan/blob/master/classification_exchange.py"
click node_enrich "https://github.com/bilbobackends/isitvegan/blob/master/enrich.py"
click node_audit "https://github.com/bilbobackends/isitvegan/blob/master/menu_audit.py"
click node_limits "https://github.com/bilbobackends/isitvegan/blob/master/usage_limits.py"
click node_filter "https://github.com/bilbobackends/isitvegan/blob/master/venue_filter.py"
click node_api "https://github.com/bilbobackends/isitvegan/blob/master/api.py"
click node_static "https://github.com/bilbobackends/isitvegan/tree/master/frontend/public/data"
click node_publish "https://github.com/bilbobackends/isitvegan/blob/master/publish_static.py"
click node_app "https://github.com/bilbobackends/isitvegan/blob/master/frontend/src/App.jsx"
click node_explore "https://github.com/bilbobackends/isitvegan/blob/master/frontend/src/ExploreHub.jsx"
click node_admin "https://github.com/bilbobackends/isitvegan/blob/master/frontend/src/Admin.jsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config,node_places,node_discover,node_add,node_ingest,node_scraper,node_score,node_structured,node_pdf,node_llmnav,node_headless,node_classify,node_classifier,node_providers,node_exchange,node_enrich,node_audit,node_limits,node_api toneBlue
class node_app,node_explore,node_admin toneAmber
class node_db,node_filter,node_static toneMint
class node_publish toneRose
```

## Components

### Backend pipeline

Local discovery, scraping, classification, and admin API

- [Config](https://github.com/bilbobackends/isitvegan/blob/master/config.py): settings · Runtime settings and secrets boundaries · `config.py`
- [Places API](https://github.com/bilbobackends/isitvegan/blob/master/places_client.py): external source · Google Places lookup client used for discovery and enrichment · `places_client.py`
- [Discover](https://github.com/bilbobackends/isitvegan/blob/master/discover.py): restaurant discovery · Finds and upserts candidate restaurants into SQLite · `discover.py`
- [Add restaurants](https://github.com/bilbobackends/isitvegan/blob/master/add_restaurants.py): operator entrypoint · Confirm-first path for named restaurant additions · `add_restaurants.py`
- [Ingest](https://github.com/bilbobackends/isitvegan/blob/master/ingest.py): scrape driver · Coordinates website scraping and stores raw menu sources · `ingest.py`
- [Scraper](https://github.com/bilbobackends/isitvegan/blob/master/scraper.py): crawler · Main crawler for menu links, probes, and accepted pages · `scraper.py`
- [Menu score](https://github.com/bilbobackends/isitvegan/blob/master/menu_score.py): page filter · Rejects marketing pages and low-quality menu candidates · `menu_score.py`
- [Structured parse](https://github.com/bilbobackends/isitvegan/blob/master/structured_menu.py): menu extraction · Extracts menu data from JSON-LD and embedded state · `structured_menu.py`
- [PDF menu](https://github.com/bilbobackends/isitvegan/blob/master/pdf_menu.py): menu extraction · Handles PDF-based menu extraction · `pdf_menu.py`
- [LLM nav](https://github.com/bilbobackends/isitvegan/blob/master/llm_nav.py): link navigation · Cheap language-model-assisted menu link discovery · `llm_nav.py`
- [Headless browse](https://github.com/bilbobackends/isitvegan/blob/master/headless.py): browser crawler · Playwright fallback for JS-rendered or virtualized menus · `headless.py`
- [Classify](https://github.com/bilbobackends/isitvegan/blob/master/classify.py): classification driver · Runs change-aware dish classification over scraped menus · `classify.py`
- [Dish classifier](https://github.com/bilbobackends/isitvegan/blob/master/classifier.py): extraction engine · Converts menu text into structured dishes and vegan verdicts · `classifier.py`
- [Providers](https://github.com/bilbobackends/isitvegan/blob/master/classification_providers.py): model chain · Claude CLI, Codex CLI, and Anthropic API backend chain · `classification_providers.py`
- [Job exchange](https://github.com/bilbobackends/isitvegan/blob/master/classification_exchange.py): import/export · Manual import and export of classification job results · `classification_exchange.py`
- [Enrich](https://github.com/bilbobackends/isitvegan/blob/master/enrich.py): metadata enrichment · Adds Google-derived restaurant context for the UI · `enrich.py`
- [Menu audit](https://github.com/bilbobackends/isitvegan/blob/master/menu_audit.py): quality check · Finds likely false positives and incomplete scrapes · `menu_audit.py`
- [Usage limits](https://github.com/bilbobackends/isitvegan/blob/master/usage_limits.py): quota tracking · Tracks subscription usage and cooldowns · `usage_limits.py`
- [Flask API](https://github.com/bilbobackends/isitvegan/blob/master/api.py): admin api · Local JSON API for data, actions, and reconnectable job progress · `api.py`

### Frontend

React consumer site plus local admin surface

- [App shell](https://github.com/bilbobackends/isitvegan/blob/master/frontend/src/App.jsx): react app · Frontend entrypoint that routes consumer and admin surfaces · `frontend/src/App.jsx`
- [Explore hub](https://github.com/bilbobackends/isitvegan/blob/master/frontend/src/ExploreHub.jsx): consumer ui · Consumer browse, saved items, filters, and dish details · `frontend/src/ExploreHub.jsx`
- [Admin panel](https://github.com/bilbobackends/isitvegan/blob/master/frontend/src/Admin.jsx): admin ui · Local-only operational dashboard for discovery, scraping, and review · `frontend/src/Admin.jsx`

### State and data

Persistent store and exported static snapshots

- [SQLite DB](https://github.com/bilbobackends/isitvegan/blob/master/db.py): state store · Central persistent state for restaurants, menus, dishes, runs, and jobs · `db.py`
- [Venue filter](https://github.com/bilbobackends/isitvegan/blob/master/venue_filter.py): visibility gate · Hides non-food, hidden, and archived rows from consumers · `venue_filter.py`
- [Static data](https://github.com/bilbobackends/isitvegan/tree/master/frontend/public/data): published snapshots · Consumer-ready JSON exported for static hosting · `frontend/public/data`

### Publish

Static site build and deployment

- [Publish static](https://github.com/bilbobackends/isitvegan/blob/master/publish_static.py): exporter · Builds public snapshots and prepares the static site · `publish_static.py`

## Connections

- Config → Places API: API keys · Provides credentials and runtime settings
- Config → Flask API: runtime · Configures backend behavior and endpoints
- Places API → Discover: candidates · Feeds restaurant discovery results
- Places API → Add restaurants: search · Resolves named restaurant queries to candidates
- Discover → SQLite DB: upsert · Stores discovered restaurants and metadata
- Add restaurants → Enrich: optional · Can trigger enrichment after confirmation
- Add restaurants → Ingest: optional · Can launch scraping for selected places
- Add restaurants → Classify: optional · Can launch classification after ingestion
- Ingest → Scraper: drives · Orchestrates crawl and capture flow
- Scraper → Menu score: filters · Rejects non-menu pages and weak candidates
- Scraper → Structured parse: parse · Uses structured signals when available
- Scraper → PDF menu: fallback · Extracts menus from PDF sources
- Scraper → LLM nav: fallback · Uses lightweight LLM-based link navigation
- Scraper → Headless browse: fallback · Falls back to browser automation for JS-heavy sites
- Scraper → SQLite DB: store · Persists accepted menu pages and crawl memory
- Structured parse → SQLite DB: menu data · Contributes parsed menu content to storage
- PDF menu → SQLite DB: menu data · Contributes extracted PDF menu content
- LLM nav → SQLite DB: route memory · Can persist successful navigation outcomes
- Headless browse → SQLite DB: captures · Stores rendered menu captures and fingerprints
- Classify → Dish classifier: drives · Coordinates change-aware extraction runs
- Dish classifier → Providers: model calls · Delegates to the provider chain for verdicts
- Dish classifier → SQLite DB: writebacks · Stores dish records, verdicts, and history
- Classify → Usage limits: tracks · Consumes and records provider usage limits
- Job exchange → Classify: import/export · Moves job results in and out of the pipeline
- Enrich → Places API: metadata · Pulls Google-derived restaurant context
- Enrich → SQLite DB: update · Stores enrichment fields on restaurants
- Menu audit → SQLite DB: scan · Reads stored menus and flags suspect captures
- Menu audit → Flask API: rescrape actions · Surfaces review and recovery actions in admin
- Venue filter → SQLite DB: visibility · Enforces consumer-facing inclusion rules
- Flask API → SQLite DB: serve data · Reads and mutates SQLite-backed state
- Flask API → Discover: trigger · Starts discovery jobs from the admin UI
- Flask API → Ingest: trigger · Starts scraping jobs from the admin UI
- Flask API → Classify: trigger · Starts classification jobs from the admin UI
- Flask API → Menu audit: trigger · Starts audit and recovery actions
- Publish static → Static data: exports · Writes consumer JSON snapshots for static hosting
- Publish static → Venue filter: applies · Uses visibility rules before publishing
- Static data → App shell: loads · Feeds the consumer UI from static snapshots
- Flask API → Admin panel: serves · Supports the local-only admin dashboard
- Static data → Explore hub: consumer data · Drives browse and saved experiences
- Flask API → Explore hub: dev proxy · Supplies live data during local development
- App shell → Explore hub: routes · Main consumer experience
- App shell → Admin panel: routes · Local operator experience
