How bilbobackends/isitvegan works
1. System shape
- This repo is a two-plane application: a local Python pipeline + Flask admin API for discovery/scraping/classification, and a consumer React frontend that is also exported as a fully static site for GitHub Pages.
- The public site never talks to external services or holds credentials; only the local backend does.
2. Persistent store
- The central state is a SQLite database (db.py) that holds restaurants, menu sources, dish records, classification history, crawl profiles, menu versions, dish changes, hidden/archived flags, and job state.
- The architecture is intentionally history-oriented: menu snapshots and classification runs are preserved so the UI can show change over time instead of only the latest state.
3. Restaurant discovery
- Discovery starts with Google Places API via places_client.py and discover.py, which finds candidate restaurants in a configured area and upserts them into SQLite.
- add_restaurants.py is the confirm-first path for named additions: it resolves text queries to Places candidates, lets the operator pick the exact match, then optionally runs enrichment, scraping, and classification.
4. Menu ingestion pipeline
- ingest.py drives scraping of restaurant websites and stores raw menu text as restaurant-level sources.
- scraper.py is the main crawler: it follows menu-like links, probes /menu, rejects marketing pages using menu_score.py, and persists each accepted page.
- The scraper has layered fallbacks: structured JSON-LD/menu state mining (structured_menu.py), PDF extraction (pdf_menu.py), cheap LLM link navigation (llm_nav.py), and Playwright-based headless browsing (headless.py) for JS-rendered or virtualized menus.
5. Menu versioning and crawl memory
- Every distinct menu capture is written as an immutable menu version, while repeated identical crawls are ignored.
- Successful scrapes also update crawl profiles with validated URLs, transport type, size/score, and fingerprints so later recrawls can try the best-known route first.
- This makes the pipeline adaptive: it prefers previously successful menu paths but can re-discover when a site changes.
6. Dish classification pipeline
- classify.py and classifier.py convert scraped menu text into structured dish records and vegan verdicts.
- Classification is change-aware: unchanged menus are skipped; changed menus run in delta mode; --full forces re-extraction.
- The model output includes verdict, confidence, evidence, dietary attributes, meal/sides role, ingredients, calories when explicitly stated, and other normalized fields used by search and filtering.
7. Classification transport and fallbacks
- classification_providers.py implements a provider chain with three backends: Claude subscription CLI, Codex subscription CLI, and the Anthropic API.
- The default “auto” mode uses subscription-based providers first and only falls back further when needed; usage-limit cooldowns prevent repeatedly hitting a constrained provider.
- classification_exchange.py supports manual import/export of classification job results when needed.
8. Enrichment, limits, and quality checks
- enrich.py adds Google-derived metadata such as vegetarian flag, ratings, rating counts, hours, and other restaurant context used in the consumer UI.
- usage_limits.py powers the Admin subscription-usage panel.
- menu_audit.py scans stored menus for likely false positives or incomplete scrapes and surfaces one-click rescrape actions in Admin.
- venue_filter.py is the visibility boundary that keeps non-food venues, hidden rows, and archived listings out of consumer-facing views.
9. Backend API and job execution
- api.py is the Flask JSON API for the local dashboard: it serves SQLite-backed data, triggers discovery/ingest/classify actions, exposes menu/version history, and tracks job progress.
- Bulk runs and single-item actions execute in the backend process; progress is reconnectable from the browser, but job state is process-local, so restarting the backend stops in-flight work.
10. Frontend and publishing split
- frontend/ is a React + Vite app with two major surfaces: the consumer Explore/Saved experience and the local-only Admin dashboard.
- In dev, Vite proxies /api/* to the Flask backend; in production, publish_static.py exports consumer JSON snapshots into frontend/public/data/, and GitHub Pages serves the built static site via .github/workflows/deploy-pages.yml.
- The static build strips admin-only fields and excludes hidden/archived/non-food records, so the published site is safe to host with no backend at all.
11. Testing and fixtures
- tests/ contains network-free regression tests for the scraper, classification logic, menu history, deletion/visibility rules, and API job behavior.
- fixtures/ provides mock discovery data for offline or deterministic runs, which supports local development without live API calls.
Main entry points: db.py SQLite DB · api.py Flask API · scraper.py Scraper · classify.py Classify
Components 26
Backend pipeline
Local discovery, scraping, classification, and admin API
- Config · Runtime settings and secrets boundaries
config.py - Places API · Google Places lookup client used for discovery and enrichment
places_client.py - Discover · Finds and upserts candidate restaurants into SQLite
discover.py - Add restaurants · Confirm-first path for named restaurant additions
add_restaurants.py - Ingest · Coordinates website scraping and stores raw menu sources
ingest.py - Scraper · Main crawler for menu links, probes, and accepted pages
scraper.py - Menu score · Rejects marketing pages and low-quality menu candidates
menu_score.py - Structured parse · Extracts menu data from JSON-LD and embedded state
structured_menu.py - PDF menu · Handles PDF-based menu extraction
pdf_menu.py - LLM nav · Cheap language-model-assisted menu link discovery
llm_nav.py - Headless browse · Playwright fallback for JS-rendered or virtualized menus
headless.py - Classify · Runs change-aware dish classification over scraped menus
classify.py - Dish classifier · Converts menu text into structured dishes and vegan verdicts
classifier.py - Providers · Claude CLI, Codex CLI, and Anthropic API backend chain
classification_providers.py - Job exchange · Manual import and export of classification job results
classification_exchange.py - Enrich · Adds Google-derived restaurant context for the UI
enrich.py - Menu audit · Finds likely false positives and incomplete scrapes
menu_audit.py - Usage limits · Tracks subscription usage and cooldowns
usage_limits.py - Flask API · Local JSON API for data, actions, and reconnectable job progress
api.py
Frontend
React consumer site plus local admin surface
- App shell · Frontend entrypoint that routes consumer and admin surfaces
frontend/src/App.jsx - Explore hub · Consumer browse, saved items, filters, and dish details
frontend/src/ExploreHub.jsx - Admin panel · Local-only operational dashboard for discovery, scraping, and review
frontend/src/Admin.jsx
State and data
Persistent store and exported static snapshots
- SQLite DB · Central persistent state for restaurants, menus, dishes, runs, and jobs
db.py - Venue filter · Hides non-food, hidden, and archived rows from consumers
venue_filter.py - Static data · Consumer-ready JSON exported for static hosting
frontend/public/data
Publish
Static site build and deployment
- Publish static · Builds public snapshots and prepares the static site
publish_static.py
Connections 42
- Config → Places API: API keys · Provides credentials and runtime settings
- Config → Flask API: runtime · Configures backend behavior and endpoints
- Places API → Discover: candidates · Feeds restaurant discovery results
- Places API → Add restaurants: search · Resolves named restaurant queries to candidates
- Discover → SQLite DB: upsert · Stores discovered restaurants and metadata
- Add restaurants → Enrich: optional · Can trigger enrichment after confirmation
- Add restaurants → Ingest: optional · Can launch scraping for selected places
- Add restaurants → Classify: optional · Can launch classification after ingestion
- Ingest → Scraper: drives · Orchestrates crawl and capture flow
- Scraper → Menu score: filters · Rejects non-menu pages and weak candidates
- Scraper → Structured parse: parse · Uses structured signals when available
- Scraper → PDF menu: fallback · Extracts menus from PDF sources
- Scraper → LLM nav: fallback · Uses lightweight LLM-based link navigation
- Scraper → Headless browse: fallback · Falls back to browser automation for JS-heavy sites
- Scraper → SQLite DB: store · Persists accepted menu pages and crawl memory
- Structured parse → SQLite DB: menu data · Contributes parsed menu content to storage
- PDF menu → SQLite DB: menu data · Contributes extracted PDF menu content
- LLM nav → SQLite DB: route memory · Can persist successful navigation outcomes
- Headless browse → SQLite DB: captures · Stores rendered menu captures and fingerprints
- Classify → Dish classifier: drives · Coordinates change-aware extraction runs
- Dish classifier → Providers: model calls · Delegates to the provider chain for verdicts
- Dish classifier → SQLite DB: writebacks · Stores dish records, verdicts, and history
- Classify → Usage limits: tracks · Consumes and records provider usage limits
- Job exchange → Classify: import/export · Moves job results in and out of the pipeline
- Enrich → Places API: metadata · Pulls Google-derived restaurant context
- Enrich → SQLite DB: update · Stores enrichment fields on restaurants
- Menu audit → SQLite DB: scan · Reads stored menus and flags suspect captures
- Menu audit → Flask API: rescrape actions · Surfaces review and recovery actions in admin
- Venue filter → SQLite DB: visibility · Enforces consumer-facing inclusion rules
- Flask API → SQLite DB: serve data · Reads and mutates SQLite-backed state
- Flask API → Discover: trigger · Starts discovery jobs from the admin UI
- Flask API → Ingest: trigger · Starts scraping jobs from the admin UI
- Flask API → Classify: trigger · Starts classification jobs from the admin UI
- Flask API → Menu audit: trigger · Starts audit and recovery actions
- Publish static → Static data: exports · Writes consumer JSON snapshots for static hosting
- Publish static → Venue filter: applies · Uses visibility rules before publishing
- Static data → App shell: loads · Feeds the consumer UI from static snapshots
- Flask API → Admin panel: serves · Supports the local-only admin dashboard
- Static data → Explore hub: consumer data · Drives browse and saved experiences
- Flask API → Explore hub: dev proxy · Supplies live data during local development
- App shell → Explore hub: routes · Main consumer experience
- App shell → Admin panel: routes · Local operator experience