# ethanmckanna/robotaxi-plate-scraper architecture

> Architecture diagram of the GitHub repository ethanmckanna/robotaxi-plate-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ethanmckanna/robotaxi-plate-scraper
- Repository: https://github.com/ethanmckanna/robotaxi-plate-scraper

## Overview

1. **Purpose and runtime**
   - This is a TypeScript/Node.js scraping pipeline that runs periodically, primarily through **GitHub Actions** on an hourly schedule.
   - Its job is to find robotaxi license plate evidence, process the images, upload them, and create moderation records in a **Supabase-backed tracker**.

2. **Top-level orchestration**
   - `src/index.ts` is the main pipeline coordinator.
   - It ties together configuration, provider scrapers, vision analysis, uploads, and database writes into one end-to-end run.
   - The entrypoint is also used by the one-off local/script execution path.

3. **Provider abstraction**
   - `src/scrapers/` contains the source-specific ingestion layer.
   - Providers are registered through `src/scrapers/registry.ts`, with shared types in `src/scrapers/types.ts`.
   - Current concrete providers include **HTTP-based scraping** and **Reddit**, showing the system is designed to ingest from multiple external content sources without changing the rest of the pipeline.

4. **Scraper boundary**
   - Each scraper is responsible only for discovering candidate content from its source and normalizing it into a common shape.
   - That isolates source-specific logic from downstream stages, so new providers can be added by dropping in a scraper and registering it.

5. **Vision / detection pipeline**
   - `src/vision/` is the image-analysis layer.
   - It includes a detector, a Gemini integration, and a plate-extractor, suggesting the pipeline first identifies likely robotaxi imagery and then extracts the license-plate evidence needed for moderation.
   - This is the main AI-dependent boundary in the system.

6. **Storage of evidence**
   - `src/storage/uploader.ts` handles image upload.
   - The architecture implies evidence is persisted outside the scraper process before moderation submission is created, so the tracker can reference durable uploaded assets rather than transient source URLs alone.

7. **Database responsibilities**
   - `src/database/` is the Supabase access layer.
   - `client.ts` centralizes connectivity, while `submissions.ts` and `tracking.ts` separate the moderation-submission record flow from the operational tracking flow.
   - This is where the pipeline records what was found, what was uploaded, and what still needs moderation attention.

8. **Configuration and setup**
   - `src/config/` contains environment loading and search-term configuration, including an interactive setup path and provider/search-term defaults.
   - This makes the crawler behavior largely configuration-driven, with secrets such as Gemini and Supabase credentials provided through environment variables.

9. **Operational scripts**
   - `scripts/run-once.ts` is the one-shot execution path used for local runs or manual execution.
   - `scripts/setup.ts` and `scripts/seed-bot-user.ts` support initial environment bootstrap and creating the bot identity used in the tracker.

10. **Workflow and deployment model**
   - `.github/workflows/scrape.yml` is the scheduling and execution boundary.
   - The workflow depends on repo secrets for **GEMINI_API_KEY**, **SUPABASE_URL**, **SUPABASE_SERVICE_ROLE_KEY**, and **BOT_USER_ID**, which are the main external dependencies needed for unattended execution.

11. **Utility layer**
   - `src/utils/` contains shared concerns like delay, logging, and validation.
   - These are cross-cutting helpers rather than business logic, supporting rate control, observability, and input sanity checks across scrapers and pipeline stages.

12. **End-to-end data flow**
   - The typical flow is: scheduled run starts → provider scrapers collect candidate posts/pages → vision logic filters and extracts plate evidence → evidence is uploaded → Supabase records are created/updated for tracking and moderation.
   - The clear architectural boundaries are: source ingestion, AI analysis, durable asset storage, and tracker persistence.

Main entry points: [`src/index.ts`](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/index.ts) (Main pipeline), [`src/config/index.ts`](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/index.ts) (Config load), [`src/database/client.ts`](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/client.ts) (Supabase client), [`src/scrapers/registry.ts`](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/registry.ts) (Scraper registry)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_runtime["Runtime & Orchestration"]
  node_node_workflow["Scrape workflow<br/>github action<br/>[scrape.yml]"]
  node_node_index["Main pipeline<br/>orchestrator<br/>[index.ts]"]
  node_node_run_once["Run once<br/>entry script<br/>[run-once.ts]"]
end

subgraph group_group_ingest["Source Ingestion"]
  node_node_registry["Scraper registry<br/>provider registry<br/>[registry.ts]"]
  node_node_scraper_types["Scraper types<br/>shared contract<br/>[types.ts]"]
  node_node_http_scraper["HTTP scraper<br/>source scraper<br/>[http.ts]"]
  node_node_reddit_scraper["Reddit scraper<br/>source scraper<br/>[reddit.ts]"]
end

subgraph group_group_vision["Vision Pipeline"]
  node_node_detector["Detector<br/>image filter<br/>[detector.ts]"]
  node_node_gemini["Gemini vision<br/>ai model<br/>[gemini.ts]"]
  node_node_plate_extractor["Plate extractor<br/>evidence extractor<br/>[plate-extractor.ts]"]
end

subgraph group_group_storage["Evidence Storage"]
  node_node_uploader["Uploader<br/>storage adapter<br/>[uploader.ts]"]
end

subgraph group_group_db["Tracker Database"]
  node_node_db_client["Supabase client<br/>db client<br/>[client.ts]"]
  node_node_submissions["Submissions<br/>moderation writes<br/>[submissions.ts]"]
  node_node_tracking["Tracking<br/>operational writes<br/>[tracking.ts]"]
end

subgraph group_group_config["Configuration"]
  node_node_config_index["Config load<br/>env config<br/>[index.ts]"]
  node_node_search_terms["Search terms<br/>term config<br/>[search-terms.ts]"]
  node_node_interactive_setup["Setup flow<br/>interactive config"]
end

subgraph group_group_ops["Operations"]
  node_node_seed_bot["Seed bot user<br/>bootstrap script<br/>[seed-bot-user.ts]"]
  node_node_setup["Setup script<br/>bootstrap script<br/>[setup.ts]"]
end

subgraph group_group_utils["Shared Utilities"]
  node_node_logger["Logger<br/>[logger.ts]"]
  node_node_delay["Delay<br/>[delay.ts]"]
  node_node_validation["Validation<br/>[validation.ts]"]
end

node_node_workflow -->|"runs"| node_node_index
node_node_run_once -->|"invokes"| node_node_index
node_node_index -->|"loads"| node_node_config_index
node_node_index -->|"discovers"| node_node_registry
node_node_registry -->|"includes"| node_node_http_scraper
node_node_registry -->|"includes"| node_node_reddit_scraper
node_node_http_scraper -->|"normalizes to"| node_node_scraper_types
node_node_reddit_scraper -->|"normalizes to"| node_node_scraper_types
node_node_index -->|"analyzes"| node_node_detector
node_node_detector -->|"refines with"| node_node_gemini
node_node_gemini -->|"supports"| node_node_plate_extractor
node_node_index -->|"uploads"| node_node_uploader
node_node_uploader -->|"references"| node_node_db_client
node_node_index -->|"writes"| node_node_submissions
node_node_index -->|"updates"| node_node_tracking
node_node_submissions -->|"uses"| node_node_db_client
node_node_tracking -->|"uses"| node_node_db_client
node_node_config_index -->|"includes"| node_node_search_terms
node_node_config_index -->|"supports"| node_node_interactive_setup
node_node_index -->|"logs via"| node_node_logger
node_node_index -->|"paces with"| node_node_delay
node_node_index -->|"validates with"| node_node_validation
node_node_setup -->|"bootstraps"| node_node_config_index
node_node_seed_bot -->|"provisions"| node_node_db_client

click node_node_workflow "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/.github/workflows/scrape.yml"
click node_node_index "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/index.ts"
click node_node_run_once "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/scripts/run-once.ts"
click node_node_config_index "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/index.ts"
click node_node_search_terms "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/search-terms.ts"
click node_node_interactive_setup "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/interactive-setup.ts"
click node_node_registry "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/registry.ts"
click node_node_scraper_types "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/types.ts"
click node_node_http_scraper "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/http.ts"
click node_node_reddit_scraper "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/reddit.ts"
click node_node_detector "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/vision/detector.ts"
click node_node_gemini "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/vision/gemini.ts"
click node_node_plate_extractor "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/vision/plate-extractor.ts"
click node_node_uploader "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/storage/uploader.ts"
click node_node_db_client "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/client.ts"
click node_node_submissions "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/submissions.ts"
click node_node_tracking "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/tracking.ts"
click node_node_seed_bot "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/scripts/seed-bot-user.ts"
click node_node_setup "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/scripts/setup.ts"
click node_node_logger "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/utils/logger.ts"
click node_node_delay "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/utils/delay.ts"
click node_node_validation "https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/utils/validation.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_workflow,node_node_index,node_node_run_once,node_node_seed_bot,node_node_setup toneBlue
class node_node_registry,node_node_scraper_types,node_node_http_scraper,node_node_reddit_scraper,node_node_logger,node_node_delay,node_node_validation toneAmber
class node_node_detector,node_node_gemini,node_node_plate_extractor toneMint
class node_node_uploader toneRose
class node_node_db_client,node_node_submissions,node_node_tracking toneIndigo
class node_node_config_index,node_node_search_terms,node_node_interactive_setup toneTeal
```

## Components

### Runtime & Orchestration

Scheduled and one-off execution entrypoints that coordinate the pipeline.

- [Scrape workflow](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/.github/workflows/scrape.yml): github action · Hourly scheduled automation that runs the pipeline with secrets. · `.github/workflows/scrape.yml`
- [Main pipeline](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/index.ts): orchestrator · Coordinates the end-to-end scrape, analysis, upload, and persistence flow. · `src/index.ts`
- [Run once](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/scripts/run-once.ts): entry script · One-off local/manual execution path for the pipeline. · `scripts/run-once.ts`

### Source Ingestion

Provider-specific scrapers and registry for normalized discovery of candidate content.

- [Scraper registry](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/registry.ts): provider registry · Registers available source scrapers behind a common interface. · `src/scrapers/registry.ts`
- [Scraper types](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/types.ts): shared contract · Common normalized types used by all provider scrapers. · `src/scrapers/types.ts`
- [HTTP scraper](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/http.ts): source scraper · HTTP-based provider that discovers candidate content from a web source. · `src/scrapers/http.ts`
- [Reddit scraper](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/scrapers/reddit.ts): source scraper · Reddit ingestion provider for discovering candidate posts and media. · `src/scrapers/reddit.ts`

### Vision Pipeline

Image analysis and plate evidence extraction.

- [Detector](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/vision/detector.ts): image filter · First-pass vision stage that identifies likely robotaxi imagery. · `src/vision/detector.ts`
- [Gemini vision](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/vision/gemini.ts): ai model · LLM/vision integration used for deeper image understanding. · `src/vision/gemini.ts`
- [Plate extractor](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/vision/plate-extractor.ts): evidence extractor · Extracts license-plate evidence from analyzed imagery. · `src/vision/plate-extractor.ts`

### Evidence Storage

Durable upload boundary for discovered media.

- [Uploader](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/storage/uploader.ts): storage adapter · Uploads evidence media to durable external storage. · `src/storage/uploader.ts`

### Tracker Database

Supabase-backed persistence for submissions and operational tracking.

- [Supabase client](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/client.ts): db client · Shared Supabase connectivity and client setup. · `src/database/client.ts`
- [Submissions](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/submissions.ts): moderation writes · Creates and updates moderation submission records. · `src/database/submissions.ts`
- [Tracking](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/database/tracking.ts): operational writes · Maintains operational tracking state for discovered evidence. · `src/database/tracking.ts`

### Configuration

Environment loading and search-term/provider setup.

- [Config load](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/index.ts): env config · Central environment and runtime configuration loading. · `src/config/index.ts`
- [Search terms](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/search-terms.ts): term config · Provider and query defaults that shape discovery behavior. · `src/config/search-terms.ts`
- [Setup flow](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/config/interactive-setup.ts): interactive config · Interactive bootstrap path for creating local configuration. · `src/config/interactive-setup.ts`

### Operations

Manual bootstrap and seed scripts for local or initial setup.

- [Seed bot user](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/scripts/seed-bot-user.ts): bootstrap script · Creates the bot identity required by the tracker. · `scripts/seed-bot-user.ts`
- [Setup script](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/scripts/setup.ts): bootstrap script · Initial environment/setup helper for local or first-run configuration. · `scripts/setup.ts`

### Shared Utilities

Cross-cutting helpers used across the pipeline.

- [Logger](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/utils/logger.ts): utility · Shared logging helper for pipeline observability. · `src/utils/logger.ts`
- [Delay](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/utils/delay.ts): utility · Shared timing helper for rate control and pacing. · `src/utils/delay.ts`
- [Validation](https://github.com/ethanmckanna/robotaxi-plate-scraper/blob/main/src/utils/validation.ts): utility · Input and payload validation helpers used across stages. · `src/utils/validation.ts`

## Connections

- Scrape workflow → Main pipeline: runs · The scheduled GitHub Action triggers the main pipeline.
- Run once → Main pipeline: invokes · Manual execution path shares the same orchestrator.
- Main pipeline → Config load: loads · Pipeline starts by loading runtime configuration.
- Main pipeline → Scraper registry: discovers · Orchestrator selects available scrapers through the registry.
- Scraper registry → HTTP scraper: includes · HTTP provider is one registered source.
- Scraper registry → Reddit scraper: includes · Reddit provider is one registered source.
- HTTP scraper → Scraper types: normalizes to · Provider output conforms to shared scraper types.
- Reddit scraper → Scraper types: normalizes to · Provider output conforms to shared scraper types.
- Main pipeline → Detector: analyzes · Discovered media is first filtered by the detector.
- Detector → Gemini vision: refines with · Detector feeds AI vision analysis for stronger identification.
- Gemini vision → Plate extractor: supports · Vision output is converted into plate evidence.
- Main pipeline → Uploader: uploads · Extracted evidence is persisted in external storage.
- Uploader → Supabase client: references · Uploaded assets are associated with tracker records.
- Main pipeline → Submissions: writes · Pipeline creates moderation submission records.
- Main pipeline → Tracking: updates · Pipeline records operational tracking state.
- Submissions → Supabase client: uses · Submission writes go through the shared Supabase client.
- Tracking → Supabase client: uses · Tracking writes go through the shared Supabase client.
- Config load → Search terms: includes · Configuration includes search-term/provider defaults.
- Config load → Setup flow: supports · Interactive setup populates local configuration.
- Main pipeline → Logger: logs via · Orchestrator and stages share logging helpers.
- Main pipeline → Delay: paces with · Pipeline uses delay helpers for throttling and retries.
- Main pipeline → Validation: validates with · Pipeline uses validation helpers before downstream actions.
- Setup script → Config load: bootstraps · Setup script prepares configuration values.
- Seed bot user → Supabase client: provisions · Bot user seeding requires database connectivity.