How ethanmckanna/robotaxi-plate-scraper works
1. Purpose and runtime
- This is a TypeScript/Node.js scraping pipeline that runs periodically, primarily through GitHub Actions on an hourly schedule.
- Its job is to find robotaxi license plate evidence, process the images, upload them, and create moderation records in a Supabase-backed tracker.
2. Top-level orchestration
- src/index.ts is the main pipeline coordinator.
- It ties together configuration, provider scrapers, vision analysis, uploads, and database writes into one end-to-end run.
- The entrypoint is also used by the one-off local/script execution path.
3. Provider abstraction
- src/scrapers/ contains the source-specific ingestion layer.
- Providers are registered through src/scrapers/registry.ts, with shared types in src/scrapers/types.ts.
- Current concrete providers include HTTP-based scraping and Reddit, showing the system is designed to ingest from multiple external content sources without changing the rest of the pipeline.
4. Scraper boundary
- Each scraper is responsible only for discovering candidate content from its source and normalizing it into a common shape.
- That isolates source-specific logic from downstream stages, so new providers can be added by dropping in a scraper and registering it.
5. Vision / detection pipeline
- src/vision/ is the image-analysis layer.
- It includes a detector, a Gemini integration, and a plate-extractor, suggesting the pipeline first identifies likely robotaxi imagery and then extracts the license-plate evidence needed for moderation.
- This is the main AI-dependent boundary in the system.
6. Storage of evidence
- src/storage/uploader.ts handles image upload.
- The architecture implies evidence is persisted outside the scraper process before moderation submission is created, so the tracker can reference durable uploaded assets rather than transient source URLs alone.
7. Database responsibilities
- src/database/ is the Supabase access layer.
- client.ts centralizes connectivity, while submissions.ts and tracking.ts separate the moderation-submission record flow from the operational tracking flow.
- This is where the pipeline records what was found, what was uploaded, and what still needs moderation attention.
8. Configuration and setup
- src/config/ contains environment loading and search-term configuration, including an interactive setup path and provider/search-term defaults.
- This makes the crawler behavior largely configuration-driven, with secrets such as Gemini and Supabase credentials provided through environment variables.
9. Operational scripts
- scripts/run-once.ts is the one-shot execution path used for local runs or manual execution.
- scripts/setup.ts and scripts/seed-bot-user.ts support initial environment bootstrap and creating the bot identity used in the tracker.
10. Workflow and deployment model
- .github/workflows/scrape.yml is the scheduling and execution boundary.
- The workflow depends on repo secrets for GEMINI_API_KEY, SUPABASE_URL, SUPABASE_SERVICE_ROLE_KEY, and BOT_USER_ID, which are the main external dependencies needed for unattended execution.
11. Utility layer
- src/utils/ contains shared concerns like delay, logging, and validation.
- These are cross-cutting helpers rather than business logic, supporting rate control, observability, and input sanity checks across scrapers and pipeline stages.
12. End-to-end data flow
- The typical flow is: scheduled run starts → provider scrapers collect candidate posts/pages → vision logic filters and extracts plate evidence → evidence is uploaded → Supabase records are created/updated for tracking and moderation.
- The clear architectural boundaries are: source ingestion, AI analysis, durable asset storage, and tracker persistence.
Main entry points: src/index.ts Main pipeline · config/index.ts Config load · client.ts Supabase client · registry.ts Scraper registry
Components 22
Runtime & Orchestration
Scheduled and one-off execution entrypoints that coordinate the pipeline.
- Scrape workflow · Hourly scheduled automation that runs the pipeline with secrets.
.github/workflows/scrape.yml - Main pipeline · Coordinates the end-to-end scrape, analysis, upload, and persistence flow.
src/index.ts - Run once · One-off local/manual execution path for the pipeline.
scripts/run-once.ts
Source Ingestion
Provider-specific scrapers and registry for normalized discovery of candidate content.
- Scraper registry · Registers available source scrapers behind a common interface.
src/scrapers/registry.ts - Scraper types · Common normalized types used by all provider scrapers.
src/scrapers/types.ts - HTTP scraper · HTTP-based provider that discovers candidate content from a web source.
src/scrapers/http.ts - Reddit scraper · Reddit ingestion provider for discovering candidate posts and media.
src/scrapers/reddit.ts
Vision Pipeline
Image analysis and plate evidence extraction.
- Detector · First-pass vision stage that identifies likely robotaxi imagery.
src/vision/detector.ts - Gemini vision · LLM/vision integration used for deeper image understanding.
src/vision/gemini.ts - Plate extractor · Extracts license-plate evidence from analyzed imagery.
src/vision/plate-extractor.ts
Evidence Storage
Durable upload boundary for discovered media.
- Uploader · Uploads evidence media to durable external storage.
src/storage/uploader.ts
Tracker Database
Supabase-backed persistence for submissions and operational tracking.
- Supabase client · Shared Supabase connectivity and client setup.
src/database/client.ts - Submissions · Creates and updates moderation submission records.
src/database/submissions.ts - Tracking · Maintains operational tracking state for discovered evidence.
src/database/tracking.ts
Configuration
Environment loading and search-term/provider setup.
- Config load · Central environment and runtime configuration loading.
src/config/index.ts - Search terms · Provider and query defaults that shape discovery behavior.
src/config/search-terms.ts - Setup flow · Interactive bootstrap path for creating local configuration.
src/config/interactive-setup.ts
Operations
Manual bootstrap and seed scripts for local or initial setup.
- Seed bot user · Creates the bot identity required by the tracker.
scripts/seed-bot-user.ts - Setup script · Initial environment/setup helper for local or first-run configuration.
scripts/setup.ts
Shared Utilities
Cross-cutting helpers used across the pipeline.
- Logger · Shared logging helper for pipeline observability.
src/utils/logger.ts - Delay · Shared timing helper for rate control and pacing.
src/utils/delay.ts - Validation · Input and payload validation helpers used across stages.
src/utils/validation.ts
Connections 24
- Scrape workflow → Main pipeline: runs · The scheduled GitHub Action triggers the main pipeline.
- Run once → Main pipeline: invokes · Manual execution path shares the same orchestrator.
- Main pipeline → Config load: loads · Pipeline starts by loading runtime configuration.
- Main pipeline → Scraper registry: discovers · Orchestrator selects available scrapers through the registry.
- Scraper registry → HTTP scraper: includes · HTTP provider is one registered source.
- Scraper registry → Reddit scraper: includes · Reddit provider is one registered source.
- HTTP scraper → Scraper types: normalizes to · Provider output conforms to shared scraper types.
- Reddit scraper → Scraper types: normalizes to · Provider output conforms to shared scraper types.
- Main pipeline → Detector: analyzes · Discovered media is first filtered by the detector.
- Detector → Gemini vision: refines with · Detector feeds AI vision analysis for stronger identification.
- Gemini vision → Plate extractor: supports · Vision output is converted into plate evidence.
- Main pipeline → Uploader: uploads · Extracted evidence is persisted in external storage.
- Uploader → Supabase client: references · Uploaded assets are associated with tracker records.
- Main pipeline → Submissions: writes · Pipeline creates moderation submission records.
- Main pipeline → Tracking: updates · Pipeline records operational tracking state.
- Submissions → Supabase client: uses · Submission writes go through the shared Supabase client.
- Tracking → Supabase client: uses · Tracking writes go through the shared Supabase client.
- Config load → Search terms: includes · Configuration includes search-term/provider defaults.
- Config load → Setup flow: supports · Interactive setup populates local configuration.
- Main pipeline → Logger: logs via · Orchestrator and stages share logging helpers.
- Main pipeline → Delay: paces with · Pipeline uses delay helpers for throttling and retries.
- Main pipeline → Validation: validates with · Pipeline uses validation helpers before downstream actions.
- Setup script → Config load: bootstraps · Setup script prepares configuration values.
- Seed bot user → Supabase client: provisions · Bot user seeding requires database connectivity.