How georgekhananaev/google-reviews-scraper-pro works
1. High-level shape
This repository is a Python scraping system with two front doors: a command-line launcher (start.py) and a REST API server (api_server.py). Both drive the same core scrape-and-persist pipeline, so the architecture is centered around shared modules rather than separate CLI/API implementations.
2. Core execution path
The main runtime flow is: configuration is loaded, a business/place URL is resolved, Selenium-based scraping collects review data, and the results are normalized and written into storage. The pipeline also decides whether to download images, sync to MongoDB, upload to S3-compatible storage, and emit JSON backups.
3. Scraping layer
modules/scraper.py is the browser automation boundary. It is the only part that interacts directly with Google Maps/Google Reviews in the browser. The README and tests indicate SeleniumBase in undetected/UC mode is used to reduce detection, and the scraper includes logic for search-based navigation, scrolling, and review extraction.
4. Orchestration layer
modules/pipeline.py appears to coordinate an entire scrape session end-to-end. It likely ties together scraper output, data normalization, database writes, history tracking, and optional side effects like image handling or external sync. This is the main “application service” boundary for both CLI runs and API-triggered jobs.
5. Primary persistence model
SQLite is the canonical datastore. The README’s schema implies a relational core with places, reviews, review_history, scrape_sessions, and schema_version. The modules/review_db.py, modules/database_backend.py, and modules/migration.py modules separate business-level review persistence from lower-level DB access and schema evolution.
6. Change tracking and review lifecycle
This repo is not just storing snapshots; it tracks review evolution over time. modules/data_logic.py and the database/history tables support change detection, audit history, soft delete/restore, and scrape session metadata. That creates an important boundary: the scraper produces raw review records, while the data layer determines whether they are new, updated, restored, hidden, or unchanged.
7. Configuration and identity resolution
modules/config.py, modules/place_id.py, and modules/utils.py likely handle config parsing, business overrides, and mapping between Google URLs and canonical place_id values. The multi-business setup in the README means configuration is hierarchical: global defaults are inherited, but each business can override DB, MongoDB, or S3 settings independently.
8. Media handling
Images are a separate concern from review text. modules/image_handler.py handles downloading, threading, and resizing locally, while modules/s3_handler.py uploads the same image set to S3-compatible storage. The storage layout is per-place, which keeps each business isolated in both local directories and object storage prefixes.
9. Optional external sync
MongoDB is an optional downstream sink, not the source of truth. The README describes incremental sync, so the architecture likely pushes only changed reviews to MongoDB after the SQLite write succeeds. This keeps SQLite as the authoritative audit store while allowing MongoDB to serve search/analytics use cases.
10. REST API subsystem
api_server.py exposes the scraper and the database through HTTP endpoints. The API includes system stats, job control, places, reviews, and audit-log access. Authentication is managed in SQLite via modules/api_keys.py, which means API access is governed by the same local database that stores review data.
11. Background job management
modules/job_manager.py is the concurrency boundary for API-triggered scraping. Instead of blocking HTTP requests during long scrape runs, jobs are created, tracked, started, canceled, and cleaned up asynchronously. That separates request handling from long-running browser automation.
12. Logging and observability
modules/log_manager.py provides structured JSON logging with rotation, and the CLI can tail or filter logs. The system appears designed for operational use: scrape sessions, API requests, and audit events are all observable after the fact.
13. Migration and maintenance tools
Several CLI commands are maintenance-oriented rather than scraping-oriented: export/import, database stats, clear data, hide/restore reviews, prune history, and sync-status checks. modules/migration.py and modules/review_db.py support moving data into the canonical store from JSON or MongoDB, which suggests the project treats ingestion and migration as first-class workflows.
14. Tests as architecture map
The test suite mirrors the major subsystems: config, CLI, database backend, review DB, S3 providers, MongoDB connection, SeleniumBase integration, pipeline, API keys, and migration. That confirms the repo is intentionally modular, with clear boundaries around scraping, persistence, sync, and operational tooling.
15. Suggested diagram boundaries
For an architecture diagram, the clearest top-level boxes are: entry points (CLI/API), orchestration pipeline, browser scraper, SQLite core datastore, optional MongoDB sync, optional S3/image storage, and supporting services for config, logging, and background jobs. The key data flow is one-way from Google Maps into normalization/persistence, with optional fan-out to MongoDB, S3, JSON backups, and API consumers.
Main entry points: pipeline.py Pipeline · review_db.py Review DB · api_server.py API server · log_manager.py Logging
Components 18
Entry points
Command-line and HTTP front doors
- CLI launcher · Starts command-line scraping and maintenance flows
start.py - API server · Exposes scraping, stats, jobs, and review data over HTTP
api_server.py
Scrape core
Shared orchestration and browser automation
- Pipeline · Coordinates scrape sessions end-to-end
modules/pipeline.py - Scraper · Selenium-based Google Maps/Reviews extraction boundary
modules/scraper.py - Data logic · Normalizes reviews and applies change detection, restore, and history rules
modules/data_logic.py
Data & sync
Canonical storage and optional sinks
- Review DB · Business-level review persistence and query operations
modules/review_db.py - DB backend · Low-level SQLite connection and schema operations
modules/database_backend.py - Migrations · Schema upgrades and import/export maintenance paths
modules/migration.py - Data storage · Shared storage helpers and persistence abstractions
modules/data_storage.py - Mongo sync · Incremental sync target for changed review data
modules/data_storage.py - Image handler · Downloads and prepares review images locally
modules/image_handler.py - S3 uploader · Uploads images and backups to S3-compatible storage
modules/s3_handler.py
Support services
Configuration, logging, jobs, and identity
- Config · Loads global and per-business settings
modules/config.py - Place ID · Resolves Google URLs and business identifiers to canonical place IDs
modules/place_id.py - Job manager · Tracks and controls background scraping jobs for the API
modules/job_manager.py - Logging · Structured logging and log rotation
modules/log_manager.py - API keys · API authentication backed by local SQLite data
modules/api_keys.py - Utilities · Common helper functions used across modules
modules/utils.py
Connections 21
- CLI launcher → Pipeline: runs · CLI invokes the shared scrape workflow
- API server → Job manager: submits jobs · HTTP requests enqueue background scraping work
- API server → API keys: authenticates · API access is checked against local keys
- Job manager → Pipeline: starts · Background jobs execute the same core pipeline
- Pipeline → Config: loads · Pipeline reads runtime and per-business settings
- Pipeline → Place ID: resolves · Canonical place identity is determined before scraping
- Pipeline → Scraper: scrapes · Pipeline drives browser extraction
- Scraper → Data logic: emits reviews · Raw review data is normalized and classified
- Data logic → Review DB: persists · Processed reviews are written through the review persistence layer
- Review DB → DB backend: uses · Review persistence relies on SQLite backend primitives
- Review DB → Migrations: evolves schema · Schema changes and imports are managed alongside persistence
- Review DB → Data storage: shares storage · Higher-level storage helpers support canonical persistence
- Data storage → Mongo sync: syncs · Changed data can be pushed to MongoDB as an optional sink
- Pipeline → Image handler: downloads images · Image collection is coordinated as part of the scrape session
- Image handler → S3 uploader: uploads · Prepared media can be published to object storage
- API server → Review DB: reads data · API endpoints expose stored review and audit data
- API server → Logging: logs · HTTP activity and job events are recorded centrally
- CLI launcher → Logging: logs · CLI runs share the same operational logging
- Pipeline → Logging: logs · Scrape sessions and outcomes are recorded
- Config → Utilities: uses · Configuration helpers rely on shared utility code
- Place ID → Utilities: uses · Identity resolution uses shared parsing and helper routines