# georgekhananaev/google-reviews-scraper-pro architecture

> Architecture diagram of the GitHub repository georgekhananaev/google-reviews-scraper-pro, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/georgekhananaev/google-reviews-scraper-pro
- Repository: https://github.com/georgekhananaev/google-reviews-scraper-pro

## Overview

1. **High-level shape**
This repository is a Python scraping system with two front doors: a command-line launcher (`start.py`) and a REST API server (`api_server.py`). Both drive the same core scrape-and-persist pipeline, so the architecture is centered around shared modules rather than separate CLI/API implementations.

2. **Core execution path**
The main runtime flow is: configuration is loaded, a business/place URL is resolved, Selenium-based scraping collects review data, and the results are normalized and written into storage. The pipeline also decides whether to download images, sync to MongoDB, upload to S3-compatible storage, and emit JSON backups.

3. **Scraping layer**
`modules/scraper.py` is the browser automation boundary. It is the only part that interacts directly with Google Maps/Google Reviews in the browser. The README and tests indicate SeleniumBase in undetected/UC mode is used to reduce detection, and the scraper includes logic for search-based navigation, scrolling, and review extraction.

4. **Orchestration layer**
`modules/pipeline.py` appears to coordinate an entire scrape session end-to-end. It likely ties together scraper output, data normalization, database writes, history tracking, and optional side effects like image handling or external sync. This is the main “application service” boundary for both CLI runs and API-triggered jobs.

5. **Primary persistence model**
SQLite is the canonical datastore. The README’s schema implies a relational core with `places`, `reviews`, `review_history`, `scrape_sessions`, and `schema_version`. The `modules/review_db.py`, `modules/database_backend.py`, and `modules/migration.py` modules separate business-level review persistence from lower-level DB access and schema evolution.

6. **Change tracking and review lifecycle**
This repo is not just storing snapshots; it tracks review evolution over time. `modules/data_logic.py` and the database/history tables support change detection, audit history, soft delete/restore, and scrape session metadata. That creates an important boundary: the scraper produces raw review records, while the data layer determines whether they are new, updated, restored, hidden, or unchanged.

7. **Configuration and identity resolution**
`modules/config.py`, `modules/place_id.py`, and `modules/utils.py` likely handle config parsing, business overrides, and mapping between Google URLs and canonical `place_id` values. The multi-business setup in the README means configuration is hierarchical: global defaults are inherited, but each business can override DB, MongoDB, or S3 settings independently.

8. **Media handling**
Images are a separate concern from review text. `modules/image_handler.py` handles downloading, threading, and resizing locally, while `modules/s3_handler.py` uploads the same image set to S3-compatible storage. The storage layout is per-place, which keeps each business isolated in both local directories and object storage prefixes.

9. **Optional external sync**
MongoDB is an optional downstream sink, not the source of truth. The README describes incremental sync, so the architecture likely pushes only changed reviews to MongoDB after the SQLite write succeeds. This keeps SQLite as the authoritative audit store while allowing MongoDB to serve search/analytics use cases.

10. **REST API subsystem**
`api_server.py` exposes the scraper and the database through HTTP endpoints. The API includes system stats, job control, places, reviews, and audit-log access. Authentication is managed in SQLite via `modules/api_keys.py`, which means API access is governed by the same local database that stores review data.

11. **Background job management**
`modules/job_manager.py` is the concurrency boundary for API-triggered scraping. Instead of blocking HTTP requests during long scrape runs, jobs are created, tracked, started, canceled, and cleaned up asynchronously. That separates request handling from long-running browser automation.

12. **Logging and observability**
`modules/log_manager.py` provides structured JSON logging with rotation, and the CLI can tail or filter logs. The system appears designed for operational use: scrape sessions, API requests, and audit events are all observable after the fact.

13. **Migration and maintenance tools**
Several CLI commands are maintenance-oriented rather than scraping-oriented: export/import, database stats, clear data, hide/restore reviews, prune history, and sync-status checks. `modules/migration.py` and `modules/review_db.py` support moving data into the canonical store from JSON or MongoDB, which suggests the project treats ingestion and migration as first-class workflows.

14. **Tests as architecture map**
The test suite mirrors the major subsystems: config, CLI, database backend, review DB, S3 providers, MongoDB connection, SeleniumBase integration, pipeline, API keys, and migration. That confirms the repo is intentionally modular, with clear boundaries around scraping, persistence, sync, and operational tooling.

15. **Suggested diagram boundaries**
For an architecture diagram, the clearest top-level boxes are: entry points (CLI/API), orchestration pipeline, browser scraper, SQLite core datastore, optional MongoDB sync, optional S3/image storage, and supporting services for config, logging, and background jobs. The key data flow is one-way from Google Maps into normalization/persistence, with optional fan-out to MongoDB, S3, JSON backups, and API consumers.

Main entry points: [`modules/pipeline.py`](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/pipeline.py) (Pipeline), [`modules/review_db.py`](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/review_db.py) (Review DB), [`api_server.py`](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/api_server.py) (API server), [`modules/log_manager.py`](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/log_manager.py) (Logging)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["Entry points"]
  node_start_py["CLI launcher<br/>entrypoint<br/>[start.py]"]
  node_api_server_py["API server<br/>http entrypoint<br/>[api_server.py]"]
end

subgraph group_core["Scrape core"]
  node_pipeline["Pipeline<br/>orchestrator<br/>[pipeline.py]"]
  node_scraper["Scraper<br/>browser automation<br/>[scraper.py]"]
  node_data_logic["Data logic<br/>review lifecycle<br/>[data_logic.py]"]
end

subgraph group_storage["Data & sync"]
  node_review_db["Review DB<br/>persistence facade<br/>[review_db.py]"]
  node_database_backend["DB backend<br/>sqlite access"]
  node_migration["Migrations<br/>schema evolution<br/>[migration.py]"]
  node_data_storage["Data storage<br/>repository layer<br/>[data_storage.py]"]
  node_mongo_sync[("Mongo sync<br/>optional sink<br/>[data_storage.py]")]
  node_image_handler["Image handler<br/>media pipeline<br/>[image_handler.py]"]
  node_s3_handler["S3 uploader<br/>object storage<br/>[s3_handler.py]"]
end

subgraph group_ops["Support services"]
  node_config["Config<br/>[config.py]"]
  node_place_id["Place ID<br/>identity resolution<br/>[place_id.py]"]
  node_job_manager["Job manager<br/>async jobs<br/>[job_manager.py]"]
  node_log_manager["Logging<br/>observability<br/>[log_manager.py]"]
  node_api_keys["API keys<br/>auth<br/>[api_keys.py]"]
  node_utils["Utilities<br/>shared helpers<br/>[utils.py]"]
end

node_start_py -->|"runs"| node_pipeline
node_api_server_py -->|"submits jobs"| node_job_manager
node_api_server_py -->|"authenticates"| node_api_keys
node_job_manager -->|"starts"| node_pipeline
node_pipeline -->|"loads"| node_config
node_pipeline -->|"resolves"| node_place_id
node_pipeline -->|"scrapes"| node_scraper
node_scraper -->|"emits reviews"| node_data_logic
node_data_logic -->|"persists"| node_review_db
node_review_db -->|"uses"| node_database_backend
node_review_db -->|"evolves schema"| node_migration
node_review_db -->|"shares storage"| node_data_storage
node_data_storage -->|"syncs"| node_mongo_sync
node_pipeline -->|"downloads images"| node_image_handler
node_image_handler -->|"uploads"| node_s3_handler
node_api_server_py -->|"reads data"| node_review_db
node_api_server_py -->|"logs"| node_log_manager
node_start_py -->|"logs"| node_log_manager
node_pipeline -->|"logs"| node_log_manager
node_config -->|"uses"| node_utils
node_place_id -->|"uses"| node_utils

click node_start_py "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/start.py"
click node_api_server_py "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/api_server.py"
click node_pipeline "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/pipeline.py"
click node_scraper "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/scraper.py"
click node_data_logic "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/data_logic.py"
click node_review_db "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/review_db.py"
click node_database_backend "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/database_backend.py"
click node_migration "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/migration.py"
click node_data_storage "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/data_storage.py"
click node_mongo_sync "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/data_storage.py"
click node_image_handler "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/image_handler.py"
click node_s3_handler "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/s3_handler.py"
click node_config "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/config.py"
click node_place_id "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/place_id.py"
click node_job_manager "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/job_manager.py"
click node_log_manager "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/log_manager.py"
click node_api_keys "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/api_keys.py"
click node_utils "https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/utils.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_start_py,node_api_server_py toneBlue
class node_pipeline,node_scraper,node_data_logic toneAmber
class node_review_db,node_database_backend,node_migration,node_data_storage,node_mongo_sync,node_image_handler,node_s3_handler toneMint
class node_config,node_place_id,node_job_manager,node_log_manager,node_api_keys,node_utils toneRose
```

## Components

### Entry points

Command-line and HTTP front doors

- [CLI launcher](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/start.py): entrypoint · Starts command-line scraping and maintenance flows · `start.py`
- [API server](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/api_server.py): http entrypoint · Exposes scraping, stats, jobs, and review data over HTTP · `api_server.py`

### Scrape core

Shared orchestration and browser automation

- [Pipeline](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/pipeline.py): orchestrator · Coordinates scrape sessions end-to-end · `modules/pipeline.py`
- [Scraper](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/scraper.py): browser automation · Selenium-based Google Maps/Reviews extraction boundary · `modules/scraper.py`
- [Data logic](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/data_logic.py): review lifecycle · Normalizes reviews and applies change detection, restore, and history rules · `modules/data_logic.py`

### Data & sync

Canonical storage and optional sinks

- [Review DB](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/review_db.py): persistence facade · Business-level review persistence and query operations · `modules/review_db.py`
- [DB backend](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/database_backend.py): sqlite access · Low-level SQLite connection and schema operations · `modules/database_backend.py`
- [Migrations](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/migration.py): schema evolution · Schema upgrades and import/export maintenance paths · `modules/migration.py`
- [Data storage](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/data_storage.py): repository layer · Shared storage helpers and persistence abstractions · `modules/data_storage.py`
- [Mongo sync](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/data_storage.py): optional sink · Incremental sync target for changed review data · `modules/data_storage.py`
- [Image handler](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/image_handler.py): media pipeline · Downloads and prepares review images locally · `modules/image_handler.py`
- [S3 uploader](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/s3_handler.py): object storage · Uploads images and backups to S3-compatible storage · `modules/s3_handler.py`

### Support services

Configuration, logging, jobs, and identity

- [Config](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/config.py): runtime config · Loads global and per-business settings · `modules/config.py`
- [Place ID](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/place_id.py): identity resolution · Resolves Google URLs and business identifiers to canonical place IDs · `modules/place_id.py`
- [Job manager](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/job_manager.py): async jobs · Tracks and controls background scraping jobs for the API · `modules/job_manager.py`
- [Logging](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/log_manager.py): observability · Structured logging and log rotation · `modules/log_manager.py`
- [API keys](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/api_keys.py): auth · API authentication backed by local SQLite data · `modules/api_keys.py`
- [Utilities](https://github.com/georgekhananaev/google-reviews-scraper-pro/blob/master/modules/utils.py): shared helpers · Common helper functions used across modules · `modules/utils.py`

## Connections

- CLI launcher → Pipeline: runs · CLI invokes the shared scrape workflow
- API server → Job manager: submits jobs · HTTP requests enqueue background scraping work
- API server → API keys: authenticates · API access is checked against local keys
- Job manager → Pipeline: starts · Background jobs execute the same core pipeline
- Pipeline → Config: loads · Pipeline reads runtime and per-business settings
- Pipeline → Place ID: resolves · Canonical place identity is determined before scraping
- Pipeline → Scraper: scrapes · Pipeline drives browser extraction
- Scraper → Data logic: emits reviews · Raw review data is normalized and classified
- Data logic → Review DB: persists · Processed reviews are written through the review persistence layer
- Review DB → DB backend: uses · Review persistence relies on SQLite backend primitives
- Review DB → Migrations: evolves schema · Schema changes and imports are managed alongside persistence
- Review DB → Data storage: shares storage · Higher-level storage helpers support canonical persistence
- Data storage → Mongo sync: syncs · Changed data can be pushed to MongoDB as an optional sink
- Pipeline → Image handler: downloads images · Image collection is coordinated as part of the scrape session
- Image handler → S3 uploader: uploads · Prepared media can be published to object storage
- API server → Review DB: reads data · API endpoints expose stored review and audit data
- API server → Logging: logs · HTTP activity and job events are recorded centrally
- CLI launcher → Logging: logs · CLI runs share the same operational logging
- Pipeline → Logging: logs · Scrape sessions and outcomes are recorded
- Config → Utilities: uses · Configuration helpers rely on shared utility code
- Place ID → Utilities: uses · Identity resolution uses shared parsing and helper routines
