# mohammedcha/gplay-scraper architecture

> Architecture diagram of the GitHub repository mohammedcha/gplay-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mohammedcha/gplay-scraper
- Repository: https://github.com/mohammedcha/gplay-scraper

## Overview

1. **What this repo is**
   - A Python package named `gplay_scraper` that scrapes Google Play Store pages and endpoints.
   - It is a library, not a service: the main boundary is the public Python API exposed to callers.

2. **Primary entrypoint**
   - The user-facing object is `GPlayScraper`, exposed from the package root and implemented through `gplay_scraper/app.py` plus the core modules.
   - The API is organized around 7 method families: app, search, reviews, developer, list, similar, and suggest.

3. **Top-level request flow**
   - Caller invokes a method such as `app_analyze(...)`, `search_print_all(...)`, or `reviews_get_field(...)`.
   - That call is routed into the core scraping layer, which builds the request, fetches Google Play content, parses the response, and returns structured data or prints JSON.

4. **Scraping engine**
   - `gplay_scraper/core/gplay_scraper.py` appears to be the central orchestrator.
   - `gplay_scraper/core/gplay_methods.py` likely groups the shared behaviors for the 7 feature areas and their common operations like analyze/get_field/get_fields/print_*.
   - This layer is where the method families are coordinated and normalized.

5. **Parsing and field extraction**
   - `gplay_scraper/core/gplay_parser.py` is the parsing boundary between raw Google Play responses and Python data.
   - `gplay_scraper/models/element_specs.py` suggests field definitions/specifications for extracting app metadata, review fields, chart items, and related structures.
   - This is the main transformation layer that turns HTML/response data into dictionaries, lists, and printable output.

6. **Transport abstraction**
   - `gplay_scraper/utils/http_client.py` is a key architectural component: it abstracts HTTP access behind multiple client implementations.
   - The README indicates support for several transport backends with fallback behavior: `requests`, `curl_cffi`, `tls_client`, `httpx`, `urllib3`, `cloudscraper`, and `aiohttp`.
   - This makes network access a swappable subsystem rather than a hard dependency on one library.

7. **Reliability and error handling**
   - `gplay_scraper/utils/error_handling.py` and `gplay_scraper/exceptions.py` define the library’s failure boundaries.
   - The README highlights retry logic, rate limiting, graceful degradation, and specific fixes for empty/partial review responses.
   - These concerns are important because scraping is brittle and the library is built to keep returning partial results instead of failing hard where possible.

8. **Configuration and shared helpers**
   - `gplay_scraper/config.py`, `gplay_scraper/utils/constants.py`, and `gplay_scraper/utils/helpers.py` provide shared defaults, constants, and reusable helper logic.
   - These modules likely centralize things like locale/country handling, URL construction, parsing utilities, and output formatting rules.

9. **Data domains exposed by the library**
   - The repo is organized around Google Play data domains rather than internal technical layers:
     - app metadata and media
     - search results
     - user reviews
     - developer app portfolios
     - top charts/listing data
     - similar apps
     - search suggestions
   - The input parameters across these domains are typically `app_id`, keyword, developer ID, chart/category, plus locale and count controls.

10. **Output shape**
   - The library returns structured Python data and can also print formatted JSON-like output.
   - The `output/` directory contains example JSON artifacts for each method family, showing the expected result shapes.
   - This matters for architecture because the parser/output layer is a core responsibility, not an afterthought.

11. **Documentation and examples as a separate boundary**
   - `docs/`, `README/`, `examples/`, and `build_docs.py` form a documentation and usage layer separate from runtime scraping.
   - The Sphinx docs and example scripts mirror the method families, so they are useful reference points for the public API surface.

12. **Testing and CI**
   - `tests/` contains coverage per method family plus basic/package tests, which mirrors the library’s subsystem split.
   - GitHub Actions workflows run tests and docs builds, indicating the repo treats API stability and documentation generation as part of the delivery pipeline.

13. **How to diagram it**
   - The cleanest architecture diagram would show:
     - external caller → `GPlayScraper` public API
     - API → method family/orchestration layer
     - orchestration → HTTP client abstraction
     - HTTP client → Google Play
     - response → parser/model specs
     - parser → structured result / printed output
     - shared utilities feeding config, errors, and helpers across the stack
   - The key boundary is between reusable Python API logic and the external Google Play source.

Main entry points: [`gplay_scraper/core/gplay_methods.py`](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_methods.py) (Method families), [`gplay_scraper/core/gplay_parser.py`](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_parser.py) (Parser), [`gplay_scraper/core/gplay_scraper.py`](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_scraper.py) (Scraper core), [`gplay_scraper/utils/http_client.py`](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/http_client.py) (HTTP client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_public_api["Public API"]
  node_gplay_scraper_public["GPlayScraper<br/>public facade<br/>[__init__.py]"]
  node_app_module["app.py<br/>entry facade<br/>[app.py]"]
  node_method_families["Method families<br/>API surface<br/>[gplay_methods.py]"]
end

subgraph group_core_scraping["Core Scraping"]
  node_core_orchestrator["Scraper core<br/>orchestrator<br/>[gplay_scraper.py]"]
  node_gplay_methods_core["Shared methods<br/>coordination layer<br/>[gplay_methods.py]"]
  node_parser["Parser<br/>[gplay_parser.py]"]
  node_element_specs["Field specs<br/>schema/specification<br/>[element_specs.py]"]
end

subgraph group_shared_support["Shared Support"]
  node_http_client["HTTP client<br/>transport abstraction<br/>[http_client.py]"]
  node_error_handling["Error handling<br/>reliability layer<br/>[error_handling.py]"]
  node_exceptions["Exceptions<br/>failure model<br/>[exceptions.py]"]
  node_config["Config<br/>defaults<br/>[config.py]"]
  node_constants["Constants<br/>[constants.py]"]
  node_helpers["Helpers<br/>utility layer<br/>[helpers.py]"]
end

subgraph group_docs_examples["Docs & Examples"]
  node_docs["Docs<br/>documentation<br/>[index.rst]"]
  node_examples["Examples<br/>usage samples<br/>[README.md]"]
  node_sample_output["Sample output<br/>result artifacts<br/>[app_example.json]"]
end

subgraph group_validation["Validation"]
  node_tests["Tests<br/>validation suite<br/>[test_basic.py]"]
  node_ci["CI workflows<br/>automation<br/>[test.yml]"]
end

node_google_play(("Google Play<br/>external source"))

node_app_module -->|"exports"| node_gplay_scraper_public
node_gplay_scraper_public -->|"uses"| node_method_families
node_method_families -->|"delegates"| node_core_orchestrator
node_core_orchestrator -->|"coordinates"| node_gplay_methods_core
node_gplay_methods_core -->|"parses via"| node_parser
node_parser -->|"applies"| node_element_specs
node_core_orchestrator -->|"fetches with"| node_http_client
node_http_client -->|"requests"| node_google_play
node_core_orchestrator -->|"wraps"| node_error_handling
node_error_handling -->|"raises"| node_exceptions
node_config -->|"configures"| node_http_client
node_constants -->|"supports"| node_parser
node_helpers -->|"supports"| node_parser
node_helpers -->|"supports"| node_http_client
node_docs -->|"documents"| node_method_families
node_examples -->|"demonstrates"| node_method_families
node_sample_output -->|"illustrates"| node_parser
node_tests -->|"verifies"| node_method_families
node_ci -->|"runs"| node_tests
node_ci -->|"builds"| node_docs

click node_gplay_scraper_public "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/__init__.py"
click node_app_module "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/app.py"
click node_method_families "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_methods.py"
click node_core_orchestrator "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_scraper.py"
click node_gplay_methods_core "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_methods.py"
click node_parser "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_parser.py"
click node_element_specs "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/models/element_specs.py"
click node_http_client "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/http_client.py"
click node_error_handling "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/error_handling.py"
click node_exceptions "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/exceptions.py"
click node_config "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/config.py"
click node_constants "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/constants.py"
click node_helpers "https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/helpers.py"
click node_docs "https://github.com/mohammedcha/gplay-scraper/blob/main/docs/index.rst"
click node_examples "https://github.com/mohammedcha/gplay-scraper/blob/main/examples/README.md"
click node_sample_output "https://github.com/mohammedcha/gplay-scraper/blob/main/output/app_example.json"
click node_tests "https://github.com/mohammedcha/gplay-scraper/blob/main/tests/test_basic.py"
click node_ci "https://github.com/mohammedcha/gplay-scraper/blob/main/.github/workflows/test.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gplay_scraper_public,node_app_module,node_method_families toneBlue
class node_core_orchestrator,node_gplay_methods_core,node_parser,node_element_specs toneAmber
class node_http_client,node_error_handling,node_exceptions,node_config,node_constants,node_helpers toneMint
class node_docs,node_examples,node_sample_output toneRose
class node_tests,node_ci toneIndigo
class node_google_play toneNeutral
```

## Components

### Public API

User-facing Python entrypoints and method families.

- [GPlayScraper](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/__init__.py): public facade · Main package object exposed to callers. · `gplay_scraper/__init__.py`
- [app.py](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/app.py): entry facade · Public module that wires the top-level scraper object. · `gplay_scraper/app.py`
- [Method families](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_methods.py): API surface · App, search, reviews, developer, list, similar, and suggest operations. · `gplay_scraper/core/gplay_methods.py`

### Core Scraping

Orchestration, parsing, and field extraction.

- [Scraper core](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_scraper.py): orchestrator · Central request coordinator for scrape operations. · `gplay_scraper/core/gplay_scraper.py`
- [Shared methods](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_methods.py): coordination layer · Normalizes analyze/get_field/get_fields/print behavior across domains. · `gplay_scraper/core/gplay_methods.py`
- [Parser](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/core/gplay_parser.py): response parser · Transforms Google Play responses into Python structures. · `gplay_scraper/core/gplay_parser.py`
- [Field specs](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/models/element_specs.py): schema/specification · Defines extractable fields and domain-specific element shapes. · `gplay_scraper/models/element_specs.py`

### Shared Support

Transport, config, errors, and reusable helpers.

- [HTTP client](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/http_client.py): transport abstraction · Pluggable backend layer for requests, curl_cffi, httpx, urllib3, aiohttp, and others. · `gplay_scraper/utils/http_client.py`
- [Error handling](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/error_handling.py): reliability layer · Retries, rate limiting, and graceful degradation helpers. · `gplay_scraper/utils/error_handling.py`
- [Exceptions](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/exceptions.py): failure model · Library-specific error types and failure boundaries. · `gplay_scraper/exceptions.py`
- [Config](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/config.py): defaults · Shared runtime defaults and option handling. · `gplay_scraper/config.py`
- [Constants](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/constants.py): shared constants · Common values for locale, URLs, and parser behavior. · `gplay_scraper/utils/constants.py`
- [Helpers](https://github.com/mohammedcha/gplay-scraper/blob/main/gplay_scraper/utils/helpers.py): utility layer · Reusable helper functions for scraping and formatting. · `gplay_scraper/utils/helpers.py`

### Docs & Examples

Documentation, example scripts, and sample outputs.

- [Docs](https://github.com/mohammedcha/gplay-scraper/blob/main/docs/index.rst): documentation · Sphinx docs and method reference pages. · `docs/index.rst`
- [Examples](https://github.com/mohammedcha/gplay-scraper/blob/main/examples/README.md): usage samples · Example scripts mirroring the public method families. · `examples/README.md`
- [Sample output](https://github.com/mohammedcha/gplay-scraper/blob/main/output/app_example.json): result artifacts · Example JSON outputs for each method family. · `output/app_example.json`

### Validation

Tests and automation for package quality.

- [Tests](https://github.com/mohammedcha/gplay-scraper/blob/main/tests/test_basic.py): validation suite · Method-level and package-level tests for the library. · `tests/test_basic.py`
- [CI workflows](https://github.com/mohammedcha/gplay-scraper/blob/main/.github/workflows/test.yml): automation · GitHub Actions for tests and docs builds. · `.github/workflows/test.yml`

### Other components

- **Google Play**: external source · Upstream store pages and endpoints being scraped.

## Connections

- app.py → GPlayScraper: exports · The app module exposes the main scraper object.
- GPlayScraper → Method families: uses · Public API routes user calls into the method families.
- Method families → Scraper core: delegates · Method families delegate request execution and normalization.
- Scraper core → Shared methods: coordinates · Core orchestrator uses shared method behavior.
- Shared methods → Parser: parses via · Shared operations rely on parsing and field extraction.
- Parser → Field specs: applies · Parser uses field specifications to shape output.
- Scraper core → HTTP client: fetches with · Requests are executed through the transport abstraction.
- HTTP client → Google Play: requests · Transport layer reaches Google Play endpoints and pages.
- Scraper core → Error handling: wraps · Execution is guarded by retries and graceful recovery.
- Error handling → Exceptions: raises · Failures are represented using library exceptions.
- Config → HTTP client: configures · Runtime settings influence transport selection and behavior.
- Constants → Parser: supports · Shared constants feed parsing and request shaping.
- Helpers → Parser: supports · Helper logic assists extraction and formatting.
- Helpers → HTTP client: supports · Helpers support request and response utilities.
- Docs → Method families: documents · Docs mirror the public API surface.
- Examples → Method families: demonstrates · Examples show the method families in use.
- Sample output → Parser: illustrates · Output artifacts show the structured results returned by parsing.
- Tests → Method families: verifies · Tests cover the main API families.
- CI workflows → Tests: runs · CI executes the test suite and docs build.
- CI workflows → Docs: builds · CI validates documentation generation.
