# oxylabs/how-to-scrape-google-trends architecture

> Architecture diagram of the GitHub repository oxylabs/how-to-scrape-google-trends, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/oxylabs/how-to-scrape-google-trends
- Repository: https://github.com/oxylabs/how-to-scrape-google-trends

## Overview

1. **Repository purpose**
   - This is a small Python data-scraping utility, not an application server.
   - Its job is to fetch Google Trends Explore data through Oxylabs’ SERP Scraper API, normalize the response, and write CSV outputs for later analysis.

2. **Main runtime entry point**
   - `scraper.py` is the executable script.
   - The flow is batch-oriented: iterate over a list of keywords, request trends data for each, persist per-keyword CSVs, then generate cross-keyword comparison CSVs.

3. **External dependency boundary**
   - The only live external service is the Oxylabs API endpoint at `https://realtime.oxylabs.io/v1/queries`.
   - Authentication is basic auth with username/password credentials.
   - The script depends on this service to translate a Google Trends query into structured results; it does not talk directly to Google Trends.

4. **Input model**
   - The input is a keyword string such as `cat` or `cats`.
   - In the sample logic, the keyword list is hardcoded in `main()`, so there is no UI, config file, or database input layer.
   - This makes the repo a simple offline batch job with a fixed set of queries per run.

5. **Request/response subsystem**
   - `get_trend_data()` builds a POST request payload with `source: google_trends_explore` and the query keyword.
   - The function is the network boundary: it isolates HTTP handling, timeout/error handling, and response parsing.
   - The API response is JSON, but the useful payload is nested inside `results[0]["content"]`, which is then decoded into a Python dictionary.

6. **Data shaping subsystem**
   - The API response contains four logical datasets:
     - `interest_over_time`
     - `breakdown_by_region`
     - `related_topics`
     - `related_queries`
   - `dump_trend_data_to_csv()` converts each of these into a Pandas DataFrame.
   - `related_topics` needs a special flattening step because its structure is nested; `flatten_topic_data()` normalizes that nested shape into a tabular record set.

7. **Storage boundary**
   - Output is written to the local filesystem only; there is no database or object store.
   - Each keyword gets its own directory under `./csv/<keyword>/`.
   - Inside that directory, four CSV files are created, one for each dataset category.
   - This creates a clean separation between source keywords and per-keyword artifacts.

8. **Comparison subsystem**
   - `create_comparison()` combines the per-keyword DataFrames into cross-keyword comparison tables.
   - It merges datasets on category-specific keys:
     - `time` for interest over time
     - `geo_code` for region breakdown
     - `title` for related topics
     - `query` for related queries
   - The result is a second set of CSVs at the repository root, intended for comparative analysis across keywords.

9. **Data flow end to end**
   - Keyword list in `main()` → API request per keyword → raw JSON response → normalized Python dict → category DataFrames → keyword-scoped CSVs → merged comparison CSVs.
   - The architecture is linear and pipeline-like, with each stage producing the input for the next.

10. **Technology stack that matters architecturally**
    - Python is the runtime.
    - `requests` handles HTTP communication with the scraper API.
    - `pandas` is the transformation layer for tabular normalization and CSV output.
    - The repo is intentionally lightweight: no framework, no async job runner, no persistence layer beyond local files.

11. **Repository structure**
    - `README.md` documents the workflow and provides the reference implementation.
    - `scraper.py` is the operational script.
    - `images/` contains screenshots used in documentation and does not affect runtime behavior.

12. **Important architectural constraint**
    - The script assumes the API returns the expected nested schema and indexes directly into those fields.
    - That means schema stability is important: the runtime depends on the current Oxylabs response format for Google Trends Explore results.

Main entry points: [`scraper.py`](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py) (Main loop), [`README.md`](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/README.md) (README.md), [`images`](https://github.com/oxylabs/how-to-scrape-google-trends/tree/main/images) (images/)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_scraper_py["scraper.py<br/>batch script<br/>[scraper.py]"]
  node_main_flow(("Main loop<br/>orchestration<br/>[scraper.py]"))
  node_get_trend_data["Fetch trends<br/>http boundary<br/>[scraper.py]"]
  node_oxylabs_api[("Oxylabs API<br/>external service")]
  node_response_decode["Decode payload<br/>json parsing<br/>[scraper.py]"]
  node_trend_datasets{{"Trend datasets<br/>dataset bundle"}}
  node_flatten_topics["Flatten topics<br/>nested normalization<br/>[scraper.py]"]
  node_tabular_frames["DataFrames<br/>tabular shaping<br/>[scraper.py]"]
end

subgraph group_storage["Storage"]
  node_dump_csv["Write keyword CSVs<br/>csv export"]
  node_keyword_csvs[("Keyword outputs<br/>local files")]
  node_comparison["Compare keywords<br/>merge export<br/>[scraper.py]"]
  node_comparison_csvs["Comparison CSVs<br/>local files"]
end

subgraph group_docs["Docs"]
  node_readme_md["README.md<br/>documentation<br/>[README.md]"]
  node_images_dir["images/<br/>docs assets"]
end

node_scraper_py -->|"runs"| node_main_flow
node_main_flow -->|"per keyword"| node_get_trend_data
node_get_trend_data -->|"POST /queries"| node_oxylabs_api
node_oxylabs_api -->|"JSON response"| node_response_decode
node_response_decode -->|"decoded data"| node_trend_datasets
node_trend_datasets -->|"related topics"| node_flatten_topics
node_trend_datasets -->|"other datasets"| node_tabular_frames
node_flatten_topics -->|"normalized records"| node_tabular_frames
node_tabular_frames -->|"ready to write"| node_dump_csv
node_dump_csv -->|"writes"| node_keyword_csvs
node_main_flow -->|"after batch"| node_comparison
node_comparison -->|"writes"| node_comparison_csvs
node_readme_md -.->|"documents"| node_scraper_py
node_readme_md -.->|"references"| node_images_dir

click node_scraper_py "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_main_flow "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_get_trend_data "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_response_decode "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_flatten_topics "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_tabular_frames "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_comparison "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py"
click node_readme_md "https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/README.md"
click node_images_dir "https://github.com/oxylabs/how-to-scrape-google-trends/tree/main/images"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper_py,node_main_flow,node_get_trend_data,node_oxylabs_api,node_response_decode,node_trend_datasets,node_flatten_topics,node_tabular_frames toneBlue
class node_dump_csv,node_keyword_csvs,node_comparison,node_comparison_csvs toneAmber
class node_readme_md,node_images_dir toneMint
```

## Components

### Runtime

Batch scraping pipeline executed by the Python script.

- [scraper.py](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): batch script · Executable entry point that loops over keywords, fetches trends data, and coordinates output generation. · `scraper.py`
- [Main loop](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): orchestration · Hardcoded keyword batch that drives one run from input list through export. · `scraper.py`
- [Fetch trends](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): http boundary · Builds the POST request, handles authentication and response parsing, and isolates the Oxylabs API call. · `scraper.py`
- **Oxylabs API**: external service · Live SERP Scraper endpoint used to translate Google Trends Explore queries into structured results.
- [Decode payload](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): json parsing · Extracts the nested content field from the API response and turns it into Python data structures. · `scraper.py`
- **Trend datasets**: dataset bundle · The four logical outputs returned by the API: interest over time, region breakdown, related topics, and related queries.
- [Flatten topics](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): nested normalization · Special flattening step that turns nested related-topics records into a tabular structure. · `scraper.py`
- [DataFrames](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): tabular shaping · Pandas transformations that convert each dataset category into a CSV-ready table. · `scraper.py`

### Storage

Local CSV artifacts produced per keyword and for comparisons.

- **Write keyword CSVs**: csv export · Writes four per-keyword CSV files after normalization is complete.
- **Keyword outputs**: local files · Per-keyword directories under the local CSV output area containing the four dataset files.
- [Compare keywords](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/scraper.py): merge export · Combines per-keyword tables into cross-keyword comparison outputs keyed by dataset-specific fields. · `scraper.py`
- **Comparison CSVs**: local files · Repository-root CSV outputs intended for comparative analysis across keywords.

### Docs

Repository documentation and illustrative assets.

- [README.md](https://github.com/oxylabs/how-to-scrape-google-trends/blob/main/README.md): documentation · Explains the workflow and includes the reference implementation. · `README.md`
- [images/](https://github.com/oxylabs/how-to-scrape-google-trends/tree/main/images): docs assets · Screenshots used by the documentation. · `images`

## Connections

- scraper.py → Main loop: runs · The executable script starts the batch workflow.
- Main loop → Fetch trends: per keyword · Each keyword triggers one API request.
- Fetch trends → Oxylabs API: POST /queries · Sends authenticated requests to the scraper endpoint.
- Oxylabs API → Decode payload: JSON response · Returns nested response content for Google Trends Explore data.
- Decode payload → Trend datasets: decoded data · Decoded content yields the four logical dataset groups.
- Trend datasets → Flatten topics: related topics · Nested topic records require special flattening before export.
- Trend datasets → DataFrames: other datasets · Each dataset category is converted into a DataFrame for downstream writing.
- Flatten topics → DataFrames: normalized records · Flattened topics are folded into the tabular export pipeline.
- DataFrames → Write keyword CSVs: ready to write · Prepared DataFrames are exported as keyword-scoped CSVs.
- Write keyword CSVs → Keyword outputs: writes · Creates local per-keyword CSV artifacts.
- Main loop → Compare keywords: after batch · Once all keywords are processed, comparison tables are generated.
- Compare keywords → Comparison CSVs: writes · Outputs merged CSVs for cross-keyword analysis.
- README.md → scraper.py: documents · Documentation points to the operational script.
- README.md → images/: references · Documentation includes screenshots stored in the images folder.