How oxylabs/how-to-scrape-google-trends works
1. Repository purpose
- This is a small Python data-scraping utility, not an application server.
- Its job is to fetch Google Trends Explore data through Oxylabs’ SERP Scraper API, normalize the response, and write CSV outputs for later analysis.
2. Main runtime entry point
- scraper.py is the executable script.
- The flow is batch-oriented: iterate over a list of keywords, request trends data for each, persist per-keyword CSVs, then generate cross-keyword comparison CSVs.
3. External dependency boundary
- The only live external service is the Oxylabs API endpoint at https://realtime.oxylabs.io/v1/queries.
- Authentication is basic auth with username/password credentials.
- The script depends on this service to translate a Google Trends query into structured results; it does not talk directly to Google Trends.
4. Input model
- The input is a keyword string such as cat or cats.
- In the sample logic, the keyword list is hardcoded in main(), so there is no UI, config file, or database input layer.
- This makes the repo a simple offline batch job with a fixed set of queries per run.
5. Request/response subsystem
- get_trend_data() builds a POST request payload with source: google_trends_explore and the query keyword.
- The function is the network boundary: it isolates HTTP handling, timeout/error handling, and response parsing.
- The API response is JSON, but the useful payload is nested inside results[0]["content"], which is then decoded into a Python dictionary.
6. Data shaping subsystem
- The API response contains four logical datasets:
- interest_over_time
- breakdown_by_region
- related_topics
- related_queries
- dump_trend_data_to_csv() converts each of these into a Pandas DataFrame.
- related_topics needs a special flattening step because its structure is nested; flatten_topic_data() normalizes that nested shape into a tabular record set.
7. Storage boundary
- Output is written to the local filesystem only; there is no database or object store.
- Each keyword gets its own directory under ./csv/<keyword>/.
- Inside that directory, four CSV files are created, one for each dataset category.
- This creates a clean separation between source keywords and per-keyword artifacts.
8. Comparison subsystem
- create_comparison() combines the per-keyword DataFrames into cross-keyword comparison tables.
- It merges datasets on category-specific keys:
- time for interest over time
- geo_code for region breakdown
- title for related topics
- query for related queries
- The result is a second set of CSVs at the repository root, intended for comparative analysis across keywords.
9. Data flow end to end
- Keyword list in main() → API request per keyword → raw JSON response → normalized Python dict → category DataFrames → keyword-scoped CSVs → merged comparison CSVs.
- The architecture is linear and pipeline-like, with each stage producing the input for the next.
10. Technology stack that matters architecturally
- Python is the runtime.
- requests handles HTTP communication with the scraper API.
- pandas is the transformation layer for tabular normalization and CSV output.
- The repo is intentionally lightweight: no framework, no async job runner, no persistence layer beyond local files.
11. Repository structure
- README.md documents the workflow and provides the reference implementation.
- scraper.py is the operational script.
- images/ contains screenshots used in documentation and does not affect runtime behavior.
12. Important architectural constraint
- The script assumes the API returns the expected nested schema and indexes directly into those fields.
- That means schema stability is important: the runtime depends on the current Oxylabs response format for Google Trends Explore results.
Main entry points: scraper.py Main loop · README.md README.md · images images/
Components 14
Runtime
Batch scraping pipeline executed by the Python script.
- scraper.py · Executable entry point that loops over keywords, fetches trends data, and coordinates output generation.
scraper.py - Main loop · Hardcoded keyword batch that drives one run from input list through export.
scraper.py - Fetch trends · Builds the POST request, handles authentication and response parsing, and isolates the Oxylabs API call.
scraper.py - Oxylabs API · Live SERP Scraper endpoint used to translate Google Trends Explore queries into structured results.
- Decode payload · Extracts the nested content field from the API response and turns it into Python data structures.
scraper.py - Trend datasets · The four logical outputs returned by the API: interest over time, region breakdown, related topics, and related queries.
- Flatten topics · Special flattening step that turns nested related-topics records into a tabular structure.
scraper.py - DataFrames · Pandas transformations that convert each dataset category into a CSV-ready table.
scraper.py
Storage
Local CSV artifacts produced per keyword and for comparisons.
- Write keyword CSVs · Writes four per-keyword CSV files after normalization is complete.
- Keyword outputs · Per-keyword directories under the local CSV output area containing the four dataset files.
- Compare keywords · Combines per-keyword tables into cross-keyword comparison outputs keyed by dataset-specific fields.
scraper.py - Comparison CSVs · Repository-root CSV outputs intended for comparative analysis across keywords.
Connections 14
- scraper.py → Main loop: runs · The executable script starts the batch workflow.
- Main loop → Fetch trends: per keyword · Each keyword triggers one API request.
- Fetch trends → Oxylabs API: POST /queries · Sends authenticated requests to the scraper endpoint.
- Oxylabs API → Decode payload: JSON response · Returns nested response content for Google Trends Explore data.
- Decode payload → Trend datasets: decoded data · Decoded content yields the four logical dataset groups.
- Trend datasets → Flatten topics: related topics · Nested topic records require special flattening before export.
- Trend datasets → DataFrames: other datasets · Each dataset category is converted into a DataFrame for downstream writing.
- Flatten topics → DataFrames: normalized records · Flattened topics are folded into the tabular export pipeline.
- DataFrames → Write keyword CSVs: ready to write · Prepared DataFrames are exported as keyword-scoped CSVs.
- Write keyword CSVs → Keyword outputs: writes · Creates local per-keyword CSV artifacts.
- Main loop → Compare keywords: after batch · Once all keywords are processed, comparison tables are generated.
- Compare keywords → Comparison CSVs: writes · Outputs merged CSVs for cross-keyword analysis.
- README.md → scraper.py: documents · Documentation points to the operational script.
- README.md → images/: references · Documentation includes screenshots stored in the images folder.