# madilekci/scrape-google-search architecture

> Architecture diagram of the GitHub repository madilekci/scrape-google-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/madilekci/scrape-google-search
- Repository: https://github.com/madilekci/scrape-google-search

## Overview

##### Project type and purpose
- This is a CLI-driven Node.js web-scraping script. It uses Puppeteer to fetch Google SERP results, saves the raw HTML, then uses Cheerio to parse that HTML and extract structured JSON data.
- The goal is to produce HTML snapshots and JSON outputs for sections like knowledge graph, paid ads, organic results, etc., all stored under a “results” folder.

##### File structure
- Top-level files
- index.js: orchestrator and entry point
- package.json / package-lock.json: dependency definitions
- README.md: project overview and usage
- helpers/ directory
- get-args.js: CLI argument parsing
- puppeteer-helpers.js: functions to launch browser, navigate, capture HTML
- read-file.js / write-file.js: file I/O abstractions
- results/ directory
- .html files: raw snapshots
- results/json/*.json: parsed output

##### Extract insights from README
- Confirms use of Node.js (v10+) and Puppeteer + Cheerio
- Workflow: keyword comes in → HTML fetch → HTML persist → parse → JSON output

##### Main components
a. CLI Input Module
- Parses command-line flags (keyword)
- get-args.js
b. Scraper Module (Puppeteer)
- launches headless browser
- navigates to Google, performs search
- captures HTML snapshot
c. Storage Module (File System)
- write-file.js: saves HTML to results/*.html
- Directory structure
d. Parser Module (Cheerio)
- read-file.js: loads saved HTML
- cheerio-based extraction logic in index.js or helpers
- produces structured JSON
e. Output Module (File System)
- write-file.js: saves extracted JSON to results/json/*.json

##### Relationships & data flow
- Sequence is strictly linear: CLI → Scraper → HTML Store → Parser → JSON Store

##### Architectural patterns & design principles
- Modular helpers (single-responsibility per file)
- Clear separation between I/O (helpers/read-file, write-file), scraping logic, and parsing logic
- Sequential pipeline rather than event-driven or microservices
- Use of well-encapsulated third-party libraries (Puppeteer, Cheerio)

##### Relevant technologies/frameworks
- Node.js (v10+)
- NPM modules: puppeteer, cheerio, yargs (or custom CLI parser)
- Built-in fs module (wrapped by read-file/write-file)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Components
    subgraph "CLI & Orchestrator"
        cli["CLI Parser\n(get-args.js)"]:::external
        orchestrator["Orchestrator\n(index.js)"]:::process
    end

    subgraph "Scraping Stage"
        scraper["Puppeteer Scraper\n(puppeteer-helpers.js)"]:::process
        writeHTML["Write HTML\n(write-file.js)"]:::process
        htmlStore[(results/*.html)]:::storage
    end

    subgraph "Parsing & Output Stage"
        readHTML["Read HTML\n(read-file.js)"]:::process
        parser["Cheerio Parser\n(index.js)"]:::process
        writeJSON["Write JSON\n(write-file.js)"]:::process
        jsonStore[(results/json/*.json)]:::storage
    end

    %% Data Flow
    cli -->|"--keyword=..."| orchestrator
    orchestrator -->|"keyword"| scraper
    scraper -->|"navigate & fetch HTML"| writeHTML
    writeHTML -->|"save results/keyword.html"| htmlStore
    htmlStore -->|"read HTML"| readHTML
    readHTML -->|"load HTML"| parser
    parser -->|"extract JSON"| writeJSON
    writeJSON -->|"save results/json/keyword.json"| jsonStore

    %% Click Events
    click cli "https://github.com/madilekci/scrape-google-search/blob/main/helpers/get-args.js"
    click orchestrator "https://github.com/madilekci/scrape-google-search/blob/main/index.js"
    click scraper "https://github.com/madilekci/scrape-google-search/blob/main/helpers/puppeteer-helpers.js"
    click writeHTML "https://github.com/madilekci/scrape-google-search/blob/main/helpers/write-file.js"
    click htmlStore "https://github.com/madilekci/scrape-google-search/blob/main/results/nodejs.html"
    click readHTML "https://github.com/madilekci/scrape-google-search/blob/main/helpers/read-file.js"
    click parser "https://github.com/madilekci/scrape-google-search/blob/main/index.js"
    click writeJSON "https://github.com/madilekci/scrape-google-search/blob/main/helpers/write-file.js"
    click jsonStore "https://github.com/madilekci/scrape-google-search/blob/main/results/json/nodejs.json"

    %% Styles
    classDef external fill:#a2d5f2,stroke:#0b3d91,stroke-width:2px
    classDef process fill:#f8b195,stroke:#c94c4c,stroke-width:2px
    classDef storage fill:#c3aed6,stroke:#6a0572,stroke-width:2px
```
