How madilekci/scrape-google-search works
Project type and purpose
• This is a CLI-driven Node.js web-scraping script. It uses Puppeteer to fetch Google SERP results, saves the raw HTML, then uses Cheerio to parse that HTML and extract structured JSON data.
• The goal is to produce HTML snapshots and JSON outputs for sections like knowledge graph, paid ads, organic results, etc., all stored under a “results” folder.
File structure
• Top-level files
• index.js: orchestrator and entry point
• package.json / package-lock.json: dependency definitions
• README.md: project overview and usage
• helpers/ directory
• get-args.js: CLI argument parsing
• puppeteer-helpers.js: functions to launch browser, navigate, capture HTML
• read-file.js / write-file.js: file I/O abstractions
• results/ directory
• .html files: raw snapshots
• results/json/*.json: parsed output
Extract insights from README
• Confirms use of Node.js (v10+) and Puppeteer + Cheerio
• Workflow: keyword comes in → HTML fetch → HTML persist → parse → JSON output
Main components
a. CLI Input Module
• Parses command-line flags (keyword)
• get-args.js
b. Scraper Module (Puppeteer)
• launches headless browser
• navigates to Google, performs search
• captures HTML snapshot
c. Storage Module (File System)
• write-file.js: saves HTML to results/*.html
• Directory structure
d. Parser Module (Cheerio)
• read-file.js: loads saved HTML
• cheerio-based extraction logic in index.js or helpers
• produces structured JSON
e. Output Module (File System)
• write-file.js: saves extracted JSON to results/json/*.json
Relationships & data flow
• Sequence is strictly linear: CLI → Scraper → HTML Store → Parser → JSON Store
Architectural patterns & design principles
• Modular helpers (single-responsibility per file)
• Clear separation between I/O (helpers/read-file, write-file), scraping logic, and parsing logic
• Sequential pipeline rather than event-driven or microservices
• Use of well-encapsulated third-party libraries (Puppeteer, Cheerio)
Relevant technologies/frameworks
• Node.js (v10+)
• NPM modules: puppeteer, cheerio, yargs (or custom CLI parser)
• Built-in fs module (wrapped by read-file/write-file)