How barr1qs/pcpartpicker-scraper works
PCPartPicker Full Hardware Scraper is a CLI-driven catalog scraper for six hardware categories. The workflow configures categories and runtime options, opens a persistent Playwright browser, navigates category pages while handling Cloudflare and page responses, parses product rows, checkpoints progress, and exports JSONL, JSON, and CSV. The graph retains the documented browser countermeasures and resumable storage capabilities; connections are limited to calls visible in the excerpts or README.
Main entry points: scraper.py Scrape orchestrator · storage.py Batch exporter · config.py Category config · main.py CLI options
Components 12
CLI entry
- CLI options
main.py
Scraping runtime
- Category config
pcpartpicker_scraper/config.py - Scrape orchestrator
pcpartpicker_scraper/scraper.py - Persistent browser
pcpartpicker_scraper/browser.py - Turnstile resolver
pcpartpicker_scraper/browser.py - Product parser
pcpartpicker_scraper/parser.py
Persistence and exports
- Checkpoint manager
pcpartpicker_scraper/storage.py - JSONL stream
pcpartpicker_scraper/storage.py - Batch exporter
pcpartpicker_scraper/storage.py - JSON and CSV
pcpartpicker_scraper/storage.py
Other components
- CLI user
- PCPartPicker
Connections 16
- CLI user → CLI options: sets options
main.py - CLI options → Category config: reads categories
main.py - CLI options → Scrape orchestrator: starts scrape
main.py - Scrape orchestrator → Category config: reads category settings
pcpartpicker_scraper/scraper.py - Scrape orchestrator → Persistent browser: opens context
pcpartpicker_scraper/scraper.py - Scrape orchestrator → Turnstile resolver: checks challenge
pcpartpicker_scraper/scraper.py - Scrape orchestrator → PCPartPicker: navigates and requests pages
pcpartpicker_scraper/scraper.py - PCPartPicker → Scrape orchestrator: returns catalog data
pcpartpicker_scraper/scraper.py - Scrape orchestrator → Product parser: parses response HTML
pcpartpicker_scraper/scraper.py - Product parser → Category config: reads spec keys
pcpartpicker_scraper/parser.py - Scrape orchestrator → Checkpoint manager: checks and records progress
pcpartpicker_scraper/scraper.py - Scrape orchestrator → JSONL stream: appends products
pcpartpicker_scraper/scraper.py - Scrape orchestrator → Batch exporter: exports category
pcpartpicker_scraper/scraper.py - Batch exporter → JSONL stream: reads stream
pcpartpicker_scraper/storage.py - Batch exporter → Category config: reads CSV spec keys
pcpartpicker_scraper/storage.py - Batch exporter → JSON and CSV: writes JSON and CSV
pcpartpicker_scraper/storage.py