# barr1qs/pcpartpicker-scraper architecture

> Architecture diagram of the GitHub repository barr1qs/pcpartpicker-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/barr1qs/pcpartpicker-scraper
- Repository: https://github.com/barr1qs/pcpartpicker-scraper

## Overview

PCPartPicker Full Hardware Scraper is a CLI-driven catalog scraper for six hardware categories. The workflow configures categories and runtime options, opens a persistent Playwright browser, navigates category pages while handling Cloudflare and page responses, parses product rows, checkpoints progress, and exports JSONL, JSON, and CSV. The graph retains the documented browser countermeasures and resumable storage capabilities; connections are limited to calls visible in the excerpts or README.

Main entry points: [`pcpartpicker_scraper/scraper.py`](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/scraper.py) (Scrape orchestrator), [`pcpartpicker_scraper/storage.py`](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py) (Batch exporter), [`pcpartpicker_scraper/config.py`](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/config.py) (Category config), [`main.py`](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/main.py) (CLI options)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_entry["CLI entry"]
  node_main["CLI options<br/>[main.py]"]
end

subgraph group_scraping["Scraping runtime"]
  node_config["Category config<br/>[config.py]"]
  node_orchestrator["Scrape orchestrator<br/>[scraper.py]"]
  node_browser["Persistent browser<br/>[browser.py]"]
  node_challenge["Turnstile resolver<br/>[browser.py]"]
  node_parser["Product parser<br/>[parser.py]"]
end

subgraph group_outputs["Persistence and exports"]
  node_state[("Checkpoint manager<br/>[storage.py]")]
  node_jsonl["JSONL stream<br/>[storage.py]"]
  node_exporter["Batch exporter<br/>[storage.py]"]
  node_files["JSON and CSV<br/>[storage.py]"]
end

node_user(("CLI user"))
node_site{{"PCPartPicker"}}

node_user -->|"sets options"| node_main
node_main -->|"reads categories"| node_config
node_main -->|"starts scrape"| node_orchestrator
node_orchestrator -->|"reads category settings"| node_config
node_orchestrator -->|"opens context"| node_browser
node_orchestrator -->|"checks challenge"| node_challenge
node_orchestrator -->|"navigates and requests pages"| node_site
node_site -->|"returns catalog data"| node_orchestrator
node_orchestrator -->|"parses response HTML"| node_parser
node_parser -->|"reads spec keys"| node_config
node_orchestrator -->|"checks and records progress"| node_state
node_orchestrator -->|"appends products"| node_jsonl
node_orchestrator -->|"exports category"| node_exporter
node_exporter -->|"reads stream"| node_jsonl
node_exporter -->|"reads CSV spec keys"| node_config
node_exporter -->|"writes JSON and CSV"| node_files

click node_main "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/main.py"
click node_config "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/config.py"
click node_orchestrator "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/scraper.py"
click node_browser "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/browser.py"
click node_challenge "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/browser.py"
click node_parser "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/parser.py"
click node_state "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py"
click node_jsonl "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py"
click node_exporter "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py"
click node_files "https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main,node_user toneBlue
class node_config,node_orchestrator,node_browser,node_challenge,node_parser toneAmber
class node_state,node_jsonl,node_exporter,node_files toneMint
class node_site toneIndigo
```

## Components

### CLI entry

- [CLI options](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/main.py): `main.py`

### Scraping runtime

- [Category config](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/config.py): `pcpartpicker_scraper/config.py`
- [Scrape orchestrator](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/scraper.py): `pcpartpicker_scraper/scraper.py`
- [Persistent browser](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/browser.py): `pcpartpicker_scraper/browser.py`
- [Turnstile resolver](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/browser.py): `pcpartpicker_scraper/browser.py`
- [Product parser](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/parser.py): `pcpartpicker_scraper/parser.py`

### Persistence and exports

- [Checkpoint manager](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py): `pcpartpicker_scraper/storage.py`
- [JSONL stream](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py): `pcpartpicker_scraper/storage.py`
- [Batch exporter](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py): `pcpartpicker_scraper/storage.py`
- [JSON and CSV](https://github.com/barr1qs/pcpartpicker-scraper/blob/master/pcpartpicker_scraper/storage.py): `pcpartpicker_scraper/storage.py`

### Other components

- **CLI user**
- **PCPartPicker**

## Connections

- CLI user → CLI options: sets options (evidence: `main.py`)
- CLI options → Category config: reads categories (evidence: `main.py`)
- CLI options → Scrape orchestrator: starts scrape (evidence: `main.py`)
- Scrape orchestrator → Category config: reads category settings (evidence: `pcpartpicker_scraper/scraper.py`)
- Scrape orchestrator → Persistent browser: opens context (evidence: `pcpartpicker_scraper/scraper.py`)
- Scrape orchestrator → Turnstile resolver: checks challenge (evidence: `pcpartpicker_scraper/scraper.py`)
- Scrape orchestrator → PCPartPicker: navigates and requests pages (evidence: `pcpartpicker_scraper/scraper.py`)
- PCPartPicker → Scrape orchestrator: returns catalog data (evidence: `pcpartpicker_scraper/scraper.py`)
- Scrape orchestrator → Product parser: parses response HTML (evidence: `pcpartpicker_scraper/scraper.py`)
- Product parser → Category config: reads spec keys (evidence: `pcpartpicker_scraper/parser.py`)
- Scrape orchestrator → Checkpoint manager: checks and records progress (evidence: `pcpartpicker_scraper/scraper.py`)
- Scrape orchestrator → JSONL stream: appends products (evidence: `pcpartpicker_scraper/scraper.py`)
- Scrape orchestrator → Batch exporter: exports category (evidence: `pcpartpicker_scraper/scraper.py`)
- Batch exporter → JSONL stream: reads stream (evidence: `pcpartpicker_scraper/storage.py`)
- Batch exporter → Category config: reads CSV spec keys (evidence: `pcpartpicker_scraper/storage.py`)
- Batch exporter → JSON and CSV: writes JSON and CSV (evidence: `pcpartpicker_scraper/storage.py`)
