# ixdi/cqscraper architecture

> Architecture diagram of the GitHub repository ixdi/cqscraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ixdi/cqscraper
- Repository: https://github.com/ixdi/cqscraper

## Overview

##### Project Type and Purpose
- This is a Python‐based command‐line web scraper (an open‐source tool) whose job is to:
- Product URLs from a seed page (“Lister”)
- Crawl each product page to extract details (“Crawler”)
- Orchestrate the two and write results to a JSON file (“ProductScraper”)

##### High‑Level File Structure
- src/cqscraper/
- __main__.py – Entry point for the CLI (“python -m cqscraper”)
- confs.py – Configuration (URLs, output file name, defaults)
- lister.py – Fetches and parses the listing page via requests + lxml
- crawler.py – Fetches each product page, parses with BeautifulSoup
- product_scraper.py – Orchestrator that launches Lister + multiple Crawler tasks
- helpers.py – Utility routines (image processing, PDF extraction)
- tests/ – Unit tests for lister, etc.
- CI/config files – tox.ini, GitHub Actions workflows

##### Key Components
a. CLI Layer
- `cqscraper` CLI entry point (maps to __main__.py)
- Accepts runtime arguments (e.g. number_of_crawlers)
b. Orchestration Layer
- ProductScraper class
- Reads Confs, spawns Lister, then spawns multiple Crawler instances (thread/process pool)
c. Lister Component
- HTTP request to base URL (requests library)
- HTML parsing with lxml
- Outputs list of product URLs
d. Crawler Component
- Fetch each product URL (requests)
- Parse HTML with BeautifulSoup
- Extract fields (title, price, images, specs)
- Use Helpers for image/PDF tasks
- Return structured record
e. Helpers
- Image downloading, resizing, PDF extraction, etc.
f. Data Storage
- Aggregator in ProductScraper collects all records
- Writes a single JSON file to disk
g. External Dependencies
- External web server(s) hosting product pages
- Python libraries: requests, lxml, BeautifulSoup, json

##### Relationships & Data Flow
- User → CLI
- CLI → ProductScraper
- ProductScraper → Confs (read settings)
- ProductScraper → Lister → Web Server → Lister → URL list
- ProductScraper → Crawler pool (multiple) → Web Server → Crawler → Data records
- Crawler → Helpers (for images/PDF)
- ProductScraper → JSON output

##### Architectural Patterns & Principles
- Modular single‐responsibility classes (Lister, Crawler, Helpers)
- Orchestrator pattern in ProductScraper
- Pluggable configuration via confs.py
- Concurrency (configurable number_of_crawlers)
- Clear separation of I/O (HTTP, file system) vs. parsing logic

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interaction
    User((User)):::external
    User --> CLI

    %% CLI Layer
    subgraph "CLI Layer"
        CLI["CLI (__main__.py)"]:::internal
    end
    click CLI "https://github.com/ixdi/cqscraper/blob/main/src/cqscraper/__main__.py"

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        ProductScraper["ProductScraper"]:::internal
        Confs["Confs"]:::internal
    end
    click ProductScraper "https://github.com/ixdi/cqscraper/blob/main/src/cqscraper/product_scraper.py"
    click Confs "https://github.com/ixdi/cqscraper/blob/main/src/cqscraper/confs.py"

    CLI --> ProductScraper
    ProductScraper --> Confs

    %% Lister Component
    subgraph "Lister Component"
        Lister["Lister"]:::internal
    end
    click Lister "https://github.com/ixdi/cqscraper/blob/main/src/cqscraper/lister.py"

    ProductScraper --> Lister
    Lister --> WebServer
    WebServer --> Lister
    Lister -->|"List<URL>"| ProductScraper

    %% Crawler Pool and Component
    subgraph "Crawler Pool"
        CrawlerPool["Pool of N Crawlers"]:::internal
        Crawler["Crawler"]:::internal
    end
    click Crawler "https://github.com/ixdi/cqscraper/blob/main/src/cqscraper/crawler.py"

    ProductScraper --> CrawlerPool
    CrawlerPool --> Crawler
    Crawler --> WebServer
    WebServer --> Crawler
    Crawler -->|"Record"| ProductScraper

    %% Helpers Module
    Helpers["Helpers"]:::internal
    click Helpers "https://github.com/ixdi/cqscraper/blob/main/src/cqscraper/helpers.py"
    Crawler --> Helpers

    %% External Web Server
    WebServer((External Web Server)):::external

    %% Data Storage
    JSONWriter(("JSON File")):::datastore
    ProductScraper --> JSONWriter

    %% Project Config and CI
    subgraph "Project Configuration"
        Packaging["pyproject.toml\nrequirements.txt\nMANIFEST.in"]:::internal
        CIYml[".github/workflows/ci.yml"]:::internal
        Tox["tox.ini"]:::internal
    end
    click CIYml "https://github.com/ixdi/cqscraper/blob/main/.github/workflows/ci.yml"
    click Tox "https://github.com/ixdi/cqscraper/blob/main/tox.ini"

    %% Testing
    TestLister["Unit Tests (test_lister.py)"]:::internal
    click TestLister "https://github.com/ixdi/cqscraper/blob/main/tests/test_lister.py"
    TestLister --> Lister

    %% Relations
    Confs -.-> Packaging
    Confs -.-> CIYml
    Confs -.-> Tox

    %% Styles
    classDef internal fill:#D0E8FF,stroke:#0366d6,color:#0366d6
    classDef external fill:#D0FFD0,stroke:#228B22,color:#228B22
    classDef datastore fill:#FFE0B2,stroke:#FF8C00,color:#FF8C00
```
