How ixdi/cqscraper works
Project Type and Purpose
• This is a Python‐based command‐line web scraper (an open‐source tool) whose job is to:
• Product URLs from a seed page (“Lister”)
• Crawl each product page to extract details (“Crawler”)
• Orchestrate the two and write results to a JSON file (“ProductScraper”)
High‑Level File Structure
• src/cqscraper/
• __main__.py – Entry point for the CLI (“python -m cqscraper”)
• confs.py – Configuration (URLs, output file name, defaults)
• lister.py – Fetches and parses the listing page via requests + lxml
• crawler.py – Fetches each product page, parses with BeautifulSoup
• product_scraper.py – Orchestrator that launches Lister + multiple Crawler tasks
• helpers.py – Utility routines (image processing, PDF extraction)
• tests/ – Unit tests for lister, etc.
• CI/config files – tox.ini, GitHub Actions workflows
Key Components
a. CLI Layer
• cqscraper CLI entry point (maps to __main__.py)
• Accepts runtime arguments (e.g. number_of_crawlers)
b. Orchestration Layer
• ProductScraper class
• Reads Confs, spawns Lister, then spawns multiple Crawler instances (thread/process pool)
c. Lister Component
• HTTP request to base URL (requests library)
• HTML parsing with lxml
• Outputs list of product URLs
d. Crawler Component
• Fetch each product URL (requests)
• Parse HTML with BeautifulSoup
• Extract fields (title, price, images, specs)
• Use Helpers for image/PDF tasks
• Return structured record
e. Helpers
• Image downloading, resizing, PDF extraction, etc.
f. Data Storage
• Aggregator in ProductScraper collects all records
• Writes a single JSON file to disk
g. External Dependencies
• External web server(s) hosting product pages
• Python libraries: requests, lxml, BeautifulSoup, json
Relationships & Data Flow
• User → CLI
• CLI → ProductScraper
• ProductScraper → Confs (read settings)
• ProductScraper → Lister → Web Server → Lister → URL list
• ProductScraper → Crawler pool (multiple) → Web Server → Crawler → Data records
• Crawler → Helpers (for images/PDF)
• ProductScraper → JSON output
Architectural Patterns & Principles
• Modular single‐responsibility classes (Lister, Crawler, Helpers)
• Orchestrator pattern in ProductScraper
• Pluggable configuration via confs.py
• Concurrency (configurable number_of_crawlers)
• Clear separation of I/O (HTTP, file system) vs. parsing logic