How scrapfly/scrapfly-scrapers works
This repo is a monorepo of many small, standalone Python scraper projects (“example scrapers”) that all demonstrate the same scraping stack:
• Python 3.10+
• ScrapFly Web Scraping API via the ScrapFly Python SDK (HTTP fetching + anti-bot bypass)
• asyncio concurrency
• HTML parsing via ScrapFly SDK’s built-in parsel selectors
• JSON extraction via jmespath + nested-lookup (when targets expose JSON blobs)
• Logging via loguru
• Outputs saved as sample datasets under each scraper’s results/
So the “system” is not a single deployed service; it’s a suite of CLI-run scrapers that interact with ScrapFly (external SaaS) and target websites and produce JSON artifacts.
• “Developer runs scraper locally/CI”
• “Scraper calls ScrapFly API”
• “ScrapFly fetches target site(s)”
• “Scraper parses + writes JSON outputs”
Repository structure signals (how to decompose components)
From the file tree:
• Top-level: many sibling directories like amazon-scraper/, google-scraper/, zillow-scraper/ etc.
• Each scraper directory typically contains:
• README.md (scraper-specific usage/notes)
• <domain>.py (core scraping logic; e.g., amazon.py, google.py)
• run.py (entrypoint / CLI-style runner)
• test.py (basic validation / smoke tests)
• pyproject.toml (packaging + dependencies per scraper)
• results/ (example output JSON files)
• sometimes snippets/ (small focused scripts demonstrating sub-tasks)
• Repo-level .github/workflows/test_scrapers.yaml: CI that likely executes tests across scrapers.
This implies two “architecture levels” worth diagramming:
Developer/CI → run.py
• “invoke” (CLI execution, parameters/seed URLs)
run.py → <x>.py
• “call scraper functions / orchestrate tasks”
<x>.py → asyncio
• “spawn concurrent tasks” (gather, semaphore throttling if present)
<x>.py → ScrapFly SDK
• “ScrapFlyClient.scrape(...)”
ScrapFly SDK → ScrapFly API (HTTPS)
• “HTTPS requests (ScrapFly scrape requests)”
ScrapFly API → Target Website(s)
• “fetch pages / render / bypass blocks” (conceptual)
Target Website(s) → ScrapFly API → ScrapFly SDK
• “HTML/JSON response” (often HTML with embedded JSON)
ScrapFly SDK / <x>.py → Parsing layer
• “parsel selectors” for DOM extraction
• “jmespath/nested-lookup” for JSON blob extraction
Parsing layer → <x>.py (data model / normalization)
• “structured dict/list objects”
<x>.py / run.py → results/*.json
• “write sample dataset”
loguru spans across run.py and <x>.py as cross-cutting
• “logs” to console/file
CI flow (repo-level)
• GitHub Actions → runs matrix of scrapers/tests
• Workflow → python -m … / pytest or python test.py
• Artifacts (optional): test logs, maybe generated result JSON
This repo is best represented as a repeated template architecture per scraper:
• Orchestrator (run.py)
• Domain-specific scraper module (amazon.py, zillow.py, etc.)
• External fetch provider (ScrapFly API)
• Parse + transform (parsel + JSON tools)
• Output adapters (write JSON to results/)
Also call out two cross-cutting concerns:
• Concurrency via asyncio (important for throughput and for “best practice” scraping)
• Observability via loguru