# scrapfly/scrapfly-scrapers architecture

> Architecture diagram of the GitHub repository scrapfly/scrapfly-scrapers, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scrapfly/scrapfly-scrapers
- Repository: https://github.com/scrapfly/scrapfly-scrapers

## Overview

This repo is a **monorepo of many small, standalone Python scraper projects** (“example scrapers”) that all demonstrate the same scraping stack:
- Python 3.10+
- ScrapFly Web Scraping API via the ScrapFly Python SDK (HTTP fetching + anti-bot bypass)
- `asyncio` concurrency
- HTML parsing via ScrapFly SDK’s built-in `parsel` selectors
- JSON extraction via `jmespath` + `nested-lookup` (when targets expose JSON blobs)
- Logging via `loguru`
- Outputs saved as **sample datasets** under each scraper’s `results/`

So the “system” is not a single deployed service; it’s a **suite of CLI-run scrapers** that interact with **ScrapFly (external SaaS)** and **target websites** and produce **JSON artifacts**.

- “Developer runs scraper locally/CI”
- “Scraper calls ScrapFly API”
- “ScrapFly fetches target site(s)”
- “Scraper parses + writes JSON outputs”

##### Repository structure signals (how to decompose components)
From the file tree:
- Top-level: many sibling directories like `amazon-scraper/`, `google-scraper/`, `zillow-scraper/` etc.
- Each scraper directory typically contains:
- `README.md` (scraper-specific usage/notes)
- `<domain>.py` (core scraping logic; e.g., `amazon.py`, `google.py`)
- `run.py` (entrypoint / CLI-style runner)
- `test.py` (basic validation / smoke tests)
- `pyproject.toml` (packaging + dependencies per scraper)
- `results/` (example output JSON files)
- sometimes `snippets/` (small focused scripts demonstrating sub-tasks)
- Repo-level `.github/workflows/test_scrapers.yaml`: CI that likely executes tests across scrapers.

This implies two “architecture levels” worth diagramming:

##### Developer/CI → `run.py`
- “invoke” (CLI execution, parameters/seed URLs)
##### `run.py` → `<x>.py`
- “call scraper functions / orchestrate tasks”
##### `<x>.py` → `asyncio`
- “spawn concurrent tasks” (gather, semaphore throttling if present)
##### `<x>.py` → ScrapFly SDK
- “ScrapFlyClient.scrape(...)”
##### ScrapFly SDK → ScrapFly API (HTTPS)
- “HTTPS requests (ScrapFly scrape requests)”
##### ScrapFly API → Target Website(s)
- “fetch pages / render / bypass blocks” (conceptual)
##### Target Website(s) → ScrapFly API → ScrapFly SDK
- “HTML/JSON response” (often HTML with embedded JSON)
##### ScrapFly SDK / `<x>.py` → Parsing layer
- “parsel selectors” for DOM extraction
- “jmespath/nested-lookup” for JSON blob extraction
##### Parsing layer → `<x>.py` (data model / normalization)
- “structured dict/list objects”
##### `<x>.py` / `run.py` → `results/*.json`
- “write sample dataset”
##### loguru spans across `run.py` and `<x>.py` as cross-cutting
- “logs” to console/file

##### CI flow (repo-level)
- **GitHub Actions** → runs matrix of scrapers/tests
- **Workflow** → `python -m …` / `pytest` or `python test.py`
- **Artifacts** (optional): test logs, maybe generated result JSON

This repo is best represented as a **repeated template architecture** per scraper:
- **Orchestrator** (`run.py`)
- **Domain-specific scraper module** (`amazon.py`, `zillow.py`, etc.)
- **External fetch provider** (ScrapFly API)
- **Parse + transform** (parsel + JSON tools)
- **Output adapters** (write JSON to `results/`)

Also call out two cross-cutting concerns:
- **Concurrency** via `asyncio` (important for throughput and for “best practice” scraping)
- **Observability** via `loguru`

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% =========================
    %% Actors / Runtime
    %% =========================
    Dev["Developer Machine\n(Python 3.10+, CLI)"]:::actor
    CI["GitHub Actions Runner\n(Matrix across scrapers)"]:::actor

    %% =========================
    %% Repository (Monorepo)
    %% =========================
    subgraph "ScrapFly Scrapers Monorepo (Suite of standalone CLI scrapers)"
        RepoReadme["Repo README.md\n(Suite overview & usage)"]:::doc

        subgraph "CI / Automation"
            Workflow["test_scrapers.yaml\n(Run tests across scrapers)"]:::ci
        end

        subgraph "Scraper Packages (N instances; repeated template)"
            OtherScrapers["Other scraper folders\n(e.g., google-scraper/, zillow-scraper/, ...)"]:::pkg

            subgraph "Representative Scraper: amazon-scraper/"
                A_Readme["Per-scraper README.md\n(Usage, notes)"]:::doc
                A_PyProject["pyproject.toml\n(Deps per scraper)"]:::config

                subgraph "Runtime Entry + Core Logic"
                    A_Run["run.py\n(CLI orchestration,\nseed inputs, writes outputs)"]:::code
                    A_Core["amazon.py\n(URL builders, pagination,\nparsers, normalization)"]:::code
                end

                subgraph "Quality & Examples"
                    A_Test["test.py\n(Smoke validation)"]:::test
                    A_Snippets["snippets/\n(Focused examples)"]:::aux
                    A_SnippetsReadme["snippets/README.md"]:::doc
                end

                subgraph "Artifacts"
                    A_Results["results/\n(Sample datasets)"]:::store
                    A_ResultsJson["results/*.json\n(Normalized extracted data)"]:::data
                end
            end
        end
    end

    %% =========================
    %% Libraries (Cross-cutting)
    %% =========================
    subgraph "Scraper Runtime Libraries"
        Asyncio(["asyncio\n(concurrency, tasks, gather,\nthrottling)"]):::lib
        ScrapflySDK(["ScrapFly Python SDK\n(client + response abstraction\n+ parsel integration)"]):::lib
        Parsel(["parsel selectors\n(HTML/DOM extraction)"]):::lib
        JMES(["jmespath + nested-lookup\n(JSON blob traversal)"]):::lib
        Loguru(["loguru\n(logging / observability)"]):::lib
    end

    %% =========================
    %% External Systems
    %% =========================
    ScrapflyAPI(("ScrapFly API (SaaS)\nanti-bot bypass, fetching,\noptional rendering")):::external
    subgraph "Target Websites (Multiple Domains)"
        TargetSites(("Target Site(s)\n(Amazon/Google/Zillow/...)\nHTML/JSON responses")):::external
        WAF["Anti-bot / WAF / Rate limits\n(constraints)"]:::constraint
    end

    %% =========================
    %% Repo-level / CI Flow
    %% =========================
    Dev -->|"read_docs"| RepoReadme
    CI -->|"checkout_repo"| RepoReadme
    CI -->|"run_workflow"| Workflow
    Workflow -->|"execute_tests"| A_Test
    Workflow -->|"execute_tests"| OtherScrapers

    %% =========================
    %% Per-scraper Primary Data Flow
    %% =========================
    Dev -->|"invoke_cli"| A_Run
    CI -->|"invoke_cli"| A_Run

    A_Run -->|"orchestrate_calls"| A_Core
    A_Run -->|"configure_deps"| A_PyProject

    A_Core -->|"spawn_tasks"| Asyncio
    A_Core -->|"ScrapFlyClient.scrape(...)"| ScrapflySDK
    ScrapflySDK -->|"HTTPS_scrape_request"| ScrapflyAPI
    ScrapflyAPI -->|"fetch_pages_render_bypass"| TargetSites
    WAF -.->|"blocks_throttle_challenge"| TargetSites

    TargetSites -.->|"HTML_or_JSON"| ScrapflyAPI
    ScrapflyAPI -.->|"HTTP_response"| ScrapflySDK
    ScrapflySDK -.->|"response_object"| A_Core

    A_Core -->|"DOM_extract"| Parsel
    A_Core -->|"JSON_extract"| JMES
    Parsel -.->|"extracted_fields"| A_Core
    JMES -.->|"extracted_objects"| A_Core

    A_Core -->|"write_JSON_files"| A_Results
    A_Results -->|"persist"| A_ResultsJson

    %% Cross-cutting logs
    A_Run -->|"logs"| Loguru
    A_Core -->|"logs"| Loguru
    A_Test -->|"logs"| Loguru
    Loguru -.->|"stdout/stderr"| Dev
    Loguru -.->|"job_logs"| CI

    %% Optional: snippets as side branch
    A_Snippets -.->|"examples_call"| A_Core
    A_SnippetsReadme -.->|"documents"| A_Snippets

    %% =========================
    %% Click Events (from component_mapping)
    %% =========================
    click RepoReadme "https://github.com/scrapfly/scrapfly-scrapers/blob/main/README.md"
    click Workflow "https://github.com/scrapfly/scrapfly-scrapers/blob/main/.github/workflows/test_scrapers.yaml"

    click OtherScrapers "https://github.com/scrapfly/scrapfly-scrapers/tree/main/*/"

    click A_Readme "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/README.md"
    click A_Run "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/run.py"
    click A_Core "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/amazon.py"
    click A_Test "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/test.py"
    click A_PyProject "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/pyproject.toml"
    click A_Results "https://github.com/scrapfly/scrapfly-scrapers/tree/main/amazon-scraper/results/"
    click A_ResultsJson "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/results/product.json"
    click A_Snippets "https://github.com/scrapfly/scrapfly-scrapers/tree/main/amazon-scraper/snippets/"
    click A_SnippetsReadme "https://github.com/scrapfly/scrapfly-scrapers/blob/main/amazon-scraper/snippets/README.md"

    %% =========================
    %% Styles (color is critical)
    %% =========================
    classDef actor fill:#E3F2FD,stroke:#1565C0,stroke-width:2px,color:#0D47A1
    classDef ci fill:#E8EAF6,stroke:#3949AB,stroke-width:2px,color:#1A237E
    classDef pkg fill:#FFF3E0,stroke:#EF6C00,stroke-width:2px,color:#E65100
    classDef code fill:#E8F5E9,stroke:#2E7D32,stroke-width:2px,color:#1B5E20
    classDef test fill:#F1F8E9,stroke:#558B2F,stroke-width:2px,color:#33691E
    classDef aux fill:#F9FBE7,stroke:#9E9D24,stroke-width:2px,color:#827717
    classDef lib fill:#F3E5F5,stroke:#6A1B9A,stroke-width:2px,color:#4A148C
    classDef external fill:#FFEBEE,stroke:#C62828,stroke-width:2px,color:#B71C1C
    classDef constraint fill:#FFFDE7,stroke:#F9A825,stroke-width:2px,color:#F57F17
    classDef store fill:#E0F7FA,stroke:#00838F,stroke-width:2px,color:#006064
    classDef data fill:#E0F2F1,stroke:#00796B,stroke-width:2px,color:#004D40
    classDef doc fill:#FAFAFA,stroke:#616161,stroke-width:2px,color:#212121
    classDef config fill:#F5F5F5,stroke:#455A64,stroke-width:2px,color:#263238
```
