# rmax/scrapy-redis architecture

> Architecture diagram of the GitHub repository rmax/scrapy-redis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rmax/scrapy-redis
- Repository: https://github.com/rmax/scrapy-redis

## Overview

##### Project type and purpose
- This is not an application per se but a Python library/plugin (“scrapy-redis”) that extends the Scrapy web-crawling framework with Redis-backed components.
- Its purpose is to enable distributed crawling and distributed item processing by decoupling Scrapy spiders from in-memory queues and offering Redis as a central broker.

##### File structure
- src/scrapy_redis/: the core library modules (scheduler, queue, dupefilter, pipelines, spiders, utils, connection).
- example-project/: a sample Scrapy project that shows how to wire in the scrapy-redis components (settings.py, spiders/, pipelines.py).
- docs/: user documentation.
- tests/: unit tests covering each redis-based component.
- Dockerfile & docker-compose.yaml: environment for running Redis + example crawlers.
- CI configs (.github/workflows, .travis.yml, tox.ini): build/test automation.

##### README for additional insights
- Main features: “Scheduler + Duplication Filter, Item Pipeline, Base Spiders.”
- Distributed crawling: multiple Scrapy instances share a single Redis request queue.
- Distributed post-processing: scraped items are enqueued in Redis for any number of consumers.
- Technologies: Python ≥3.7, Redis ≥5.0, Scrapy ≥2.0, redis-py ≥4.0.

##### Elements
- Spider → Redis (enqueue new requests, items)
- Redis → Spider (dequeue next URL)
- Redis → Post-Processor (dequeue items)
- Blue for Scrapy-Redis library modules (Scheduler, DupeFilter, Pipeline)
- Orange for external systems (Redis server, target websites)
- Green for user-provided code (example spiders, post-processors)
- Notes or annotations on key data structures (Redis list “scrapy:requests”, set “scrapy:dupefilter”, list “scrapy:items”).

##### Detailed callouts & layering
- Layer 1 (top): External target websites
- Layer 2: Scrapy Spider Instances (show internal flow: Spider → Scheduler → Queue → DupeFilter → fetch)
- Layer 4: Post-processing consumers
- Optional lateral layer: Management (Docker-Compose orchestration, scaling up/down spider replicas)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Layer 1
    Websites["Target Websites"]:::external

    %% Layer 2: Spider Instances and Modules
    Spider["Scrapy Spider Instances"]:::user
    Websites -->|"Respond to Crawlers"| Spider

    subgraph "Scrapy-Redis Modules"
        Scheduler["RedisScheduler"]:::library
        DupeFilter["RFPDupeFilter"]:::library
        QueueComp["RedisQueue"]:::library
        PipelineComp["RedisPipeline"]:::library
        Connection["Connection Manager"]:::library
        Defaults["Default Settings"]:::library
        SpidersMod["Base Spiders"]:::library
        Utils["Utilities"]:::library
        Pickle["Pickle Compatibility Layer"]:::library
    end

    Spider -->|"Enqueue URLs (LPUSH)"| Scheduler
    Scheduler -->|"Use Queue"| QueueComp
    QueueComp -->|"Check Duplication"| DupeFilter
    DupeFilter -->|"Pass to Spider"| Spider
    Spider -->|"Push Items"| PipelineComp

    %% Layer 3: Redis Server
    subgraph "Redis Server"
        Requests["requests list\n(scrapy:requests)"]:::external
        DupeSet["dupefilter set\n(scrapy:dupefilter)"]:::external
        ItemsList["items list\n(scrapy:items)"]:::external
    end

    Spider -->|"BLPOP / BRPOP"| Requests
    Requests -->|"LPUSH"| Scheduler
    Scheduler -.->|"LPUSH"| Requests
    PipelineComp -->|"LPUSH"| ItemsList
    Requests -->|"SADD / SISMEMBER"| DupeSet
    DupeFilter -->|"SADD"| DupeSet

    %% Layer 4: Post-Processing
    Post["Post-Processing Workers"]:::user
    ItemsList -->|"BLPOP"| Post

    %% Example Project Usage
    subgraph "Example Project"
        ESP["example/settings.py"]:::user
        EPP["example/pipelines.py"]:::user
        ESPdr["docker-compose.yml"]:::user
        ES1["dmoz.py"]:::user
        ES2["mycrawler_redis.py"]:::user
        ES3["myspider_redis.py"]:::user
    end

    click ESP "https://github.com/rmax/scrapy-redis/blob/master/example-project/example/settings.py"
    click EPP "https://github.com/rmax/scrapy-redis/blob/master/example-project/example/pipelines.py"
    click ESPdr "https://github.com/rmax/scrapy-redis/blob/master/example-project/docker-compose.yml"
    click ES1 "https://github.com/rmax/scrapy-redis/blob/master/example-project/example/spiders/dmoz.py"
    click ES2 "https://github.com/rmax/scrapy-redis/blob/master/example-project/example/spiders/mycrawler_redis.py"
    click ES3 "https://github.com/rmax/scrapy-redis/blob/master/example-project/example/spiders/myspider_redis.py"

    %% Documentation & Tests
    subgraph "Documentation & Tests"
        Docs["docs/scrapy_redis.rst"]:::external
        T1["test_scrapy_redis.py"]:::library
        T2["test_dupefilter.py"]:::library
        T3["test_queue.py"]:::library
        T4["test_connection.py"]:::library
        T5["test_utils.py"]:::library
    end

    click Docs "https://github.com/rmax/scrapy-redis/blob/master/docs/scrapy_redis.rst"
    click T1 "https://github.com/rmax/scrapy-redis/blob/master/tests/test_scrapy_redis.py"
    click T2 "https://github.com/rmax/scrapy-redis/blob/master/tests/test_dupefilter.py"
    click T3 "https://github.com/rmax/scrapy-redis/blob/master/tests/test_queue.py"
    click T4 "https://github.com/rmax/scrapy-redis/blob/master/tests/test_connection.py"
    click T5 "https://github.com/rmax/scrapy-redis/blob/master/tests/test_utils.py"

    %% Infrastructure
    Infra["docker-compose.yaml"]:::external
    click Infra "https://github.com/rmax/scrapy-redis/blob/master/docker-compose.yaml"

    %% Module Clicks
    click Scheduler "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/scheduler.py"
    click DupeFilter "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/dupefilter.py"
    click QueueComp "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/queue.py"
    click PipelineComp "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/pipelines.py"
    click Connection "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/connection.py"
    click Defaults "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/defaults.py"
    click SpidersMod "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/spiders.py"
    click Utils "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/utils.py"
    click Pickle "https://github.com/rmax/scrapy-redis/blob/master/src/scrapy_redis/picklecompat.py"

    %% Styles
    classDef library fill:#ade1f9,stroke:#000,stroke-width:1px
    classDef external fill:#f9d0a6,stroke:#000,stroke-width:1px
    classDef user fill:#ace6a1,stroke:#000,stroke-width:1px

    class Scheduler,DupeFilter,QueueComp,PipelineComp,Connection,Defaults,SpidersMod,Utils,Pickle,T1,T2,T3,T4,T5 library
    class Websites,Requests,DupeSet,ItemsList,Docs,Infra external
    class Spider,Post,ESP,EPP,ESPdr,ES1,ES2,ES3 user
```
