How rmax/scrapy-redis works
Project type and purpose
• This is not an application per se but a Python library/plugin (“scrapy-redis”) that extends the Scrapy web-crawling framework with Redis-backed components.
• Its purpose is to enable distributed crawling and distributed item processing by decoupling Scrapy spiders from in-memory queues and offering Redis as a central broker.
File structure
• src/scrapy_redis/: the core library modules (scheduler, queue, dupefilter, pipelines, spiders, utils, connection).
• example-project/: a sample Scrapy project that shows how to wire in the scrapy-redis components (settings.py, spiders/, pipelines.py).
• docs/: user documentation.
• tests/: unit tests covering each redis-based component.
• Dockerfile & docker-compose.yaml: environment for running Redis + example crawlers.
• CI configs (.github/workflows, .travis.yml, tox.ini): build/test automation.
README for additional insights
• Main features: “Scheduler + Duplication Filter, Item Pipeline, Base Spiders.”
• Distributed crawling: multiple Scrapy instances share a single Redis request queue.
• Distributed post-processing: scraped items are enqueued in Redis for any number of consumers.
• Technologies: Python ≥3.7, Redis ≥5.0, Scrapy ≥2.0, redis-py ≥4.0.
Elements
• Spider → Redis (enqueue new requests, items)
• Redis → Spider (dequeue next URL)
• Redis → Post-Processor (dequeue items)
• Blue for Scrapy-Redis library modules (Scheduler, DupeFilter, Pipeline)
• Orange for external systems (Redis server, target websites)
• Green for user-provided code (example spiders, post-processors)
• Notes or annotations on key data structures (Redis list “scrapy:requests”, set “scrapy:dupefilter”, list “scrapy:items”).
Detailed callouts & layering
• Layer 1 (top): External target websites
• Layer 2: Scrapy Spider Instances (show internal flow: Spider → Scheduler → Queue → DupeFilter → fetch)
• Layer 4: Post-processing consumers
• Optional lateral layer: Management (Docker-Compose orchestration, scaling up/down spider replicas)