How ibrahimghali/spacenets.tn-scraper-cluster works
Project type and purpose
• This is an open-source, distributed web-scraping system built with Scrapy, Redis, and Docker.
• Purpose: parallelize crawling of Spacenets.tn product pages, store intermediate state in Redis, and export final data to disk.
File structure
• Top-level
• docker/ – Dockerfile and docker-compose.yml for orchestration
• scrapy.cfg & spacenets/ – the Scrapy project (spiders, items, pipelines, settings)
• utils/ – CLI scripts: push_urls_to_redis, check_redis (monitor), export_data
• Patterns
• Distributed worker pattern: multiple Scrapy “Scraper Node” containers reading from the same Redis queue
• Message queue / coordination via Redis
• Docker Compose defines three service types: redis, redis-commander, scraper nodes
Additional insights from README
• Emphasis on performance tuning in settings.py (concurrency, delay)
Main components
a. Redis Server (central queue & coordination)
b. Scraper Nodes (multiple instances running the Scrapy project)
c. Redis Commander (web UI for inspecting Redis data)
d. Host CLI utilities (push_urls_to_redis, check_redis, export_data)
e. Persistent storage on host (logs/, data/)
Relationships and interactions
• push_urls_to_redis → Redis Server (seed URL list)
• Each Scraper Node ↔ Redis Server (fetch next URL, mark visited, push scraped items)
• pipelines.py inside each node → writes items to data/ or pushes them back to Redis for export
• check_redis (monitor) → polls Redis Server for stats
• export_data → reads final items from Redis or disk → writes CSV/JSON to data/
Architectural patterns and design principles
• Distributed workers with shared queue (horizontal scaling by adding nodes)
• Single source of truth for crawl state (Redis queue + sets)
• Containerization for repeatability and isolation
• Clear separation of concerns: CLI scripts vs scraper code vs orchestration
Technologies, frameworks, libraries
• Scrapy (spiders, pipelines, settings)
• Redis (queue, Sets, Lists, Pub/Sub if used)
• Docker & Docker Compose (service definitions, networks, volumes)
• Python utility scripts under utils/
4. At the top, show the CLI utilities:
• push_urls_to_redis → Redis
• check_redis → Redis (two-way if polling)
• export_data → disk
6. Technologies next to each component.