# ibrahimghali/spacenets.tn-scraper-cluster architecture

> Architecture diagram of the GitHub repository ibrahimghali/spacenets.tn-scraper-cluster, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ibrahimghali/spacenets.tn-scraper-cluster
- Repository: https://github.com/ibrahimghali/spacenets.tn-scraper-cluster

## Overview

##### Project type and purpose
- This is an open-source, distributed web-scraping system built with Scrapy, Redis, and Docker.
- Purpose: parallelize crawling of Spacenets.tn product pages, store intermediate state in Redis, and export final data to disk.

##### File structure
- Top-level
- docker/ – Dockerfile and docker-compose.yml for orchestration
- scrapy.cfg & spacenets/ – the Scrapy project (spiders, items, pipelines, settings)
- utils/ – CLI scripts: push_urls_to_redis, check_redis (monitor), export_data
- Patterns
- Distributed worker pattern: multiple Scrapy “Scraper Node” containers reading from the same Redis queue
- Message queue / coordination via Redis
- Docker Compose defines three service types: redis, redis-commander, scraper nodes

##### Additional insights from README
- Emphasis on performance tuning in settings.py (concurrency, delay)

##### Main components
a. Redis Server (central queue & coordination)
b. Scraper Nodes (multiple instances running the Scrapy project)
c. Redis Commander (web UI for inspecting Redis data)
d. Host CLI utilities (push_urls_to_redis, check_redis, export_data)
e. Persistent storage on host (logs/, data/)

##### Relationships and interactions
- push_urls_to_redis → Redis Server (seed URL list)
- Each Scraper Node ↔ Redis Server (fetch next URL, mark visited, push scraped items)
- pipelines.py inside each node → writes items to data/ or pushes them back to Redis for export
- check_redis (monitor) → polls Redis Server for stats
- export_data → reads final items from Redis or disk → writes CSV/JSON to data/

##### Architectural patterns and design principles
- Distributed workers with shared queue (horizontal scaling by adding nodes)
- Single source of truth for crawl state (Redis queue + sets)
- Containerization for repeatability and isolation
- Clear separation of concerns: CLI scripts vs scraper code vs orchestration

##### Technologies, frameworks, libraries
- Scrapy (spiders, pipelines, settings)
- Redis (queue, Sets, Lists, Pub/Sub if used)
- Docker & Docker Compose (service definitions, networks, volumes)
- Python utility scripts under utils/

4. At the top, show the CLI utilities:
- push_urls_to_redis → Redis
- check_redis → Redis (two-way if polling)
- export_data → disk
6. Technologies next to each component.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI Utilities
    subgraph "CLI Utilities" 
        direction TB
        Push["push_urls_to_redis.py"]:::util
        Check["check_redis.py"]:::util
        Export["export_data.py"]:::util
    end

    %% Central Data Store
    Redis["Redis Server"]:::datastore

    %% Redis Commander UI
    RC["Redis Commander UI"]:::ui

    %% Scraper Nodes Pool
    subgraph "Autoscaling Scraper Nodes (×3)"
        direction TB
        SN["Scraper Node\n(Scrapy container)"]:::worker
        subgraph "Scrapy Project Components"
            direction TB
            Items["Items Definition\n(items.py)"]:::component
            Spider["Item Spider\n(item_spider.py)"]:::component
            Middlewares["Middleware Hooks\n(middlewares.py)"]:::component
            Pipelines["Item Pipeline\n(pipelines.py)"]:::component
            Settings["Settings & Redis Config\n(settings.py)"]:::component
        end
    end

    %% Host Volumes
    HostStorage["Host Volumes\nlogs/ and data/"]:::storage

    %% Interactions
    Push -->|"seed URLs"| Redis
    Check -->|poll stats| Redis
    RC -->|inspect data| Redis
    Redis -->|fetch URL| SN
    SN -->|push items| Redis
    SN -->|write files| HostStorage
    Export -->|reads items| Redis
    Export -->|writes files| HostStorage

    %% Internal Scraper Flow
    Spider -->|yields items| Pipelines
    Pipelines -->|stores data| Redis
    Pipelines -->|stores data| HostStorage
    Items --> Spider
    Middlewares --> Spider
    Settings --> SN

    %% Click Events
    click Orchestration "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/docker/docker-compose.yml"
    click Dockerfile "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/tree/master/docker/Dockerfile"
    click ScrapyCfg "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/scrapy.cfg"
    click SN "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/tree/master/spacenets/"
    click Items "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/spacenets/items.py"
    click Spider "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/spacenets/spiders/item_spider.py"
    click Middlewares "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/spacenets/middlewares.py"
    click Pipelines "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/spacenets/pipelines.py"
    click Settings "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/spacenets/settings.py"
    click Push "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/utils/push_urls_to_redis.py"
    click Check "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/utils/check_redis.py"
    click Export "https://github.com/ibrahimghali/spacenets.tn-scraper-cluster/blob/master/utils/export_data.py"

    %% Styling
    classDef datastore fill:#FFA500,stroke:#333,stroke-width:1px
    classDef worker fill:#8FD8D8,stroke:#333,stroke-width:1px
    classDef util fill:#90EE90,stroke:#333,stroke-width:1px
    classDef ui fill:#DDA0DD,stroke:#333,stroke-width:1px
    classDef storage fill:#D3D3D3,stroke:#333,stroke-width:1px
    classDef component fill:#ADD8E6,stroke:#333,stroke-width:1px
```
