# teamhg-memex/scrapy-rotating-proxies architecture

> Architecture diagram of the GitHub repository teamhg-memex/scrapy-rotating-proxies, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/teamhg-memex/scrapy-rotating-proxies
- Repository: https://github.com/teamhg-memex/scrapy-rotating-proxies

## Overview

##### Project type and purpose
- This is a Python library (a Scrapy downloader‐middleware plugin) called “scrapy‐rotating‐proxies.”
- Its purpose is to integrate with the Scrapy framework, automatically rotate HTTP proxies for each request, detect bans/failures, expire or retry proxies, and throttle per‐proxy concurrency.

##### File structure
- Top‐level files: setup.py, setup.cfg, README.rst, tests/, tox.ini, .travis.yml, etc. ⇒ standard Python package layout.
- Package directory “rotating_proxies/” holds four modules:
- middlewares.py         (RotatingProxyMiddleware, BanDetectionMiddleware)
- expire.py              (scheduling rechecks of dead proxies)
- policy.py              (BanDetectionPolicy base class)
- utils.py               (manages proxy pool, backoff calculations)
- tests/ folder mirrors expire.py and policy.py functionality.
- No frontend, no database—just integration points with Scrapy and an external proxy list (in memory or from file).

##### README for additional insights
- Hooks into Scrapy DOWNLOAD_MIDDLEWARES at two points:
- RotatingProxyMiddleware (assigns proxies)
- BanDetectionMiddleware (examines responses/exceptions)
- Configuration via settings.py keys (ROTATING_PROXY_LIST, ROTATING_PROXY_LIST_PATH, ROTATING_PROXY_BAN_POLICY, backoff parameters, etc.).
- Optional custom policy class or spider methods for ban detection.
- Explains exponential backoff for reactivating dead proxies.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Scrapy Core Components
    subgraph "Scrapy Core" 
        direction TB
        Spider["Scrapy Spider"]:::core
        EngineScheduler["Scrapy Engine / Scheduler"]:::core
    end

    %% rotating_proxies Modules
    subgraph "rotating_proxies Package" 
        direction TB
        RPMW["RotatingProxyMiddleware"]:::rp
        BDMW["BanDetectionMiddleware"]:::rp
        PPM["Proxy Pool Manager\n(utils)"]:::rp
        BDP["BanDetectionPolicy"]:::rp
        PES["Proxy Expiry Scheduler"]:::rp
    end

    %% External Resources & Config
    subgraph "External & Config" 
        direction TB
        Settings["Settings / Configuration"]:::ext
        ExtList["External Proxies\n(list / file)"]:::ext
    end

    %% Tests
    subgraph "Tests" 
        direction TB
        TestExpire["tests/test_expire.py"]:::ext
        TestPolicy["tests/test_policy.py"]:::ext
    end

    %% Request Flow
    Spider -->|"issue Request"| EngineScheduler
    EngineScheduler -->|"process Request"| RPMW
    RPMW -->|"select_proxy()"| PPM
    PPM -->|"proxy URL"| RPMW
    RPMW -->|"injects meta.proxy"| Downloader[/"Downloader"/]:::core
    Downloader -->|"Response/Exception"| BDMW

    %% Ban Detection Flow
    BDMW -->|"apply_policy()"| BDP
    BDP -->|"ban decision"| BDMW
    BDMW -->|"mark_dead() / retry"| PPM
    BDMW -->|"pass or retry"| EngineScheduler

    %% Expiry Scheduler Flow
    PES -->|"periodic recheck"| PPM

    %% Initialization Flows
    Settings --> RPMW
    Settings --> BDMW
    Settings --> BDP
    ExtList --> PPM

    %% Tests Hooks
    TestExpire --> PES
    TestPolicy --> BDP

    %% Click Events
    click RPMW "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/rotating_proxies/middlewares.py"
    click BDMW "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/rotating_proxies/middlewares.py"
    click PPM "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/rotating_proxies/utils.py"
    click BDP "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/rotating_proxies/policy.py"
    click PES "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/rotating_proxies/expire.py"
    click Settings "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/README.rst"
    click TestExpire "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/tests/test_expire.py"
    click TestPolicy "https://github.com/teamhg-memex/scrapy-rotating-proxies/blob/master/tests/test_policy.py"

    %% Styles
    classDef core fill:#a0faa0,stroke:#333,stroke-width:1px
    classDef rp fill:#a0cfff,stroke:#333,stroke-width:1px
    classDef ext fill:#ffd8a8,stroke:#333,stroke-width:1px
    class Downloader fill:#a0faa0,stroke:#333,stroke-width:1px
    class Spider,EngineScheduler,Downloader core
    class RPMW,BDMW,PPM,BDP,PES rp
    class Settings,ExtList,TestExpire,TestPolicy ext
```
