How teamhg-memex/scrapy-rotating-proxies works
Project type and purpose
• This is a Python library (a Scrapy downloader‐middleware plugin) called “scrapy‐rotating‐proxies.”
• Its purpose is to integrate with the Scrapy framework, automatically rotate HTTP proxies for each request, detect bans/failures, expire or retry proxies, and throttle per‐proxy concurrency.
File structure
• Top‐level files: setup.py, setup.cfg, README.rst, tests/, tox.ini, .travis.yml, etc. ⇒ standard Python package layout.
• Package directory “rotating_proxies/” holds four modules:
• middlewares.py (RotatingProxyMiddleware, BanDetectionMiddleware)
• expire.py (scheduling rechecks of dead proxies)
• policy.py (BanDetectionPolicy base class)
• utils.py (manages proxy pool, backoff calculations)
• tests/ folder mirrors expire.py and policy.py functionality.
• No frontend, no database—just integration points with Scrapy and an external proxy list (in memory or from file).
README for additional insights
• Hooks into Scrapy DOWNLOAD_MIDDLEWARES at two points:
• RotatingProxyMiddleware (assigns proxies)
• BanDetectionMiddleware (examines responses/exceptions)
• Configuration via settings.py keys (ROTATING_PROXY_LIST, ROTATING_PROXY_LIST_PATH, ROTATING_PROXY_BAN_POLICY, backoff parameters, etc.).
• Optional custom policy class or spider methods for ban detection.
• Explains exponential backoff for reactivating dead proxies.