How bluet/proxybroker2 works
Project Type & Purpose
• This is an asynchronous command-line tool that finds public proxies from multiple sources, verifies them concurrently, and can also serve as a local proxy server with automatic proxy rotation.
• Readme details point to CLI commands (“find”, “grab”, “serve”, “update-geo”), asynchronous operations (using aiohttp and aiodns), and components for proxy checking, negotiation, and resolving.
File Structure for Key Components
• The top-level directories/files:
• “proxybroker” directory contains the core modules (API, CLI, Checker, Negotiators, Providers, Proxy, Resolver, and Server).
• “tests” directory holds unit and integration tests that imply modularity and internal interactions.
• “examples” directory provides usage examples.
• “docs” folder contains documentation and API details.
• Configuration files (Dockerfile, setup.py, pyproject.toml, and CI workflows) indicate how the project is built, tested, and deployed.
• Notice that there is a clear division between the core business logic in “proxybroker”, external data (the GeoIP database in proxybroker/data), and the CLI/API entry points.
Main Components
a. User Interfaces:
• CLI: Command line entry point (proxybroker/cli.py and __main__.py) which interprets commands and options.
• API: HTTP API endpoints for operations like checking proxy info and removal.
b. Core Asynchronous Engine:
• Broker: Manages the asynchronous discovery and checking of proxies (likely coordinated in proxybroker/api.py and proxybroker/checker.py).
• Proxy Finder & Collector: Modules that locate proxies from multiple external sources (proxybroker/providers.py).
• Proxy Checker: Validates proxies asynchronously (proxybroker/checker.py, proxybroker/judge.py).
• Proxy Negotiators & Resolver: Handles the negotiation logic for proxy connections (proxybroker/negotiators.py and proxybroker/resolver.py).
c. Local Proxy Server:
• Server Module: Implements the proxy server functionality for routing requests through found proxies (proxybroker/server.py).
d. External Data & Dependencies:
• GeoIP Database: Provided via “proxybroker/data/GeoLite2-Country.mmdb” and referenced when filtering proxies by location.
• Third-party libraries such as aiohttp, aiodns, and maxminddb for asynchronous networking and data lookups.
e. Asynchronous Processing & Task Management:
• Asynchronous Queue handling and event-loop integration (seen in examples and usage instructions).
Relationships & Interactions
• Users trigger commands via the CLI or API endpoints.
• The CLI dispatches commands to the Broker, which orchestrates tasks such as “find”, “grab”, or “serve”.
• The Broker interacts with the Proxy Finder (Providers) to gather proxies, then passes them to the Checker for concurrent validation.
• The Checker, Judge, and Negotiators work together in an asynchronous workflow (using queues) to validate proxy speed, anonymity levels, and protocol support.
• Verified proxies are either returned to the user, stored in a queue for continuous checking, or used by the Server component that acts as a local proxy with automatic rotation.
• External GeoIP data is utilized to filter proxies by country.
• In tests and examples, developers see the async interaction pattern using asyncio queues.
Architectural Patterns & Design Principles
• Asynchronous and concurrent design: Multiple asynchronous workers checking proxies concurrently using asyncio.
• Separation of concerns: Different modules handle distinct responsibilities (CLI, discovery, checking, negotiation, serving).
• Modular design: Clearly divided into components that can be individually tested and maintained.
• Extensibility: Use of providers and negotiators makes it easy to add new sources or protocols.
Putting It All Together
• Additionally, incorporate external elements such as “GeoIP Database (MaxMind)” connected to:
• The Proxy Checker and Providers which use the data for filtering.
• Auxiliary modules or configuration files (like Docker for container deployment).
• Components used for configuration, testing, and documentation separately to show the broader ecosystem.