How oxylabs/oxylabs-mcp works
Based on both the file tree and the README, we’re dealing with a Python-based MCP server that acts as a bridge between AI assistants (like Claude) and Oxylabs’ web scraping services. This server receives scraping requests, processes them (rendering JavaScript when needed, cleaning HTML, parsing data, etc.) and then returns AI-ready data. It leverages two key internal “tools” – one for standard scraping (oxylabs_scraper) and one for the “web unblocker” functionality targeted at hard-to-access sites.
Main Components
• MCP Server: The heart of the project (as seen in src/oxylabs_mcp/server.py) which handles incoming requests. This should be represented as a core service.
• Utility Functions: Include a component representing the utility or helper functions (from src/oxylabs_mcp/utils.py) that are used for processing the scraped data.
• Scraping Modules: Distinguish two sub-components within the server:
• oxylabs_scraper – handling typical website scraping through the Oxylabs Web Scraper API.
• oxylabs_web_unblocker – dedicated to bypassing blocks and geo-restrictions using the Web Unblocker.
• Dev & Deployment Elements: Indicate Docker (through Dockerfile) for containerization, and optionally show “MCP Inspector” (accessed at http://localhost:5173) for debugging.
• Configuration & Env Setup: Note the use of environment variables (like OXYLABS_USERNAME, OXYLABS_PASSWORD) and the uv package manager which helps with dependency and environment management.
Architectural Patterns / Design Principles
• Modularity: The separation between server endpoints (server.py) and helper functions (utils.py) and, at a further granularity, the separation of concerns between scraping and unblocking modules.
• API Gateway/Facade Pattern: The MCP server acts as a standardized gateway for various kinds of scraping operations.
• Containerization: With the Dockerfile included, the project is designed for containerized deployment.
Relevant Technologies and Tools
• Python 3.12+ is a key runtime.
• uv package manager for environment and dependency management.
• Docker (via Dockerfile) for deployment.
• External Web Service endpoints from Oxylabs (Web Scraper API and Web Unblocker).
• While it is important to capture all of the key elements and interactions, avoid overcomplicating. Make sure that each module’s inputs and outputs are clearly annotated (for example, showing how parameters like url, parse, and render are used to route the request).