How apify/crawlee-python works
Main Components
• Core Engine & API: Represent the central “crawlee” package. This includes the core API, configuration, logging, error handling, and routing (e.g., files such as _cli.py, _consts.py, router.py, service_locator.py). You may show this as the “Core Engine” module.
• Crawlers Subsystem: Illustrate how different crawler types are implemented. This includes:
– Abstract HTTP Crawlers (in _abstract_http)
– Basic Crawlers (in _basic)
– Specialized Crawlers for BeautifulSoup (in _beautifulsoup), Playwright (in _playwright, _adaptive_playwright) and Parsel (in _parsel)
Each crawler type can be a component that “inherits” common behavior from the core engine while interacting with their respective parsing libraries (e.g., BeautifulSoup, Parsel, Playwright API).
• Autoscaling and Resource Management: Include the autoscaling module (src/crawlee/_autoscaling) that manages parallel crawling and pool management (e.g., autoscaled_pool, snapshotter, system_status). This shows how the system dynamically adjusts concurrency based on system resources.
• Browser Management: Show the “Browsers” module, which handles browser pools and controllers (e.g., _playwright_browser, _playwright_browser_controller). This component integrates with Playwright to provide headless browser automation.
• Request, Routing, and Session Management: Include subsystems like:
– Request loader and management (in request_loaders)
– Sessions (in sessions) that manage cookies and user sessions
– Router which dispatches requests to appropriate handlers.
• Data Storage: Represent the storage and persistence layers. This includes:
– Storage clients (for datasets, key-value stores, and request queues in storage_clients)
– Storages (in storages) which provide abstractions for saving scraped data.
• Event Handling and Fingerprinting: Include the events module and the fingerprint_suite. They handle event management (local events) and fingerprint generation for browser requests.
• External Integrations: Show integration points for external libraries such as:
– HTTP clients (e.g., httpx, curl_impersonate)
– Third-party libraries like BeautifulSoup, Playwright.
– Optional integration with the Apify platform for cloud deployment (as noted in the README).
Relations and Interactions
• That the crawler components interact with the autoscaling pool to manage concurrency and with browser controllers when using headless browsers.
• How request loaders feed URLs into the crawling process, with requests being queued and persisted via the storage clients.
• Error handling, logging, and events flows feeding back into the core engine for resilience and debugging.
• That after parsing or processing a page, the system pushes data to persistent storage (dataset or key-value store).
Architectural Patterns and Design Principles
• Modularity & Separation of Concerns: Different aspects (crawling, autoscaling, session management, storage, event handling) are clearly split into separate modules.
• Pluggable Components: The design emphasizes extensibility (for example, through different crawler templates or using either HTTP or browser-based crawling) which can be represented as interchangeable modules.
• Async Design: The library is based on Python’s asyncio for asynchronous operations.
• Template-based Project Creation: Demonstrate the inclusion of a project_template module that helps bootstrap new projects via cookiecutter.
Final Recommendations
• Detail each architectural module with annotations to explain its functionality.