# apify/crawlee-python architecture

> Architecture diagram of the GitHub repository apify/crawlee-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apify/crawlee-python
- Repository: https://github.com/apify/crawlee-python

## Overview

##### Main Components
- Core Engine & API: Represent the central “crawlee” package. This includes the core API, configuration, logging, error handling, and routing (e.g., files such as _cli.py, _consts.py, router.py, service_locator.py). You may show this as the “Core Engine” module.
- Crawlers Subsystem: Illustrate how different crawler types are implemented. This includes:
– Abstract HTTP Crawlers (in _abstract_http)
– Basic Crawlers (in _basic)
– Specialized Crawlers for BeautifulSoup (in _beautifulsoup), Playwright (in _playwright, _adaptive_playwright) and Parsel (in _parsel)
Each crawler type can be a component that “inherits” common behavior from the core engine while interacting with their respective parsing libraries (e.g., BeautifulSoup, Parsel, Playwright API).
- Autoscaling and Resource Management: Include the autoscaling module (src/crawlee/_autoscaling) that manages parallel crawling and pool management (e.g., autoscaled_pool, snapshotter, system_status). This shows how the system dynamically adjusts concurrency based on system resources.
- Browser Management: Show the “Browsers” module, which handles browser pools and controllers (e.g., _playwright_browser, _playwright_browser_controller). This component integrates with Playwright to provide headless browser automation.
- Request, Routing, and Session Management: Include subsystems like:
– Request loader and management (in request_loaders)
– Sessions (in sessions) that manage cookies and user sessions
– Router which dispatches requests to appropriate handlers.
- Data Storage: Represent the storage and persistence layers. This includes:
– Storage clients (for datasets, key-value stores, and request queues in storage_clients)
– Storages (in storages) which provide abstractions for saving scraped data.
- Event Handling and Fingerprinting: Include the events module and the fingerprint_suite. They handle event management (local events) and fingerprint generation for browser requests.
- External Integrations: Show integration points for external libraries such as:
– HTTP clients (e.g., httpx, curl_impersonate)
– Third-party libraries like BeautifulSoup, Playwright.
– Optional integration with the Apify platform for cloud deployment (as noted in the README).

##### Relations and Interactions
- That the crawler components interact with the autoscaling pool to manage concurrency and with browser controllers when using headless browsers.
- How request loaders feed URLs into the crawling process, with requests being queued and persisted via the storage clients.
- Error handling, logging, and events flows feeding back into the core engine for resilience and debugging.
- That after parsing or processing a page, the system pushes data to persistent storage (dataset or key-value store).

##### Architectural Patterns and Design Principles
- Modularity & Separation of Concerns: Different aspects (crawling, autoscaling, session management, storage, event handling) are clearly split into separate modules.
- Pluggable Components: The design emphasizes extensibility (for example, through different crawler templates or using either HTTP or browser-based crawling) which can be represented as interchangeable modules.
- Async Design: The library is based on Python’s asyncio for asynchronous operations.
- Template-based Project Creation: Demonstrate the inclusion of a project_template module that helps bootstrap new projects via cookiecutter.

##### Final Recommendations
- Detail each architectural module with annotations to explain its functionality.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Core Engine Layer
    subgraph "Core Engine"
        cli["CLI/Entry"]:::core
        config["Core Config & Constants"]:::core
        router["Router & Service Locator"]:::core
    end
    cli -->|"routes"| router
    config -->|"configures"| router

    %% Project Template (Bootstrap)
    projectTemplate["Project Template"]:::template
    cli -->|"triggers"| projectTemplate

    %% Request and Session Management
    requestLoader["Request Loader Manager"]:::request
    sessionMgr["Session Manager"]:::request
    router -->|"loads"| requestLoader
    router -->|"manages"| sessionMgr

    %% External Integrations
    httpClients["HTTP Clients"]:::external
    config -->|"integrates"| httpClients

    %% Crawlers Subsystem
    subgraph "Crawlers Subsystem"
        ahttp["Abstract HTTP Crawler"]:::crawler
        basic["Basic Crawler"]:::crawler
        bsoup["BeautifulSoup Crawler"]:::crawler
        pwt["Playwright Crawler"]:::crawler
        apwt["Adaptive Playwright Crawler"]:::crawler
        parsel["Parsel Crawler"]:::crawler
    end
    router -->|"selects"| ahttp
    router -->|"selects"| basic
    router -->|"selects"| bsoup
    router -->|"selects"| pwt
    router -->|"selects"| apwt
    router -->|"selects"| parsel

    %% Autoscaling and Browser Management
    autoscaleManager["Autoscaling Manager"]:::autoscale
    browserManager["Browser Manager"]:::browser

    ahttp -->|"scales"| autoscaleManager
    basic -->|"scales"| autoscaleManager
    bsoup -->|"scales"| autoscaleManager
    pwt -->|"scales"| autoscaleManager
    apwt -->|"scales"| autoscaleManager
    parsel -->|"scales"| autoscaleManager

    pwt -->|"controls"| browserManager
    apwt -->|"controls"| browserManager

    %% Data Storage
    storageClients["Storage Clients"]:::storage
    storageLayers["Storage Layers"]:::storage
    ahttp -->|"stores"| storageClients
    basic -->|"stores"| storageClients
    bsoup -->|"stores"| storageClients
    pwt -->|"stores"| storageClients
    apwt -->|"stores"| storageClients
    parsel -->|"stores"| storageClients
    storageClients -->|"persists"| storageLayers

    %% Event Handling and Fingerprinting
    eventMgr["Event Manager"]:::event
    fingerprintSuite["Fingerprint Suite"]:::event
    router -->|"logs"| eventMgr
    ahttp -->|"logs"| eventMgr
    basic -->|"logs"| eventMgr
    bsoup -->|"logs"| eventMgr
    pwt -->|"logs"| eventMgr
    apwt -->|"logs"| eventMgr
    parsel -->|"logs"| eventMgr
    eventMgr -->|"generates"| fingerprintSuite

    %% Click Events
    click cli "https://github.com/apify/crawlee-python/blob/master/src/crawlee/_cli.py"
    click config "https://github.com/apify/crawlee-python/blob/master/src/crawlee/_consts.py"
    click router "https://github.com/apify/crawlee-python/blob/master/src/crawlee/router.py"
    click ahttp "https://github.com/apify/crawlee-python/tree/master/src/crawlee/crawlers/_abstract_http"
    click basic "https://github.com/apify/crawlee-python/tree/master/src/crawlee/crawlers/_basic"
    click bsoup "https://github.com/apify/crawlee-python/tree/master/src/crawlee/crawlers/_beautifulsoup"
    click pwt "https://github.com/apify/crawlee-python/tree/master/src/crawlee/crawlers/_playwright"
    click apwt "https://github.com/apify/crawlee-python/tree/master/src/crawlee/crawlers/_adaptive_playwright"
    click parsel "https://github.com/apify/crawlee-python/tree/master/src/crawlee/crawlers/_parsel"
    click autoscaleManager "https://github.com/apify/crawlee-python/tree/master/src/crawlee/_autoscaling"
    click browserManager "https://github.com/apify/crawlee-python/tree/master/src/crawlee/browsers"
    click requestLoader "https://github.com/apify/crawlee-python/tree/master/src/crawlee/request_loaders"
    click sessionMgr "https://github.com/apify/crawlee-python/tree/master/src/crawlee/sessions"
    click storageClients "https://github.com/apify/crawlee-python/tree/master/src/crawlee/storage_clients"
    click storageLayers "https://github.com/apify/crawlee-python/tree/master/src/crawlee/storages"
    click eventMgr "https://github.com/apify/crawlee-python/tree/master/src/crawlee/events"
    click fingerprintSuite "https://github.com/apify/crawlee-python/tree/master/src/crawlee/fingerprint_suite"
    click httpClients "https://github.com/apify/crawlee-python/tree/master/src/crawlee/http_clients"
    click projectTemplate "https://github.com/apify/crawlee-python/tree/master/src/crawlee/project_template"

    %% Styles
    classDef core fill:#f9f,stroke:#333,stroke-width:2px;
    classDef crawler fill:#ccf,stroke:#333,stroke-width:2px;
    classDef autoscale fill:#cfc,stroke:#333,stroke-width:2px;
    classDef browser fill:#fcf,stroke:#333,stroke-width:2px;
    classDef request fill:#cff,stroke:#333,stroke-width:2px;
    classDef storage fill:#ffc,stroke:#333,stroke-width:2px;
    classDef event fill:#fdd,stroke:#333,stroke-width:2px;
    classDef external fill:#def,stroke:#333,stroke-width:2px;
    classDef template fill:#fed,stroke:#333,stroke-width:2px;
```
