# autoscrape-labs/pydoll architecture

> Architecture diagram of the GitHub repository autoscrape-labs/pydoll, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/autoscrape-labs/pydoll
- Repository: https://github.com/autoscrape-labs/pydoll

## Overview

##### Project type and purpose
- Pydoll is a Python library for native, asynchronous browser automation without WebDrivers.
- It provides a high‑level API (Browser, Page, Element, Commands) that communicates with real browsers (Chrome, Edge) via the Chrome DevTools Protocol (CDP).

##### File structure
- Top‑level modules:
- pydoll/browser: Browser classes (Chrome, Edge), options, base, constants, page, managers
- pydoll/connection: CDP connection handling, managers
- pydoll/commands: Domain‑specific commands (DOM, Network, Page, Runtime, Storage, etc.)
- pydoll/events: Event definitions for reactive automations
- pydoll/mixins: Shared functionality (e.g. find_elements)
- pydoll/common, pydoll/utils, pydoll/exceptions
- Tests mirror the commands, browser and connection modules.
- No database or external persistence; external dependency is the actual browser process.

##### Key insights from the README
- Async API with context‑managed Browser launch.
- Zero WebDrivers—uses native CDP over a socket or WebSocket connection.
- Multi‑browser support (Chrome, Edge).
- Powerful event system and human‑like interactions.

##### Main components
a. User Script / Client Code
b. Pydoll API Layer
- Browser (Chrome/Edge)
- Options
- Page & Element abstractions
- Commands (organized by domain)
- Event System
c. Connection Layer
- CDPConnection Manager
- Connection Pool or WebSocket channel
d. Browser Process
- Actual Chrome/Edge instance with CDP endpoint
e. External Services
- Remote browser (if connect_to_remote used)
- Captcha solver (optional integration)

##### Relationships & data flow
1. Client Code invokes Browser().start() via Pydoll API.
2. Browser class spins up a browser process (local or remote) and establishes a WebSocket connection through the Connection Manager.
3. Page & Element commands (e.g., .go_to(), .find_element(), .click()) serialize into CDP commands and are sent over WebSocket.
4. Browser process executes CDP commands and streams events (Network.responseReceived, DOM.childNodeInserted) back.
5. Pydoll’s Event System dispatches incoming events to subscribers or mixes them into high‑level callbacks.

##### Architectural patterns & design principles
- Layered architecture:
- Presentation layer (User Script)
- API layer (Browser, Page, Commands, Events)
- Transport layer (Connection Manager with CDP WebSocket)
- External “infrastructure” (Browser process)
- Separation of concerns: commands vs events vs connection.
- Async/Await concurrency throughout.
- Mixin pattern to share cross‑cutting behavior (find_elements).

##### Technologies & frameworks
- Python 3.10+ async/await
- Chrome DevTools Protocol (CDP) over WebSocket
- pytest for tests
- mkdocs for documentation

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client Layer"
        UserScript["User Script / Client Code"]:::client
    end

    %% API Layer
    subgraph "API Layer"
        direction TB
        %% Browser Facade
        subgraph "Browser Facade"
            direction TB
            BaseBrowser["Base Browser"]:::api
            Chrome["Chrome Browser"]:::api
            Edge["Edge Browser"]:::api
            Managers["Browser Managers"]:::api
            Options["Browser Options"]:::api
            Constants["Browser Constants"]:::api
        end
        %% Page & Element
        subgraph "Page & Element Abstractions"
            direction TB
            Page["Page Abstraction"]:::api
            WebElement["WebElement Wrapper"]:::api
            FindMixin["FindElements Mixin"]:::api
        end
        %% Commands Domains
        subgraph "Commands"
            direction TB
            BrowserCmd["Browser Commands"]:::api
            DOMCmd["DOM Commands"]:::api
            NetworkCmd["Network Commands"]:::api
            PageCmd["Page Commands"]:::api
            RuntimeCmd["Runtime Commands"]:::api
            StorageCmd["Storage Commands"]:::api
            FetchCmd["Fetch Commands"]:::api
            InputCmd["Input Commands"]:::api
            TargetCmd["Target Commands"]:::api
        end
        %% Event System
        subgraph "Event System"
            direction TB
            EventsBrowser["Browser Events"]:::api
            EventsDOM["DOM Events"]:::api
            EventsNetwork["Network Events"]:::api
            EventsPage["Page Events"]:::api
            EventsFetch["Fetch Events"]:::api
        end
    end

    %% Transport Layer
    subgraph "Transport Layer"
        direction TB
        ConnManager["CDP Connection Manager"]:::transport
        Connection["CDP Connection"]:::transport
    end

    %% External Infrastructure
    subgraph "External Infrastructure"
        direction TB
        BrowserProcess["Chrome/Edge Process\n(CDP Endpoint)"]:::external
        RemoteBrowser["Remote Browser (optional)"]:::external
    end

    %% Utilities & Exceptions
    subgraph "Common Utilities & Exceptions"
        direction TB
        Keyboard["Keyboard Utils"]:::util
        Keys["Keys Constants"]:::util
        Utils["General Utils"]:::util
        Exceptions["Exception Types"]:::util
    end

    %% Data Flows
    UserScript -->|".start()"| BaseBrowser
    BaseBrowser -->|"launch"| Managers
    Managers -->|"process start"| BrowserProcess
    BaseBrowser -->|"connect()"| ConnManager
    ConnManager -->|"open WS"| Connection
    Connection <-->|"<--CDP.send('Page.navigate') / send commands"| BrowserProcess
    BrowserProcess -->|Event: Network.responseReceived| Connection
    Connection -->|dispatch events| EventsBrowser
    EventsBrowser -->|callbacks| UserScript

    Page -->|".go_to(), .click()"| BrowserCmd
    Page --> DOMCmd
    Page --> NetworkCmd
    Page --> PageCmd
    Page --> RuntimeCmd
    Page --> StorageCmd
    Page --> FetchCmd
    Page --> InputCmd
    Page --> TargetCmd
    BrowserCmd --> Connection
    DOMCmd --> Connection
    NetworkCmd --> Connection
    PageCmd --> Connection
    RuntimeCmd --> Connection
    StorageCmd --> Connection
    FetchCmd --> Connection
    InputCmd --> Connection
    TargetCmd --> Connection

    WebElement --> FindMixin

    %% Utilities usage
    Page --> Utils
    WebElement --> Keyboard
    WebElement --> Keys

    %% Click Events
    click UserScript "https://github.com/autoscrape-labs/pydoll/blob/main/examples/cloudflare_bypass.py"
    click BaseBrowser "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/base.py"
    click Chrome "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/chrome.py"
    click Edge "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/edge.py"
    click Managers "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/managers.py"
    click Options "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/options.py"
    click Constants "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/constants.py"
    click Page "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/browser/page.py"
    click WebElement "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/element.py"
    click FindMixin "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/mixins/find_elements.py"
    click BrowserCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/browser.py"
    click DOMCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/dom.py"
    click NetworkCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/network.py"
    click PageCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/page.py"
    click RuntimeCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/runtime.py"
    click StorageCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/storage.py"
    click FetchCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/fetch.py"
    click InputCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/input.py"
    click TargetCmd "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/commands/target.py"
    click EventsBrowser "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/events/browser.py"
    click EventsDOM "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/events/dom.py"
    click EventsNetwork "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/events/network.py"
    click EventsPage "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/events/page.py"
    click EventsFetch "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/events/fetch.py"
    click ConnManager "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/connection/managers.py"
    click Connection "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/connection/connection.py"
    click Keyboard "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/common/keyboard.py"
    click Keys "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/common/keys.py"
    click Utils "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/utils.py"
    click Exceptions "https://github.com/autoscrape-labs/pydoll/blob/main/pydoll/exceptions.py"

    %% Styles
    classDef client fill:#eaf5d8,stroke:#2d572c,stroke-width:1px;
    classDef api fill:#d0f0fd,stroke:#016c9e,stroke-width:1px;
    classDef transport fill:#e0e7ff,stroke:#3b3998,stroke-width:1px;
    classDef external fill:#f2f2f2,stroke:#7a7a7a,stroke-width:1px,stroke-dasharray: 5 5;
    classDef util fill:#fff4e5,stroke:#c56e00,stroke-width:1px; 
    class UserScript client
    class BaseBrowser,Chrome,Edge,Managers,Options,Constants,Page,WebElement,FindMixin,BrowserCmd,DOMCmd,NetworkCmd,PageCmd,RuntimeCmd,StorageCmd,FetchCmd,InputCmd,TargetCmd,EventsBrowser,EventsDOM,EventsNetwork,EventsPage,EventsFetch api
    class ConnManager,Connection transport
    class BrowserProcess,RemoteBrowser external
    class Keyboard,Keys,Utils,Exceptions util
```
