How autoscrape-labs/pydoll works
Project type and purpose
• Pydoll is a Python library for native, asynchronous browser automation without WebDrivers.
• It provides a high‑level API (Browser, Page, Element, Commands) that communicates with real browsers (Chrome, Edge) via the Chrome DevTools Protocol (CDP).
File structure
• Top‑level modules:
• pydoll/browser: Browser classes (Chrome, Edge), options, base, constants, page, managers
• pydoll/connection: CDP connection handling, managers
• pydoll/commands: Domain‑specific commands (DOM, Network, Page, Runtime, Storage, etc.)
• pydoll/events: Event definitions for reactive automations
• pydoll/mixins: Shared functionality (e.g. find_elements)
• pydoll/common, pydoll/utils, pydoll/exceptions
• Tests mirror the commands, browser and connection modules.
• No database or external persistence; external dependency is the actual browser process.
Key insights from the README
• Async API with context‑managed Browser launch.
• Zero WebDrivers—uses native CDP over a socket or WebSocket connection.
• Multi‑browser support (Chrome, Edge).
• Powerful event system and human‑like interactions.
Main components
a. User Script / Client Code
b. Pydoll API Layer
• Browser (Chrome/Edge)
• Options
• Page & Element abstractions
• Commands (organized by domain)
• Event System
c. Connection Layer
• CDPConnection Manager
• Connection Pool or WebSocket channel
d. Browser Process
• Actual Chrome/Edge instance with CDP endpoint
e. External Services
• Remote browser (if connect_to_remote used)
• Captcha solver (optional integration)
Relationships & data flow
1. Client Code invokes Browser().start() via Pydoll API.
2. Browser class spins up a browser process (local or remote) and establishes a WebSocket connection through the Connection Manager.
3. Page & Element commands (e.g., .go_to(), .find_element(), .click()) serialize into CDP commands and are sent over WebSocket.
4. Browser process executes CDP commands and streams events (Network.responseReceived, DOM.childNodeInserted) back.
5. Pydoll’s Event System dispatches incoming events to subscribers or mixes them into high‑level callbacks.
Architectural patterns & design principles
• Layered architecture:
• Presentation layer (User Script)
• API layer (Browser, Page, Commands, Events)
• Transport layer (Connection Manager with CDP WebSocket)
• External “infrastructure” (Browser process)
• Separation of concerns: commands vs events vs connection.
• Async/Await concurrency throughout.
• Mixin pattern to share cross‑cutting behavior (find_elements).
Technologies & frameworks
• Python 3.10+ async/await
• Chrome DevTools Protocol (CDP) over WebSocket
• pytest for tests
• mkdocs for documentation