# scrapy-plugins/scrapy-playwright architecture

> Architecture diagram of the GitHub repository scrapy-plugins/scrapy-playwright, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/scrapy-plugins/scrapy-playwright
- Repository: https://github.com/scrapy-plugins/scrapy-playwright

## Overview

##### Main Components

a. Core Components:
- Scrapy Spider
- ScrapyPlaywrightDownloadHandler
- Playwright Browser Instance
- Browser Contexts
- Page Objects
- Request/Response Flow

b. Supporting Components:
- Memory Usage Extension
- Page Event Handlers
- Page Methods Executor
- Header Processor

##### Component Relationships and Data Flow

a. Primary Flow:
**Scrapy Spider -> ScrapyPlaywrightDownloadHandler -> Playwright Browser -> Response**

b. Context Management:
**Browser Instance -> Browser Contexts -> Pages**

##### Request Processing Layer
- To "Request Queue"
- Meta options processing (`playwright`, `playwright_context`, etc.)

##### Download Handler Layer
- ScrapyPlaywrightDownloadHandler
- Header Processor
- Context Manager
- Page Method Executor

##### Playwright Integration Layer
- Playwright Browser Instance
- Browser Contexts
- Default Context
- Custom Contexts
- Pages
- Page Events
- Page Methods

##### Response Processing Layer
- Response transformation
- Meta data addition
- Back to Scrapy Spider

- Blue: Scrapy components
- Green: Playwright components
- Yellow: Integration components
- Gray: Supporting utilities

Symbols:
- Use diamonds for decision points (e.g., "playwright: True?")
- Use rectangles for processing components

Key Elements to Highlight:

##### Configuration Flow
**Settings -> Download Handler -> Browser Launch -> Context Creation**

##### Request Processing
**Request -> Meta Processing -> Context Selection -> Page Creation -> Method Execution**

##### Memory Management
**ScrapyPlaywrightMemoryUsageExtension**
- Process Memory Monitoring
- Scrapy Process
- Browser Process

##### Event Handling
**Page Events -> Event Handlers -> Spider Callbacks**

##### Clear boundaries between
- Scrapy's domain
- Playwright's domain
- Integration layer

##### Configuration points
- Browser type selection
- Context creation
- Page method execution

##### Error handling flows
- Browser disconnection
- Page errors
- Context limits

##### Parallel processing capabilities
- Multiple contexts
- Concurrent pages
- Resource limits

##### Monitoring points
- Memory usage
- Page events
- Request/response lifecycle

This design should provide a clear understanding of how scrapy-playwright integrates Playwright's browser automation capabilities with Scrapy's crawling infrastructure while maintaining proper separation of concerns and efficient resource management.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Request["Request Processing Layer"]
        Spider["Scrapy Spider"]
        ReqQueue["Request Queue"]
        MetaOpt["Meta Options Processing"]
    end

    subgraph Handler["Download Handler Layer"]
        DLHandler["ScrapyPlaywrightDownloadHandler"]:::handler
        HeaderProc["Header Processor"]:::headers
        ContextMgr["Context Manager"]:::context
        PageExec["Page Method Executor"]:::executor
    end

    subgraph Playwright["Playwright Integration Layer"]
        Browser["Playwright Browser Instance"]:::browser
        subgraph Contexts["Browser Contexts"]
            DefContext["Default Context"]
            CustContext["Custom Contexts"]
        end
        subgraph Pages["Page Management"]
            PageObj["Page Objects"]:::page
            PageEvents["Page Events"]:::events
            PageMethods["Page Methods"]:::methods
        end
    end

    subgraph Response["Response Processing Layer"]
        RespTrans["Response Transformation"]
        MetaAdd["Metadata Addition"]
        MemUsage["Memory Usage Monitor"]:::memory
    end

    %% Connections
    Spider --> ReqQueue
    ReqQueue --> MetaOpt
    MetaOpt --> DLHandler
    DLHandler --> HeaderProc
    DLHandler --> ContextMgr
    ContextMgr --> Browser
    Browser --> DefContext
    Browser --> CustContext
    DefContext --> PageObj
    CustContext --> PageObj
    PageObj --> PageEvents
    PageObj --> PageMethods
    PageMethods --> PageExec
    PageExec --> RespTrans
    RespTrans --> MetaAdd
    MetaAdd --> Spider
    MemUsage -.-> Browser
    MemUsage -.-> Spider

    %% Styles
    classDef handler fill:#FFE5CC,stroke:#FF9933
    classDef headers fill:#E6F3FF,stroke:#3399FF
    classDef context fill:#E6FFE6,stroke:#33CC33
    classDef executor fill:#FFE6E6,stroke:#FF3333
    classDef browser fill:#F0E6FF,stroke:#9933FF
    classDef page fill:#FFE6F0,stroke:#FF33CC
    classDef events fill:#E6FFE6,stroke:#33CC33
    classDef methods fill:#FFE6E6,stroke:#FF3333
    classDef memory fill:#E6E6E6,stroke:#666666

    %% Click Events
    click DLHandler "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/scrapy_playwright/handler.py"
    click HeaderProc "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/scrapy_playwright/headers.py"
    click ContextMgr "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/tests/tests_asyncio/test_browser_contexts.py"
    click PageExec "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/tests/tests_asyncio/test_page_methods.py"
    click PageObj "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/scrapy_playwright/page.py"
    click PageEvents "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/examples/events.py"
    click Browser "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/tests/tests_asyncio/test_browser.py"
    click MemUsage "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/scrapy_playwright/memusage.py"
    click CustContext "https://github.com/scrapy-plugins/scrapy-playwright/blob/main/examples/contexts.py"
```
