How scrapy-plugins/scrapy-playwright works
Main Components
a. Core Components:
• Scrapy Spider
• ScrapyPlaywrightDownloadHandler
• Playwright Browser Instance
• Browser Contexts
• Page Objects
• Request/Response Flow
b. Supporting Components:
• Memory Usage Extension
• Page Event Handlers
• Page Methods Executor
• Header Processor
Component Relationships and Data Flow
a. Primary Flow:
Scrapy Spider -> ScrapyPlaywrightDownloadHandler -> Playwright Browser -> Response
b. Context Management:
Browser Instance -> Browser Contexts -> Pages
Request Processing Layer
• To "Request Queue"
• Meta options processing (playwright, playwright_context, etc.)
Download Handler Layer
• ScrapyPlaywrightDownloadHandler
• Header Processor
• Context Manager
• Page Method Executor
Playwright Integration Layer
• Playwright Browser Instance
• Browser Contexts
• Default Context
• Custom Contexts
• Pages
• Page Events
• Page Methods
Response Processing Layer
• Response transformation
• Meta data addition
• Back to Scrapy Spider
• Blue: Scrapy components
• Green: Playwright components
• Yellow: Integration components
• Gray: Supporting utilities
Symbols:
• Use diamonds for decision points (e.g., "playwright: True?")
• Use rectangles for processing components
Key Elements to Highlight:
Configuration Flow
Settings -> Download Handler -> Browser Launch -> Context Creation
Request Processing
Request -> Meta Processing -> Context Selection -> Page Creation -> Method Execution
Memory Management
ScrapyPlaywrightMemoryUsageExtension
• Process Memory Monitoring
• Scrapy Process
• Browser Process
Event Handling
Page Events -> Event Handlers -> Spider Callbacks
Clear boundaries between
• Scrapy's domain
• Playwright's domain
• Integration layer
Configuration points
• Browser type selection
• Context creation
• Page method execution
Error handling flows
• Browser disconnection
• Page errors
• Context limits
Parallel processing capabilities
• Multiple contexts
• Concurrent pages
• Resource limits
Monitoring points
• Memory usage
• Page events
• Request/response lifecycle
This design should provide a clear understanding of how scrapy-playwright integrates Playwright's browser automation capabilities with Scrapy's crawling infrastructure while maintaining proper separation of concerns and efficient resource management.