How psf/requests-html works
Main Components
a) Core Components:
• Requests-HTML Client Layer
• HTML Parsing Engine
• JavaScript Rendering Engine (Chromium via pyppeteer)
• Selector Engines (CSS/jQuery via PyQuery, XPath)
b) External Dependencies:
• Requests Library (HTTP client)
• Chromium Browser (for JavaScript rendering)
• PyQuery (for CSS selection)
• Async Support Layer
Key Relationships
• Data flow from HTTP requests to parsed results
• Interaction between synchronous and asynchronous components
• Communication between the main parser and selection engines
• Integration with external browser for JavaScript rendering
Specific Design Elements
• Blue: Core components
• Green: External dependencies
• Orange: Processing engines
• Gray: Storage/state management
• Rectangles: Main components
• Hexagons: Processing engines
• Clouds: External services
Important Features to Highlight
• Async/Sync dual support
• JavaScript rendering capability
• Multiple selector support (CSS, XPath)
• Session management
• Cookie persistence
• Connection pooling
Implementation Details
• Single file implementation (requests_html.py)
• Integration with existing Requests library
• Chromium browser management
• Async support architecture