# psf/requests-html architecture

> Architecture diagram of the GitHub repository psf/requests-html, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/psf/requests-html
- Repository: https://github.com/psf/requests-html

## Overview

##### Main Components

a) Core Components:
- Requests-HTML Client Layer
- HTML Parsing Engine
- JavaScript Rendering Engine (Chromium via pyppeteer)
- Selector Engines (CSS/jQuery via PyQuery, XPath)

b) External Dependencies:
- Requests Library (HTTP client)
- Chromium Browser (for JavaScript rendering)
- PyQuery (for CSS selection)
- Async Support Layer

##### Key Relationships

- Data flow from HTTP requests to parsed results
- Interaction between synchronous and asynchronous components
- Communication between the main parser and selection engines
- Integration with external browser for JavaScript rendering

##### Specific Design Elements

- Blue: Core components
- Green: External dependencies
- Orange: Processing engines
- Gray: Storage/state management

- Rectangles: Main components
- Hexagons: Processing engines
- Clouds: External services

##### Important Features to Highlight

- Async/Sync dual support
- JavaScript rendering capability
- Multiple selector support (CSS, XPath)
- Session management
- Cookie persistence
- Connection pooling

##### Implementation Details

- Single file implementation (requests_html.py)
- Integration with existing Requests library
- Chromium browser management
- Async support architecture

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Styles and classes
    classDef core fill:#b3d1ff,stroke:#4d88ff
    classDef external fill:#90EE90,stroke:#228B22
    classDef processing fill:#FFB366,stroke:#FF8000
    classDef storage fill:#D3D3D3,stroke:#808080

    %% Legend
    subgraph Legend
        L1[Core Components]:::core
        L2[External Dependencies]:::external
        L3[Processing Engines]:::processing
        L4[Storage Components]:::storage
    end

    %% Client Interface Layer
    subgraph "Client Interface Layer"
        HTMLSession[HTMLSession]:::core
        AsyncHTMLSession[AsyncHTMLSession]:::core
        HTMLParser[HTML Parser]:::core
    end

    %% Processing Layer
    subgraph "Processing Layer"
        RequestHandler[Request Handler]:::processing
        ResponseProcessor[Response Processor]:::processing
        ParsingEngine[HTML Parsing Engine]:::processing
        JSRenderer[JavaScript Renderer]:::processing
    end

    %% Selection Engines Layer
    subgraph "Selection Engines"
        CSSSelector[CSS Selector]:::processing
        XPathSelector[XPath Selector]:::processing
        LinkExtractor[Link Extractor]:::processing
    end

    %% External Services Layer
    subgraph "External Services"
        ChromiumBrowser[Chromium Browser]:::external
        RequestsLib[Requests Library]:::external
        PyQueryLib[PyQuery Library]:::external
        ConnectionPool[(Connection Pool)]:::storage
        CookieStore[(Cookie Storage)]:::storage
    end

    %% Relationships
    HTMLSession --> RequestHandler
    AsyncHTMLSession --> RequestHandler
    HTMLParser --> ParsingEngine
    RequestHandler --> RequestsLib
    RequestHandler --> ConnectionPool
    ResponseProcessor --> ParsingEngine
    ParsingEngine --> CSSSelector
    ParsingEngine --> XPathSelector
    ParsingEngine --> LinkExtractor
    JSRenderer --> ChromiumBrowser
    CSSSelector --> PyQueryLib
    RequestHandler --> CookieStore

    %% Click events for component mapping
    click HTMLSession "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click AsyncHTMLSession "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click HTMLParser "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click RequestHandler "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click ResponseProcessor "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click ParsingEngine "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click JSRenderer "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click CSSSelector "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click XPathSelector "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click LinkExtractor "https://github.com/psf/requests-html/blob/master/requests_html.py"
    click ChromiumBrowser "https://github.com/psf/requests-html/blob/master/tests/test_internet.py"
    click CookieStore "https://github.com/psf/requests-html/blob/master/tests/test_requests_html.py"
```
