# d4vinci/scrapling architecture

> Architecture diagram of the GitHub repository d4vinci/scrapling, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/d4vinci/scrapling
- Repository: https://github.com/d4vinci/scrapling
- Video tour (about a minute): https://gitdiagram.com/d4vinci/scrapling/video

## Overview

Scrapling is a Python web-scraping framework. Its main workflow is to fetch or render pages, parse responses into selectable elements, and optionally scale that work through spiders that schedule requests and produce exportable results. It also provides CLI and interactive-shell entry points, plus an MCP server for AI agents. The graph reflects documented capabilities and indexed references; some crawler internals and optional integrations are represented at directory level because their implementation was not excerpted.

Main entry points: [`scrapling/fetchers/__init__.py`](https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/__init__.py) (Fetcher API), [`scrapling/spiders/engine.py`](https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/engine.py) (Crawler engine), [`scrapling/engines/_browsers/_base.py`](https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/_browsers/_base.py) (Browser engine), [`scrapling/parser.py`](https://github.com/d4vinci/scrapling/blob/main/scrapling/parser.py) (HTML selection)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_fetch["Fetching and browsers"]
  node_fetch_api["Fetcher API<br/>[__init__.py]"]
  node_http_fetch["HTTP fetchers<br/>[requests.py]"]
  node_static_engine["Static engine<br/>[static.py]"]
  node_stealth_fetch["Stealth fetchers<br/>[stealth_chrome.py]"]
  node_browser_engine["Browser engine<br/>[_base.py]"]
  node_response["Response conversion<br/>[convertor.py]"]
  node_response_types["Response types<br/>[custom.py]"]
end

subgraph group_parse["Parsing and extraction"]
  node_parser["HTML selection<br/>[parser.py]"]
  node_adaptive[("Adaptive parsing<br/>[storage.py]")]
  node_markdown["Markdown output<br/>[shell.py]"]
end

subgraph group_crawl["Crawling and results"]
  node_spider["Spider definition<br/>[spider.py]"]
  node_crawler_engine["Crawler engine<br/>[engine.py]"]
  node_request_session["Sessions and requests<br/>[session.py]"]
  node_scheduler["Scheduling and policies"]
  node_results["Crawl results<br/>[result.py]"]
  node_templates["Spider templates"]
end

subgraph group_interfaces["User interfaces"]
  node_cli_shell["CLI and shell<br/>[cli.py]"]
end

subgraph group_ai["AI access"]
  node_mcp["MCP server<br/>[server.py]"]
  node_browser_actions["Browser actions"]
end

node_user(("Scraping user"))
node_website(("Target website"))

node_user -->|"requests pages"| node_fetch_api
node_fetch_api -->|"dispatches to"| node_http_fetch
node_fetch_api -->|"dispatches to"| node_stealth_fetch
node_http_fetch -->|"uses"| node_static_engine
node_stealth_fetch -->|"uses"| node_browser_engine
node_static_engine -->|"fetches from"| node_website
node_browser_engine -->|"renders"| node_website
node_static_engine -->|"converts response"| node_response
node_browser_engine -.->|"converts response"| node_response
node_response -->|"builds"| node_response_types
node_response_types -->|"provides page"| node_parser
node_user -->|"selects content"| node_parser
node_parser -->|"stores tracking data"| node_adaptive
node_parser -->|"renders Markdown"| node_markdown
node_user -->|"defines crawl"| node_spider
node_spider -->|"starts crawl"| node_crawler_engine
node_crawler_engine -->|"fetches requests"| node_request_session
node_request_session -->|"uses fetchers"| node_fetch_api
node_crawler_engine -->|"schedules work"| node_scheduler
node_crawler_engine -->|"collects results"| node_results
node_user -->|"uses templates"| node_templates
node_templates -->|"specializes"| node_spider
node_user -->|"runs commands"| node_cli_shell
node_cli_shell -->|"invokes fetchers"| node_fetch_api
node_cli_shell -->|"opens shell"| node_markdown
node_user -->|"asks AI agent"| node_mcp
node_mcp -->|"accesses scraping"| node_fetch_api
node_mcp -->|"delegates actions"| node_browser_actions
node_browser_actions -.->|"controls browser"| node_browser_engine

click node_fetch_api "https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/__init__.py"
click node_http_fetch "https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/requests.py"
click node_static_engine "https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/static.py"
click node_stealth_fetch "https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/stealth_chrome.py"
click node_browser_engine "https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/_browsers/_base.py"
click node_response "https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/toolbelt/convertor.py"
click node_response_types "https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/toolbelt/custom.py"
click node_parser "https://github.com/d4vinci/scrapling/blob/main/scrapling/parser.py"
click node_adaptive "https://github.com/d4vinci/scrapling/blob/main/scrapling/core/storage.py"
click node_markdown "https://github.com/d4vinci/scrapling/blob/main/scrapling/core/shell.py"
click node_spider "https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/spider.py"
click node_crawler_engine "https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/engine.py"
click node_request_session "https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/session.py"
click node_scheduler "https://github.com/d4vinci/scrapling/tree/main/scrapling/spiders"
click node_results "https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/result.py"
click node_templates "https://github.com/d4vinci/scrapling/tree/main/scrapling/spiders/templates"
click node_cli_shell "https://github.com/d4vinci/scrapling/blob/main/scrapling/cli.py"
click node_mcp "https://github.com/d4vinci/scrapling/blob/main/scrapling/core/ai/server.py"
click node_browser_actions "https://github.com/d4vinci/scrapling/blob/main/scrapling/core/ai/_browser_actions.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_fetch_api,node_http_fetch,node_static_engine,node_stealth_fetch,node_browser_engine,node_response,node_response_types,node_user toneBlue
class node_parser,node_adaptive,node_markdown toneAmber
class node_spider,node_crawler_engine,node_request_session,node_scheduler,node_results,node_templates toneMint
class node_cli_shell toneRose
class node_mcp,node_browser_actions,node_website toneIndigo
```

## Components

### Fetching and browsers

- [Fetcher API](https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/__init__.py): `scrapling/fetchers/__init__.py`
- [HTTP fetchers](https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/requests.py): `scrapling/fetchers/requests.py`
- [Static engine](https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/static.py): `scrapling/engines/static.py`
- [Stealth fetchers](https://github.com/d4vinci/scrapling/blob/main/scrapling/fetchers/stealth_chrome.py): `scrapling/fetchers/stealth_chrome.py`
- [Browser engine](https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/_browsers/_base.py): `scrapling/engines/_browsers/_base.py`
- [Response conversion](https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/toolbelt/convertor.py): `scrapling/engines/toolbelt/convertor.py`
- [Response types](https://github.com/d4vinci/scrapling/blob/main/scrapling/engines/toolbelt/custom.py): `scrapling/engines/toolbelt/custom.py`

### Parsing and extraction

- [HTML selection](https://github.com/d4vinci/scrapling/blob/main/scrapling/parser.py): `scrapling/parser.py`
- [Adaptive parsing](https://github.com/d4vinci/scrapling/blob/main/scrapling/core/storage.py): `scrapling/core/storage.py`
- [Markdown output](https://github.com/d4vinci/scrapling/blob/main/scrapling/core/shell.py): `scrapling/core/shell.py`

### Crawling and results

- [Spider definition](https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/spider.py): `scrapling/spiders/spider.py`
- [Crawler engine](https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/engine.py): `scrapling/spiders/engine.py`
- [Sessions and requests](https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/session.py): `scrapling/spiders/session.py`
- [Scheduling and policies](https://github.com/d4vinci/scrapling/tree/main/scrapling/spiders): `scrapling/spiders`
- [Crawl results](https://github.com/d4vinci/scrapling/blob/main/scrapling/spiders/result.py): `scrapling/spiders/result.py`
- [Spider templates](https://github.com/d4vinci/scrapling/tree/main/scrapling/spiders/templates): `scrapling/spiders/templates`

### User interfaces

- [CLI and shell](https://github.com/d4vinci/scrapling/blob/main/scrapling/cli.py): `scrapling/cli.py`

### AI access

- [MCP server](https://github.com/d4vinci/scrapling/blob/main/scrapling/core/ai/server.py): `scrapling/core/ai/server.py`
- [Browser actions](https://github.com/d4vinci/scrapling/blob/main/scrapling/core/ai/_browser_actions.py): `scrapling/core/ai/_browser_actions.py`

### Other components

- **Scraping user**
- **Target website**

## Connections

- Scraping user → Fetcher API: requests pages (evidence: `README.md`)
- Fetcher API → HTTP fetchers: dispatches to (evidence: `scrapling/fetchers/__init__.py`)
- Fetcher API → Stealth fetchers: dispatches to (evidence: `scrapling/fetchers/__init__.py`)
- HTTP fetchers → Static engine: uses (evidence: `scrapling/fetchers/requests.py`)
- Stealth fetchers → Browser engine: uses (evidence: `scrapling/fetchers/stealth_chrome.py`)
- Static engine → Target website: fetches from (evidence: `README.md`)
- Browser engine → Target website: renders (evidence: `README.md`)
- Static engine → Response conversion: converts response (evidence: `scrapling/engines/static.py`)
- Browser engine → Response conversion: converts response
- Response conversion → Response types: builds (evidence: `scrapling/engines/toolbelt/convertor.py`)
- Response types → HTML selection: provides page (evidence: `scrapling/engines/toolbelt/custom.py`)
- Scraping user → HTML selection: selects content (evidence: `README.md`)
- HTML selection → Adaptive parsing: stores tracking data (evidence: `scrapling/parser.py`)
- HTML selection → Markdown output: renders Markdown (evidence: `README.md`)
- Scraping user → Spider definition: defines crawl (evidence: `README.md`)
- Spider definition → Crawler engine: starts crawl (evidence: `scrapling/spiders/spider.py`)
- Crawler engine → Sessions and requests: fetches requests (evidence: `scrapling/spiders/engine.py`)
- Sessions and requests → Fetcher API: uses fetchers (evidence: `scrapling/spiders/session.py`)
- Crawler engine → Scheduling and policies: schedules work (evidence: `scrapling/spiders/engine.py`)
- Crawler engine → Crawl results: collects results (evidence: `scrapling/spiders/engine.py`)
- Scraping user → Spider templates: uses templates (evidence: `README.md`)
- Spider templates → Spider definition: specializes (evidence: `scrapling/spiders/templates/sitemap.py`)
- Scraping user → CLI and shell: runs commands (evidence: `README.md`)
- CLI and shell → Fetcher API: invokes fetchers (evidence: `scrapling/cli.py`)
- CLI and shell → Markdown output: opens shell (evidence: `scrapling/core/shell.py`)
- Scraping user → MCP server: asks AI agent (evidence: `README.md`)
- MCP server → Fetcher API: accesses scraping (evidence: `scrapling/core/ai/server.py`)
- MCP server → Browser actions: delegates actions (evidence: `scrapling/core/ai/server.py`)
- Browser actions → Browser engine: controls browser
