How d4vinci/scrapling works
Scrapling is a Python web-scraping framework. Its main workflow is to fetch or render pages, parse responses into selectable elements, and optionally scale that work through spiders that schedule requests and produce exportable results. It also provides CLI and interactive-shell entry points, plus an MCP server for AI agents. The graph reflects documented capabilities and indexed references; some crawler internals and optional integrations are represented at directory level because their implementation was not excerpted.
Main entry points: __init__.py Fetcher API · engine.py Crawler engine · _base.py Browser engine · parser.py HTML selection
Components 21
Fetching and browsers
- Fetcher API
scrapling/fetchers/__init__.py - HTTP fetchers
scrapling/fetchers/requests.py - Static engine
scrapling/engines/static.py - Stealth fetchers
scrapling/fetchers/stealth_chrome.py - Browser engine
scrapling/engines/_browsers/_base.py - Response conversion
scrapling/engines/toolbelt/convertor.py - Response types
scrapling/engines/toolbelt/custom.py
Parsing and extraction
- HTML selection
scrapling/parser.py - Adaptive parsing
scrapling/core/storage.py - Markdown output
scrapling/core/shell.py
Crawling and results
- Spider definition
scrapling/spiders/spider.py - Crawler engine
scrapling/spiders/engine.py - Sessions and requests
scrapling/spiders/session.py - Scheduling and policies
scrapling/spiders - Crawl results
scrapling/spiders/result.py - Spider templates
scrapling/spiders/templates
User interfaces
- CLI and shell
scrapling/cli.py
AI access
- MCP server
scrapling/core/ai/server.py - Browser actions
scrapling/core/ai/_browser_actions.py
Other components
- Scraping user
- Target website
Connections 29
- Scraping user → Fetcher API: requests pages
README.md - Fetcher API → HTTP fetchers: dispatches to
scrapling/fetchers/__init__.py - Fetcher API → Stealth fetchers: dispatches to
scrapling/fetchers/__init__.py - HTTP fetchers → Static engine: uses
scrapling/fetchers/requests.py - Stealth fetchers → Browser engine: uses
scrapling/fetchers/stealth_chrome.py - Static engine → Target website: fetches from
README.md - Browser engine → Target website: renders
README.md - Static engine → Response conversion: converts response
scrapling/engines/static.py - Browser engine → Response conversion: converts response
- Response conversion → Response types: builds
scrapling/engines/toolbelt/convertor.py - Response types → HTML selection: provides page
scrapling/engines/toolbelt/custom.py - Scraping user → HTML selection: selects content
README.md - HTML selection → Adaptive parsing: stores tracking data
scrapling/parser.py - HTML selection → Markdown output: renders Markdown
README.md - Scraping user → Spider definition: defines crawl
README.md - Spider definition → Crawler engine: starts crawl
scrapling/spiders/spider.py - Crawler engine → Sessions and requests: fetches requests
scrapling/spiders/engine.py - Sessions and requests → Fetcher API: uses fetchers
scrapling/spiders/session.py - Crawler engine → Scheduling and policies: schedules work
scrapling/spiders/engine.py - Crawler engine → Crawl results: collects results
scrapling/spiders/engine.py - Scraping user → Spider templates: uses templates
README.md - Spider templates → Spider definition: specializes
scrapling/spiders/templates/sitemap.py - Scraping user → CLI and shell: runs commands
README.md - CLI and shell → Fetcher API: invokes fetchers
scrapling/cli.py - CLI and shell → Markdown output: opens shell
scrapling/core/shell.py - Scraping user → MCP server: asks AI agent
README.md - MCP server → Fetcher API: accesses scraping
scrapling/core/ai/server.py - MCP server → Browser actions: delegates actions
scrapling/core/ai/server.py - Browser actions → Browser engine: controls browser