How helturkey/php-webtrawler works
PHP Scraper is a PHP toolkit for retrieving web pages, parsing and extracting data, and running resumable/concurrent crawls. The central application flow is caller → Scraper → Fetcher → HTTP transport → parsed Page, with adaptive matching and optional snapshots available during extraction. Separate product surfaces provide browser rendering, crawling, exports, CLI, MCP, and Laravel integration. Edges use the sampled source and SOURCE INDEX where available; README-documented capabilities are retained, while uncertain internal wiring is omitted.
Main entry points: Scraper.php Scraper API · SpiderRunner.php Spider runner · Fetcher.php HTTP Fetcher · Page.php Parsed Page
Components 19
Fetch and parse
- Scraper API
src/Scraper.php - HTTP Fetcher
src/Http/Fetcher.php - HTTP transports
src/Contracts/HttpTransport.php - Parsed Page
src/Parsing/Page.php - DOM elements
src/Parsing/Element.php - Browser rendering
src/Browser/PlaywrightBrowserDriver.php
Crawling and state
- Spider definition
src/Spider/Spider.php - Spider runner
src/Spider/SpiderRunner.php - Crawl checkpoint
src/Spider/JsonCrawlStateStore.php - Item pipeline
src/Contracts/ItemPipeline.php
Interfaces and integrations
- CLI application
src/Cli/Application.php - MCP server
src/Mcp/McpServer.php - Laravel integration
src/Laravel/ScraperServiceProvider.php
Matching and output
- Adaptive matching
src/Adaptive/AdaptiveMatcher.php - Selector snapshots
src/Adaptive/ElementSnapshot.php - Snapshot stores
src/Contracts/SnapshotStore.php - Structured exporters
src/Export/CsvExporter.php
Request security
- URL and host policy
src/Http/HostPolicy.php
Other components
- Application caller
Connections 19
- Application caller → Scraper API: calls
README.md - Scraper API → HTTP Fetcher: delegates to
src/Scraper.php - HTTP Fetcher → HTTP transports: sends requests
src/Http/Fetcher.php - HTTP Fetcher → Parsed Page: builds
src/Http/Fetcher.php - Parsed Page → DOM elements: exposes elements
src/Parsing/Page.php - HTTP Fetcher → URL and host policy: checks hosts
src/Http/Fetcher.php - Parsed Page → Adaptive matching: uses matcher
src/Parsing/Page.php - Parsed Page → Snapshot stores: reads and writes
src/Parsing/Page.php - Selector snapshots → DOM elements: captures structure
src/Adaptive/ElementSnapshot.php - Scraper API → Spider runner: runs crawl
src/Scraper.php - Spider runner → HTTP Fetcher: fetches pages
src/Spider/SpiderRunner.php - Spider runner → Spider definition: dispatches parsing
src/Spider/SpiderRunner.php - Spider runner → Crawl checkpoint: uses checkpoint
src/Spider/SpiderRunner.php - Spider runner → Item pipeline: processes items
src/Spider/SpiderRunner.php - Scraper API → Browser rendering: renders pages
README.md - CLI application → Scraper API: fetches pages
src/Cli/Application.php - CLI application → Structured exporters: emits formats
src/Cli/Application.php - MCP server → Scraper API: uses API
src/Mcp/McpServer.php - Laravel integration → Scraper API: registers service
src/Laravel/ScraperServiceProvider.php