# helturkey/php-webtrawler architecture

> Architecture diagram of the GitHub repository helturkey/php-webtrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-18). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/helturkey/php-webtrawler
- Repository: https://github.com/helturkey/php-webtrawler

## Overview

PHP Scraper is a PHP toolkit for retrieving web pages, parsing and extracting data, and running resumable/concurrent crawls. The central application flow is caller → Scraper → Fetcher → HTTP transport → parsed Page, with adaptive matching and optional snapshots available during extraction. Separate product surfaces provide browser rendering, crawling, exports, CLI, MCP, and Laravel integration. Edges use the sampled source and SOURCE INDEX where available; README-documented capabilities are retained, while uncertain internal wiring is omitted.

Main entry points: [`src/Scraper.php`](https://github.com/helturkey/php-webtrawler/blob/main/src/Scraper.php) (Scraper API), [`src/Spider/SpiderRunner.php`](https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/SpiderRunner.php) (Spider runner), [`src/Http/Fetcher.php`](https://github.com/helturkey/php-webtrawler/blob/main/src/Http/Fetcher.php) (HTTP Fetcher), [`src/Parsing/Page.php`](https://github.com/helturkey/php-webtrawler/blob/main/src/Parsing/Page.php) (Parsed Page)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_core["Fetch and parse"]
  node_scraper["Scraper API<br/>[Scraper.php]"]
  node_fetcher["HTTP Fetcher<br/>[Fetcher.php]"]
  node_transport["HTTP transports<br/>[HttpTransport.php]"]
  node_page["Parsed Page<br/>[Page.php]"]
  node_element["DOM elements<br/>[Element.php]"]
  node_browser["Browser rendering"]
end

subgraph group_crawl["Crawling and state"]
  node_spider["Spider definition<br/>[Spider.php]"]
  node_runner["Spider runner<br/>[SpiderRunner.php]"]
  node_crawl_state[("Crawl checkpoint")]
  node_pipeline["Item pipeline<br/>[ItemPipeline.php]"]
end

subgraph group_integrations["Interfaces and integrations"]
  node_cli["CLI application<br/>[Application.php]"]
  node_mcp["MCP server<br/>[McpServer.php]"]
  node_laravel["Laravel integration"]
end

subgraph group_results["Matching and output"]
  node_adaptive["Adaptive matching"]
  node_snapshot["Selector snapshots"]
  node_snapshot_store[("Snapshot stores<br/>[SnapshotStore.php]")]
  node_export["Structured exporters<br/>[CsvExporter.php]"]
end

subgraph group_security["Request security"]
  node_security["URL and host policy<br/>[HostPolicy.php]"]
end

node_user(("Application caller"))

node_user -->|"calls"| node_scraper
node_scraper -->|"delegates to"| node_fetcher
node_fetcher -->|"sends requests"| node_transport
node_fetcher -->|"builds"| node_page
node_page -->|"exposes elements"| node_element
node_fetcher -->|"checks hosts"| node_security
node_page -->|"uses matcher"| node_adaptive
node_page -->|"reads and writes"| node_snapshot_store
node_snapshot -->|"captures structure"| node_element
node_scraper -->|"runs crawl"| node_runner
node_runner -->|"fetches pages"| node_fetcher
node_runner -->|"dispatches parsing"| node_spider
node_runner -->|"uses checkpoint"| node_crawl_state
node_runner -.->|"processes items"| node_pipeline
node_scraper -.->|"renders pages"| node_browser
node_cli -->|"fetches pages"| node_scraper
node_cli -->|"emits formats"| node_export
node_mcp -->|"uses API"| node_scraper
node_laravel -->|"registers service"| node_scraper

click node_scraper "https://github.com/helturkey/php-webtrawler/blob/main/src/Scraper.php"
click node_fetcher "https://github.com/helturkey/php-webtrawler/blob/main/src/Http/Fetcher.php"
click node_transport "https://github.com/helturkey/php-webtrawler/blob/main/src/Contracts/HttpTransport.php"
click node_page "https://github.com/helturkey/php-webtrawler/blob/main/src/Parsing/Page.php"
click node_element "https://github.com/helturkey/php-webtrawler/blob/main/src/Parsing/Element.php"
click node_security "https://github.com/helturkey/php-webtrawler/blob/main/src/Http/HostPolicy.php"
click node_adaptive "https://github.com/helturkey/php-webtrawler/blob/main/src/Adaptive/AdaptiveMatcher.php"
click node_snapshot "https://github.com/helturkey/php-webtrawler/blob/main/src/Adaptive/ElementSnapshot.php"
click node_snapshot_store "https://github.com/helturkey/php-webtrawler/blob/main/src/Contracts/SnapshotStore.php"
click node_spider "https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/Spider.php"
click node_runner "https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/SpiderRunner.php"
click node_crawl_state "https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/JsonCrawlStateStore.php"
click node_browser "https://github.com/helturkey/php-webtrawler/blob/main/src/Browser/PlaywrightBrowserDriver.php"
click node_cli "https://github.com/helturkey/php-webtrawler/blob/main/src/Cli/Application.php"
click node_mcp "https://github.com/helturkey/php-webtrawler/blob/main/src/Mcp/McpServer.php"
click node_laravel "https://github.com/helturkey/php-webtrawler/blob/main/src/Laravel/ScraperServiceProvider.php"
click node_export "https://github.com/helturkey/php-webtrawler/blob/main/src/Export/CsvExporter.php"
click node_pipeline "https://github.com/helturkey/php-webtrawler/blob/main/src/Contracts/ItemPipeline.php"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_scraper,node_fetcher,node_transport,node_page,node_element,node_browser toneBlue
class node_spider,node_runner,node_crawl_state,node_pipeline toneAmber
class node_cli,node_mcp,node_laravel toneMint
class node_adaptive,node_snapshot,node_snapshot_store,node_export toneRose
class node_security,node_user toneIndigo
```

## Components

### Fetch and parse

- [Scraper API](https://github.com/helturkey/php-webtrawler/blob/main/src/Scraper.php): `src/Scraper.php`
- [HTTP Fetcher](https://github.com/helturkey/php-webtrawler/blob/main/src/Http/Fetcher.php): `src/Http/Fetcher.php`
- [HTTP transports](https://github.com/helturkey/php-webtrawler/blob/main/src/Contracts/HttpTransport.php): `src/Contracts/HttpTransport.php`
- [Parsed Page](https://github.com/helturkey/php-webtrawler/blob/main/src/Parsing/Page.php): `src/Parsing/Page.php`
- [DOM elements](https://github.com/helturkey/php-webtrawler/blob/main/src/Parsing/Element.php): `src/Parsing/Element.php`
- [Browser rendering](https://github.com/helturkey/php-webtrawler/blob/main/src/Browser/PlaywrightBrowserDriver.php): `src/Browser/PlaywrightBrowserDriver.php`

### Crawling and state

- [Spider definition](https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/Spider.php): `src/Spider/Spider.php`
- [Spider runner](https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/SpiderRunner.php): `src/Spider/SpiderRunner.php`
- [Crawl checkpoint](https://github.com/helturkey/php-webtrawler/blob/main/src/Spider/JsonCrawlStateStore.php): `src/Spider/JsonCrawlStateStore.php`
- [Item pipeline](https://github.com/helturkey/php-webtrawler/blob/main/src/Contracts/ItemPipeline.php): `src/Contracts/ItemPipeline.php`

### Interfaces and integrations

- [CLI application](https://github.com/helturkey/php-webtrawler/blob/main/src/Cli/Application.php): `src/Cli/Application.php`
- [MCP server](https://github.com/helturkey/php-webtrawler/blob/main/src/Mcp/McpServer.php): `src/Mcp/McpServer.php`
- [Laravel integration](https://github.com/helturkey/php-webtrawler/blob/main/src/Laravel/ScraperServiceProvider.php): `src/Laravel/ScraperServiceProvider.php`

### Matching and output

- [Adaptive matching](https://github.com/helturkey/php-webtrawler/blob/main/src/Adaptive/AdaptiveMatcher.php): `src/Adaptive/AdaptiveMatcher.php`
- [Selector snapshots](https://github.com/helturkey/php-webtrawler/blob/main/src/Adaptive/ElementSnapshot.php): `src/Adaptive/ElementSnapshot.php`
- [Snapshot stores](https://github.com/helturkey/php-webtrawler/blob/main/src/Contracts/SnapshotStore.php): `src/Contracts/SnapshotStore.php`
- [Structured exporters](https://github.com/helturkey/php-webtrawler/blob/main/src/Export/CsvExporter.php): `src/Export/CsvExporter.php`

### Request security

- [URL and host policy](https://github.com/helturkey/php-webtrawler/blob/main/src/Http/HostPolicy.php): `src/Http/HostPolicy.php`

### Other components

- **Application caller**

## Connections

- Application caller → Scraper API: calls (evidence: `README.md`)
- Scraper API → HTTP Fetcher: delegates to (evidence: `src/Scraper.php`)
- HTTP Fetcher → HTTP transports: sends requests (evidence: `src/Http/Fetcher.php`)
- HTTP Fetcher → Parsed Page: builds (evidence: `src/Http/Fetcher.php`)
- Parsed Page → DOM elements: exposes elements (evidence: `src/Parsing/Page.php`)
- HTTP Fetcher → URL and host policy: checks hosts (evidence: `src/Http/Fetcher.php`)
- Parsed Page → Adaptive matching: uses matcher (evidence: `src/Parsing/Page.php`)
- Parsed Page → Snapshot stores: reads and writes (evidence: `src/Parsing/Page.php`)
- Selector snapshots → DOM elements: captures structure (evidence: `src/Adaptive/ElementSnapshot.php`)
- Scraper API → Spider runner: runs crawl (evidence: `src/Scraper.php`)
- Spider runner → HTTP Fetcher: fetches pages (evidence: `src/Spider/SpiderRunner.php`)
- Spider runner → Spider definition: dispatches parsing (evidence: `src/Spider/SpiderRunner.php`)
- Spider runner → Crawl checkpoint: uses checkpoint (evidence: `src/Spider/SpiderRunner.php`)
- Spider runner → Item pipeline: processes items (evidence: `src/Spider/SpiderRunner.php`)
- Scraper API → Browser rendering: renders pages (evidence: `README.md`)
- CLI application → Scraper API: fetches pages (evidence: `src/Cli/Application.php`)
- CLI application → Structured exporters: emits formats (evidence: `src/Cli/Application.php`)
- MCP server → Scraper API: uses API (evidence: `src/Mcp/McpServer.php`)
- Laravel integration → Scraper API: registers service (evidence: `src/Laravel/ScraperServiceProvider.php`)
