# spatie/crawler architecture

> Architecture diagram of the GitHub repository spatie/crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/spatie/crawler
- Repository: https://github.com/spatie/crawler

## Overview

##### Project type and purpose
- This is a PHP library (packagist: spatie/crawler) that provides a highly configurable web crawler.
- It’s not a standalone application but a tool/library to be integrated into PHP code to crawl URLs concurrently, optionally execute JavaScript (via Puppeteer), filter URLs, and notify observers of crawl events.

##### File structure
- Top-level:
- composer.json, README.md, LICENSE.md, CHANGELOG.md, UPGRADING.md, phpunit.xml – metadata, configuration, docs, test runner.
- src/ – all source code sub-divided by responsibility:
– CrawlObservers (observer interface & collection)
– CrawlProfiles (URL filtering strategies)
– CrawlQueues (URL queue abstractions and default in-memory queue)
– UrlParsers (link extraction strategies: HTML links, sitemap)
– Handlers (internal request‐fulfilled/failed handlers)
– Exceptions (invalid URLs, request failures)
– Core classes: Crawler.php (main façade), CrawlerRobots.php (robots.txt logic), CrawlUrl.php, Url.php, ResponseWithCachedBody.php
- tests/ – unit and integration tests including a small NodeJS server to simulate pages and JS rendering.

##### Key insights from the README
- Under the hood:
- Guzzle promises for concurrent HTTP requests (configurable concurrency).
- Optional JavaScript execution via Browsershot (Puppeteer + Chrome).
- Extensibility by plugging in custom CrawlProfile, CrawlQueue, CrawlObserver, UrlParser, Browsershot instance, etc.
- Many builder-style configuration methods on Crawler::create().

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User code
    A["Client Code\ncalls Crawler::create()"]:::user

    %% Core Engine
    subgraph "Core Engine" 
        direction TB
        Crawler["Crawler façade\nSpatie\\Crawler\\Crawler"]:::core
        CrawlLoop["Crawler Loop"]:::core
    end

    A -->|"create()/configure"| Crawler
    Crawler -->|"seed URLs\nenqueue()"| QueueIface

    %% Queue abstractions and implementations
    subgraph "Queue Module"
        direction TB
        QueueIface["CrawlQueue Interface"]:::iface
        ArrayQueue["ArrayCrawlQueue\n(in-memory)"]:::impl
    end
    click QueueIface "https://github.com/spatie/crawler/blob/main/src/CrawlQueues/CrawlQueue.php"
    click ArrayQueue "https://github.com/spatie/crawler/blob/main/src/CrawlQueues/ArrayCrawlQueue.php"

    QueueIface -->|"dequeue()"| CrawlLoop
    CrawlLoop -->|"shouldCrawl()"| ProfileIface

    %% Profile abstractions and strategies
    subgraph "Profile Module"
        direction TB
        ProfileIface["CrawlProfile Interface"]:::iface
        AllUrls["CrawlAllUrls"]:::impl
        InternalUrls["CrawlInternalUrls"]:::impl
        Subdomains["CrawlSubdomains"]:::impl
    end
    click ProfileIface "https://github.com/spatie/crawler/blob/main/src/CrawlProfiles/CrawlProfile.php"
    click AllUrls "https://github.com/spatie/crawler/blob/main/src/CrawlProfiles/CrawlAllUrls.php"
    click InternalUrls "https://github.com/spatie/crawler/blob/main/src/CrawlProfiles/CrawlInternalUrls.php"
    click Subdomains "https://github.com/spatie/crawler/blob/main/src/CrawlProfiles/CrawlSubdomains.php"

    ProfileIface -->|"implements"| AllUrls
    ProfileIface -->|"implements"| InternalUrls
    ProfileIface -->|"implements"| Subdomains
    ProfileIface -->|"passes"| Robots
    Robots -->|"allowed()"| CrawlLoop

    %% Robots.txt
    Robots["CrawlerRobots\n(spatie/robots-txt)"]:::external
    click Robots "https://github.com/spatie/crawler/blob/main/src/CrawlerRobots.php"

    %% URL value objects
    subgraph "URL Objects"
        direction TB
        URLObj["Url"]:::core
        CrawlUrlObj["CrawlUrl"]:::core
        ResponseObj["ResponseWithCachedBody"]:::core
    end
    click URLObj "https://github.com/spatie/crawler/blob/main/src/Url.php"
    click CrawlUrlObj "https://github.com/spatie/crawler/blob/main/src/CrawlUrl.php"
    click ResponseObj "https://github.com/spatie/crawler/blob/main/src/ResponseWithCachedBody.php"

    CrawlLoop -->|"build request"| URLObj
    CrawlLoop -->|"wrap URL"| CrawlUrlObj

    %% Parser abstractions and implementations
    subgraph "Parser Module"
        direction TB
        ParserIface["UrlParser Interface"]:::iface
        LinkParser["LinkUrlParser"]:::impl
        SitemapParser["SitemapUrlParser"]:::impl
    end
    click ParserIface "https://github.com/spatie/crawler/blob/main/src/UrlParsers/UrlParser.php"
    click LinkParser "https://github.com/spatie/crawler/blob/main/src/UrlParsers/LinkUrlParser.php"
    click SitemapParser "https://github.com/spatie/crawler/blob/main/src/UrlParsers/SitemapUrlParser.php"

    CrawlLoop -->|"extract URLs"| ParserIface
    ParserIface -->|"implementation"| LinkParser
    ParserIface -->|"implementation"| SitemapParser

    %% HTTP and JS layers
    subgraph "Request Layer"
        direction TB
        HTTP["Guzzle Client\n(promises)"]:::external
        Browsershot["Browsershot"]:::optional
        Puppeteer["Puppeteer"]:::external
        Chrome["Headless Chrome"]:::external
    end
    CrawlLoop -->|"dispatchPromise()"| HTTP
    CrawlLoop -->|"dispatchPromise() if JS"| Browsershot
    Browsershot -->|calls| Puppeteer
    Puppeteer -->|runs| Chrome
    HTTP -->|"response"| ResponseObj
    Chrome -->|"rendered response"| ResponseObj

    %% Handlers and Observers
    subgraph "Event Handling"
        direction TB
        SuccessHandler["CrawlRequestFulfilled"]:::core
        FailureHandler["CrawlRequestFailed"]:::core
        ObsIface["CrawlObserver Interface"]:::iface
        ObsColl["CrawlObserverCollection"]:::impl
    end
    click SuccessHandler "https://github.com/spatie/crawler/blob/main/src/Handlers/CrawlRequestFulfilled.php"
    click FailureHandler "https://github.com/spatie/crawler/blob/main/src/Handlers/CrawlRequestFailed.php"
    click ObsIface "https://github.com/spatie/crawler/blob/main/src/CrawlObservers/CrawlObserver.php"
    click ObsColl "https://github.com/spatie/crawler/blob/main/src/CrawlObservers/CrawlObserverCollection.php"

    ResponseObj -->|"onSuccess"| SuccessHandler
    ResponseObj -->|"onFailure"| FailureHandler
    SuccessHandler -->|"notify()"| ObsIface
    FailureHandler -->|"notify()"| ObsIface
    ObsIface -->|"collection"| ObsColl
    ObsColl -->|"optionally enqueue"| QueueIface

    %% Class Definitions for styling
    classDef core fill:#cce5ff,stroke:#004085,color:#004085;
    classDef iface stroke-dasharray: 5 5,fill:#e2e3e5,stroke:#6c757d,color:#6c757d;
    classDef impl fill:#d4edda,stroke:#155724,color:#155724;
    classDef external fill:#f8f9fa,stroke:#6c757d,color:#6c757d;
    classDef optional fill:#fff3cd,stroke:#856404,color:#856404;
    classDef user fill:#d1ecf1,stroke:#0c5460,color:#0c5460;
```
