How spatie/crawler works
Project type and purpose
• This is a PHP library (packagist: spatie/crawler) that provides a highly configurable web crawler.
• It’s not a standalone application but a tool/library to be integrated into PHP code to crawl URLs concurrently, optionally execute JavaScript (via Puppeteer), filter URLs, and notify observers of crawl events.
File structure
• Top-level:
• composer.json, README.md, LICENSE.md, CHANGELOG.md, UPGRADING.md, phpunit.xml – metadata, configuration, docs, test runner.
• src/ – all source code sub-divided by responsibility:
– CrawlObservers (observer interface & collection)
– CrawlProfiles (URL filtering strategies)
– CrawlQueues (URL queue abstractions and default in-memory queue)
– UrlParsers (link extraction strategies: HTML links, sitemap)
– Handlers (internal request‐fulfilled/failed handlers)
– Exceptions (invalid URLs, request failures)
– Core classes: Crawler.php (main façade), CrawlerRobots.php (robots.txt logic), CrawlUrl.php, Url.php, ResponseWithCachedBody.php
• tests/ – unit and integration tests including a small NodeJS server to simulate pages and JS rendering.
Key insights from the README
• Under the hood:
• Guzzle promises for concurrent HTTP requests (configurable concurrency).
• Optional JavaScript execution via Browsershot (Puppeteer + Chrome).
• Extensibility by plugging in custom CrawlProfile, CrawlQueue, CrawlObserver, UrlParser, Browsershot instance, etc.
• Many builder-style configuration methods on Crawler::create().