How apify/crawlee works
Purpose and public entry points
Crawlee is a TypeScript/Node.js monorepo for reliable web scraping and browser automation. Consumers normally install the umbrella crawlee package, which re-exports crawler families and core APIs; packages can also be used independently. Main entry points: packages/crawlee/src/index.ts, packages/core/src/index.ts, and the project-bootstrap/run CLI at packages/cli/src/index.ts.
Core crawl orchestration
BasicCrawler is the execution foundation: it consumes requests, invokes user handlers, applies retries/error handling, and coordinates crawl lifecycle. Shared request models, routing, events, logging, configuration, statistics, and crawler extensions live in core. Autoscaling dynamically adjusts work based on CPU, memory, and event-loop signals.
• packages/basic-crawler/src/internals/basic-crawler.ts
• packages/core/src/request.ts
• packages/core/src/autoscaling/autoscaled_pool.ts
Request discovery and scheduling
A crawl starts with supplied URLs or request lists, then handlers can discover and enqueue links. Request queues/lists provide persistent scheduling, deduplication, and breadth/depth-oriented traversal; sitemap lists are another input source. This is the primary feedback loop: fetch/process → extract links → normalize/filter → enqueue → next request.
• packages/core/src/enqueue_links/enqueue_links.ts
• packages/core/src/storages/request_queue.ts
• packages/core/src/storages/sitemap_request_list.ts
Crawler execution variants
Specialized packages implement a common crawler-facing model over distinct acquisition/rendering strategies: raw HTTP, Cheerio DOM parsing, JSDOM/LinkeDOM DOM emulation, and browser-driven Playwright/Puppeteer. The browser crawlers add page/context semantics and launchers; Playwright also has adaptive rendering selection. HTTP clients are abstracted so crawler implementations can use built-in or custom transport.
• packages/http-crawler/src/internals/http-crawler.ts
• packages/cheerio-crawler/src/internals/cheerio-crawler.ts
• packages/playwright-crawler/src/internals/playwright-crawler.ts
Browser lifecycle, identity, and network boundary
Browser-based crawlers delegate browser creation and reuse to BrowserPool. Plugins/controllers isolate Playwright and Puppeteer integration; launch contexts, proxy servers, proxy anonymization, and fingerprinting hooks sit at the browser/network boundary. External browser runtimes are Playwright and Puppeteer, with support evidenced for Chromium and Firefox; Stagehand is an additional AI-oriented crawler integration.
• packages/browser-pool/src/browser-pool.ts
• packages/browser-pool/src/abstract-classes/browser-plugin.ts
• packages/stagehand-crawler/src/internals/stagehand-crawler.ts
State, outputs, sessions, and proxies
Core storage abstractions expose datasets for records, key-value stores for files/state, and request sources. The default local implementation is memory-storage, which supports in-memory and filesystem-backed resource clients; an alternate storage client can represent cloud-backed services such as Apify platform storage. Sessions retain cookies/identity and combine with proxy configuration for rotation and block recovery.
• packages/core/src/storages/storage_manager.ts
• packages/memory-storage/src/memory-storage.ts
• packages/core/src/session_pool/session_pool.ts
Delivery and operational surfaces
The repo ships npm packages, a CLI that creates/runs generated projects, templates with Dockerfiles, and a Docusaurus documentation website served through Nginx deployment automation. Runtime baseline is Node.js 16+. External operational targets include npm publishing, Apify platform deployment, Docker, AWS/GCP deployment guidance, and target websites/proxies.
• packages/cli/src/commands/CreateProjectCommand.ts
• packages/templates/templates/playwright-ts/Dockerfile
• website/docusaurus.config.js
Main entry points: basic-crawler.ts BasicCrawler · src/index.ts crawlee umbrella · playwright-crawler.ts PlaywrightCrawler · src/index.ts Core APIs
Components 23
Published packages
Composable crawler and runtime packages.
- crawlee umbrella · Re-exports crawler families and core APIs.
packages/crawlee/src/index.ts - Core APIs · Shared models, routing, events, configuration, and extensions.
packages/core/src/index.ts - HttpCrawler · Acquires pages through built-in or custom HTTP transport.
packages/http-crawler/src/internals/http-crawler.ts - CheerioCrawler · Processes fetched HTML with Cheerio.
packages/cheerio-crawler/src/internals/cheerio-crawler.ts - Storage manager · Exposes datasets, key-value stores, and request sources.
packages/core/src/storages/storage_manager.ts - Memory storage · Provides in-memory and filesystem-backed resources.
packages/memory-storage/src/memory-storage.ts - Session pool · Retains cookies and identity for rotation and recovery.
packages/core/src/session_pool/session_pool.ts
Crawl execution
Request processing and scheduling loop.
- BasicCrawler · Runs handlers, retries, errors, and crawl lifecycle.
packages/basic-crawler/src/internals/basic-crawler.ts - AutoscaledPool · Adjusts work from CPU, memory, and event-loop signals.
packages/core/src/autoscaling/autoscaled_pool.ts - Request model
packages/core/src/request.ts - Request queue · Deduplicates and schedules traversal requests.
packages/core/src/storages/request_queue.ts - Link enqueueing · Normalizes, filters, and enqueues discovered links.
packages/core/src/enqueue_links/enqueue_links.ts - Sitemap request list
packages/core/src/storages/sitemap_request_list.ts
Browser automation
Browser lifecycle and rendering integrations.
- PlaywrightCrawler · Adds browser page/context handling and adaptive rendering.
packages/playwright-crawler/src/internals/playwright-crawler.ts - StagehandCrawler
packages/stagehand-crawler/src/internals/stagehand-crawler.ts - BrowserPool · Creates and reuses browser instances.
packages/browser-pool/src/browser-pool.ts - Browser plugins · Isolates Playwright and Puppeteer controllers.
packages/browser-pool/src/abstract-classes/browser-plugin.ts - Browser runtimes · Playwright/Puppeteer with Chromium and Firefox support.
- Websites and proxies
Delivery surfaces
Project bootstrap, deployment, and documentation.
- CLI · Creates and runs generated projects.
packages/cli/src/index.ts - CreateProjectCommand
packages/cli/src/commands/CreateProjectCommand.ts - Playwright Docker template
packages/templates/templates/playwright-ts/Dockerfile - Documentation site · Published through Nginx deployment automation.
website/docusaurus.config.js
Connections 27
- crawlee umbrella → Core APIs: re-exports
- crawlee umbrella → BasicCrawler: re-exports
- crawlee umbrella → HttpCrawler: re-exports
- crawlee umbrella → CheerioCrawler: re-exports
- crawlee umbrella → PlaywrightCrawler: re-exports
- BasicCrawler → Core APIs: uses shared runtime
- BasicCrawler → AutoscaledPool: coordinates concurrency
- BasicCrawler → Request queue: consumes requests
- Sitemap request list → Request queue: supplies requests
- Request queue → Request model: schedules
- BasicCrawler → Link enqueueing: handlers discover links
- Link enqueueing → Request queue: enqueues next work
- HttpCrawler → BasicCrawler: specializes
- CheerioCrawler → HttpCrawler: adds DOM parsing
- PlaywrightCrawler → BasicCrawler: specializes
- StagehandCrawler → PlaywrightCrawler: extends browser crawling
- PlaywrightCrawler → BrowserPool: delegates browser reuse
- BrowserPool → Browser plugins: uses
- Browser plugins → Browser runtimes: integrates
- Browser runtimes → Websites and proxies: browses via proxies
- Session pool → Websites and proxies: rotates identity
- Core APIs → Storage manager: provides abstraction
- Storage manager → Memory storage: defaults to
- Core APIs → Session pool: manages
- CLI → CreateProjectCommand: runs
- CreateProjectCommand → Playwright Docker template: generates project from
- Playwright Docker template → PlaywrightCrawler: deploys crawler project