# apify/crawlee architecture

> Architecture diagram of the GitHub repository apify/crawlee, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/apify/crawlee
- Repository: https://github.com/apify/crawlee

## Overview

#### Purpose and public entry points
Crawlee is a TypeScript/Node.js monorepo for reliable web scraping and browser automation. Consumers normally install the umbrella `crawlee` package, which re-exports crawler families and core APIs; packages can also be used independently. Main entry points: `packages/crawlee/src/index.ts`, `packages/core/src/index.ts`, and the project-bootstrap/run CLI at `packages/cli/src/index.ts`.

#### Core crawl orchestration
`BasicCrawler` is the execution foundation: it consumes requests, invokes user handlers, applies retries/error handling, and coordinates crawl lifecycle. Shared request models, routing, events, logging, configuration, statistics, and crawler extensions live in core. Autoscaling dynamically adjusts work based on CPU, memory, and event-loop signals.
- `packages/basic-crawler/src/internals/basic-crawler.ts`
- `packages/core/src/request.ts`
- `packages/core/src/autoscaling/autoscaled_pool.ts`

#### Request discovery and scheduling
A crawl starts with supplied URLs or request lists, then handlers can discover and enqueue links. Request queues/lists provide persistent scheduling, deduplication, and breadth/depth-oriented traversal; sitemap lists are another input source. This is the primary feedback loop: fetch/process → extract links → normalize/filter → enqueue → next request.
- `packages/core/src/enqueue_links/enqueue_links.ts`
- `packages/core/src/storages/request_queue.ts`
- `packages/core/src/storages/sitemap_request_list.ts`

#### Crawler execution variants
Specialized packages implement a common crawler-facing model over distinct acquisition/rendering strategies: raw HTTP, Cheerio DOM parsing, JSDOM/LinkeDOM DOM emulation, and browser-driven Playwright/Puppeteer. The browser crawlers add page/context semantics and launchers; Playwright also has adaptive rendering selection. HTTP clients are abstracted so crawler implementations can use built-in or custom transport.
- `packages/http-crawler/src/internals/http-crawler.ts`
- `packages/cheerio-crawler/src/internals/cheerio-crawler.ts`
- `packages/playwright-crawler/src/internals/playwright-crawler.ts`

#### Browser lifecycle, identity, and network boundary
Browser-based crawlers delegate browser creation and reuse to BrowserPool. Plugins/controllers isolate Playwright and Puppeteer integration; launch contexts, proxy servers, proxy anonymization, and fingerprinting hooks sit at the browser/network boundary. External browser runtimes are Playwright and Puppeteer, with support evidenced for Chromium and Firefox; Stagehand is an additional AI-oriented crawler integration.
- `packages/browser-pool/src/browser-pool.ts`
- `packages/browser-pool/src/abstract-classes/browser-plugin.ts`
- `packages/stagehand-crawler/src/internals/stagehand-crawler.ts`

#### State, outputs, sessions, and proxies
Core storage abstractions expose datasets for records, key-value stores for files/state, and request sources. The default local implementation is `memory-storage`, which supports in-memory and filesystem-backed resource clients; an alternate storage client can represent cloud-backed services such as Apify platform storage. Sessions retain cookies/identity and combine with proxy configuration for rotation and block recovery.
- `packages/core/src/storages/storage_manager.ts`
- `packages/memory-storage/src/memory-storage.ts`
- `packages/core/src/session_pool/session_pool.ts`

#### Delivery and operational surfaces
The repo ships npm packages, a CLI that creates/runs generated projects, templates with Dockerfiles, and a Docusaurus documentation website served through Nginx deployment automation. Runtime baseline is Node.js 16+. External operational targets include npm publishing, Apify platform deployment, Docker, AWS/GCP deployment guidance, and target websites/proxies.
- `packages/cli/src/commands/CreateProjectCommand.ts`
- `packages/templates/templates/playwright-ts/Dockerfile`
- `website/docusaurus.config.js`

Main entry points: [`packages/basic-crawler/src/internals/basic-crawler.ts`](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts) (BasicCrawler), [`packages/crawlee/src/index.ts`](https://github.com/apify/crawlee/blob/master/packages/crawlee/src/index.ts) (crawlee umbrella), [`packages/playwright-crawler/src/internals/playwright-crawler.ts`](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts) (PlaywrightCrawler), [`packages/core/src/index.ts`](https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts) (Core APIs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_packages["Published packages"]
  node_crawlee["crawlee umbrella<br/>package entry<br/>[index.ts]"]
  node_core["Core APIs<br/>shared runtime<br/>[index.ts]"]
  node_http["HttpCrawler<br/>HTTP crawler<br/>[http-crawler.ts]"]
  node_cheerio["CheerioCrawler<br/>DOM crawler<br/>[cheerio-crawler.ts]"]
  node_storage["Storage manager<br/>storage abstraction<br/>[storage_manager.ts]"]
  node_memory[("Memory storage<br/>default storage client<br/>[memory-storage.ts]")]
  node_sessions["Session pool<br/>identity management<br/>[session_pool.ts]"]
end

subgraph group_execution["Crawl execution"]
  node_basic["BasicCrawler<br/>crawler foundation<br/>[basic-crawler.ts]"]
  node_autoscaling["AutoscaledPool<br/>concurrency control<br/>[autoscaled_pool.ts]"]
  node_requests["Request model<br/>crawl input<br/>[request.ts]"]
  node_queue["Request queue<br/>persistent scheduler<br/>[request_queue.ts]"]
  node_enqueue["Link enqueueing<br/>discovery pipeline<br/>[enqueue_links.ts]"]
  node_sitemap["Sitemap request list<br/>request source"]
end

subgraph group_browser["Browser automation"]
  node_playwright["PlaywrightCrawler<br/>browser crawler"]
  node_stagehand["StagehandCrawler<br/>AI browser crawler"]
  node_browserpool["BrowserPool<br/>browser lifecycle<br/>[browser-pool.ts]"]
  node_plugin["Browser plugins<br/>integration boundary<br/>[browser-plugin.ts]"]
  node_browsers{{"Browser runtimes<br/>external runtime"}}
  node_targets{{"Websites and proxies<br/>external network"}}
end

subgraph group_delivery["Delivery surfaces"]
  node_cli["CLI<br/>project bootstrap<br/>[index.ts]"]
  node_create["CreateProjectCommand<br/>project generator"]
  node_template["Playwright Docker template<br/>deployment template"]
  node_docs["Documentation site<br/>Docusaurus site"]
end

node_crawlee -->|"re-exports"| node_core
node_crawlee -->|"re-exports"| node_basic
node_crawlee -->|"re-exports"| node_http
node_crawlee -->|"re-exports"| node_cheerio
node_crawlee -->|"re-exports"| node_playwright
node_basic -->|"uses shared runtime"| node_core
node_basic -->|"coordinates concurrency"| node_autoscaling
node_basic -->|"consumes requests"| node_queue
node_sitemap -->|"supplies requests"| node_queue
node_queue -->|"schedules"| node_requests
node_basic -->|"handlers discover links"| node_enqueue
node_enqueue -->|"enqueues next work"| node_queue
node_http -->|"specializes"| node_basic
node_cheerio -->|"adds DOM parsing"| node_http
node_playwright -->|"specializes"| node_basic
node_stagehand -->|"extends browser crawling"| node_playwright
node_playwright -->|"delegates browser reuse"| node_browserpool
node_browserpool -->|"uses"| node_plugin
node_plugin -->|"integrates"| node_browsers
node_browsers -->|"browses via proxies"| node_targets
node_sessions -->|"rotates identity"| node_targets
node_core -->|"provides abstraction"| node_storage
node_storage -->|"defaults to"| node_memory
node_core -->|"manages"| node_sessions
node_cli -->|"runs"| node_create
node_create -->|"generates project from"| node_template
node_template -.->|"deploys crawler project"| node_playwright

click node_crawlee "https://github.com/apify/crawlee/blob/master/packages/crawlee/src/index.ts"
click node_core "https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts"
click node_basic "https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts"
click node_autoscaling "https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/autoscaled_pool.ts"
click node_requests "https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts"
click node_queue "https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_queue.ts"
click node_enqueue "https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/enqueue_links.ts"
click node_sitemap "https://github.com/apify/crawlee/blob/master/packages/core/src/storages/sitemap_request_list.ts"
click node_http "https://github.com/apify/crawlee/blob/master/packages/http-crawler/src/internals/http-crawler.ts"
click node_cheerio "https://github.com/apify/crawlee/blob/master/packages/cheerio-crawler/src/internals/cheerio-crawler.ts"
click node_playwright "https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts"
click node_stagehand "https://github.com/apify/crawlee/blob/master/packages/stagehand-crawler/src/internals/stagehand-crawler.ts"
click node_browserpool "https://github.com/apify/crawlee/blob/master/packages/browser-pool/src/browser-pool.ts"
click node_plugin "https://github.com/apify/crawlee/blob/master/packages/browser-pool/src/abstract-classes/browser-plugin.ts"
click node_storage "https://github.com/apify/crawlee/blob/master/packages/core/src/storages/storage_manager.ts"
click node_memory "https://github.com/apify/crawlee/blob/master/packages/memory-storage/src/memory-storage.ts"
click node_sessions "https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session_pool.ts"
click node_cli "https://github.com/apify/crawlee/blob/master/packages/cli/src/index.ts"
click node_create "https://github.com/apify/crawlee/blob/master/packages/cli/src/commands/CreateProjectCommand.ts"
click node_template "https://github.com/apify/crawlee/blob/master/packages/templates/templates/playwright-ts/Dockerfile"
click node_docs "https://github.com/apify/crawlee/blob/master/website/docusaurus.config.js"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_crawlee,node_core,node_http,node_cheerio,node_storage,node_memory,node_sessions toneBlue
class node_basic,node_autoscaling,node_requests,node_queue,node_enqueue,node_sitemap toneAmber
class node_playwright,node_stagehand,node_browserpool,node_plugin,node_browsers,node_targets toneMint
class node_cli,node_create,node_template,node_docs toneRose
```

## Components

### Published packages

Composable crawler and runtime packages.

- [crawlee umbrella](https://github.com/apify/crawlee/blob/master/packages/crawlee/src/index.ts): package entry · Re-exports crawler families and core APIs. · `packages/crawlee/src/index.ts`
- [Core APIs](https://github.com/apify/crawlee/blob/master/packages/core/src/index.ts): shared runtime · Shared models, routing, events, configuration, and extensions. · `packages/core/src/index.ts`
- [HttpCrawler](https://github.com/apify/crawlee/blob/master/packages/http-crawler/src/internals/http-crawler.ts): HTTP crawler · Acquires pages through built-in or custom HTTP transport. · `packages/http-crawler/src/internals/http-crawler.ts`
- [CheerioCrawler](https://github.com/apify/crawlee/blob/master/packages/cheerio-crawler/src/internals/cheerio-crawler.ts): DOM crawler · Processes fetched HTML with Cheerio. · `packages/cheerio-crawler/src/internals/cheerio-crawler.ts`
- [Storage manager](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/storage_manager.ts): storage abstraction · Exposes datasets, key-value stores, and request sources. · `packages/core/src/storages/storage_manager.ts`
- [Memory storage](https://github.com/apify/crawlee/blob/master/packages/memory-storage/src/memory-storage.ts): default storage client · Provides in-memory and filesystem-backed resources. · `packages/memory-storage/src/memory-storage.ts`
- [Session pool](https://github.com/apify/crawlee/blob/master/packages/core/src/session_pool/session_pool.ts): identity management · Retains cookies and identity for rotation and recovery. · `packages/core/src/session_pool/session_pool.ts`

### Crawl execution

Request processing and scheduling loop.

- [BasicCrawler](https://github.com/apify/crawlee/blob/master/packages/basic-crawler/src/internals/basic-crawler.ts): crawler foundation · Runs handlers, retries, errors, and crawl lifecycle. · `packages/basic-crawler/src/internals/basic-crawler.ts`
- [AutoscaledPool](https://github.com/apify/crawlee/blob/master/packages/core/src/autoscaling/autoscaled_pool.ts): concurrency control · Adjusts work from CPU, memory, and event-loop signals. · `packages/core/src/autoscaling/autoscaled_pool.ts`
- [Request model](https://github.com/apify/crawlee/blob/master/packages/core/src/request.ts): crawl input · `packages/core/src/request.ts`
- [Request queue](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/request_queue.ts): persistent scheduler · Deduplicates and schedules traversal requests. · `packages/core/src/storages/request_queue.ts`
- [Link enqueueing](https://github.com/apify/crawlee/blob/master/packages/core/src/enqueue_links/enqueue_links.ts): discovery pipeline · Normalizes, filters, and enqueues discovered links. · `packages/core/src/enqueue_links/enqueue_links.ts`
- [Sitemap request list](https://github.com/apify/crawlee/blob/master/packages/core/src/storages/sitemap_request_list.ts): request source · `packages/core/src/storages/sitemap_request_list.ts`

### Browser automation

Browser lifecycle and rendering integrations.

- [PlaywrightCrawler](https://github.com/apify/crawlee/blob/master/packages/playwright-crawler/src/internals/playwright-crawler.ts): browser crawler · Adds browser page/context handling and adaptive rendering. · `packages/playwright-crawler/src/internals/playwright-crawler.ts`
- [StagehandCrawler](https://github.com/apify/crawlee/blob/master/packages/stagehand-crawler/src/internals/stagehand-crawler.ts): AI browser crawler · `packages/stagehand-crawler/src/internals/stagehand-crawler.ts`
- [BrowserPool](https://github.com/apify/crawlee/blob/master/packages/browser-pool/src/browser-pool.ts): browser lifecycle · Creates and reuses browser instances. · `packages/browser-pool/src/browser-pool.ts`
- [Browser plugins](https://github.com/apify/crawlee/blob/master/packages/browser-pool/src/abstract-classes/browser-plugin.ts): integration boundary · Isolates Playwright and Puppeteer controllers. · `packages/browser-pool/src/abstract-classes/browser-plugin.ts`
- **Browser runtimes**: external runtime · Playwright/Puppeteer with Chromium and Firefox support.
- **Websites and proxies**: external network

### Delivery surfaces

Project bootstrap, deployment, and documentation.

- [CLI](https://github.com/apify/crawlee/blob/master/packages/cli/src/index.ts): project bootstrap · Creates and runs generated projects. · `packages/cli/src/index.ts`
- [CreateProjectCommand](https://github.com/apify/crawlee/blob/master/packages/cli/src/commands/CreateProjectCommand.ts): project generator · `packages/cli/src/commands/CreateProjectCommand.ts`
- [Playwright Docker template](https://github.com/apify/crawlee/blob/master/packages/templates/templates/playwright-ts/Dockerfile): deployment template · `packages/templates/templates/playwright-ts/Dockerfile`
- [Documentation site](https://github.com/apify/crawlee/blob/master/website/docusaurus.config.js): Docusaurus site · Published through Nginx deployment automation. · `website/docusaurus.config.js`

## Connections

- crawlee umbrella → Core APIs: re-exports
- crawlee umbrella → BasicCrawler: re-exports
- crawlee umbrella → HttpCrawler: re-exports
- crawlee umbrella → CheerioCrawler: re-exports
- crawlee umbrella → PlaywrightCrawler: re-exports
- BasicCrawler → Core APIs: uses shared runtime
- BasicCrawler → AutoscaledPool: coordinates concurrency
- BasicCrawler → Request queue: consumes requests
- Sitemap request list → Request queue: supplies requests
- Request queue → Request model: schedules
- BasicCrawler → Link enqueueing: handlers discover links
- Link enqueueing → Request queue: enqueues next work
- HttpCrawler → BasicCrawler: specializes
- CheerioCrawler → HttpCrawler: adds DOM parsing
- PlaywrightCrawler → BasicCrawler: specializes
- StagehandCrawler → PlaywrightCrawler: extends browser crawling
- PlaywrightCrawler → BrowserPool: delegates browser reuse
- BrowserPool → Browser plugins: uses
- Browser plugins → Browser runtimes: integrates
- Browser runtimes → Websites and proxies: browses via proxies
- Session pool → Websites and proxies: rotates identity
- Core APIs → Storage manager: provides abstraction
- Storage manager → Memory storage: defaults to
- Core APIs → Session pool: manages
- CLI → CreateProjectCommand: runs
- CreateProjectCommand → Playwright Docker template: generates project from
- Playwright Docker template → PlaywrightCrawler: deploys crawler project
