# unclecode/crawl4ai architecture

> Architecture diagram of the GitHub repository unclecode/crawl4ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/unclecode/crawl4ai
- Repository: https://github.com/unclecode/crawl4ai
- Video tour (about a minute): https://gitdiagram.com/unclecode/crawl4ai/video

## Overview

Crawl4AI is a web-crawling and extraction system, available as a Python library and self-hosted Docker API; the README also describes hosted Cloud endpoints. Its principal flow accepts a URL, loads it in a browser, processes page content into Markdown or structured data, and returns results. The graph preserves documented search, deep crawling, link previews, PDF, browser-control, and server capabilities. Source excerpts are partial, so unsampled runtime wiring is omitted or shown only where documented.

Main entry points: [`crawl4ai/async_webcrawler.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_webcrawler.py) (Async crawler), [`deploy/docker/server.py`](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/server.py) (Docker API), [`crawl4ai/browser_manager.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py) (Browser manager), [`crawl4ai/markdown_generation_strategy.py`](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/markdown_generation_strategy.py) (Markdown generation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Entry points"]
  node_async_crawler["Async crawler"]
  node_configs["Crawl configuration<br/>[async_configs.py]"]
  node_docker_client["Docker client<br/>[docker_client.py]"]
  node_search["Web search<br/>[crawler.py]"]
end

subgraph group_crawl["Crawling and browsers"]
  node_browser_manager["Browser manager<br/>[browser_manager.py]"]
  node_browser_adapter["Browser adapters<br/>[browser_adapter.py]"]
  node_browser_profiler["Browser profiler"]
  node_deep_crawl["Deep crawling"]
  node_url_seeder["URL seeding"]
  node_link_preview["Link previews<br/>[link_preview.py]"]
end

subgraph group_process["Content processing"]
  node_scraping["Content scraping"]
  node_markdown["Markdown generation"]
  node_html2text["HTML to Markdown<br/>[__init__.py]"]
  node_filters["Content filters"]
  node_extraction["Structured extraction"]
  node_chunking["Text chunking"]
  node_tables["Table extraction"]
  node_pdf["PDF processing<br/>[processor.py]"]
  node_results["Crawl results<br/>[models.py]"]
end

subgraph group_cloud["Server and integrations"]
  node_docker_server["Docker API<br/>[server.py]"]
  node_pool["Browser pool<br/>[crawler_pool.py]"]
  node_egress["Egress broker<br/>[egress_broker.py]"]
  node_llm_broker["LLM broker<br/>[llm_broker.py]"]
  node_monitor["Server monitoring<br/>[monitor.py]"]
  node_monitor_routes["Monitor API<br/>[monitor_routes.py]"]
end

node_user(("Caller"))
node_cloud_service["Crawl4AI Cloud"]
node_web(("Target websites"))
node_llm_provider["LLM provider"]

node_user -->|"calls"| node_async_crawler
node_async_crawler -->|"uses settings"| node_configs
node_async_crawler -.->|"crawls pages"| node_browser_manager
node_browser_manager -->|"uses adapter"| node_browser_adapter
node_browser_manager -->|"profiles browser"| node_browser_profiler
node_async_crawler -.->|"supports exploration"| node_deep_crawl
node_async_crawler -.->|"supports seeding"| node_url_seeder
node_link_preview -->|"extracts link heads"| node_url_seeder
node_async_crawler -.->|"processes page"| node_scraping
node_scraping -.->|"generates output"| node_markdown
node_markdown -->|"converts HTML"| node_html2text
node_markdown -.->|"supports filtering"| node_filters
node_async_crawler -.->|"supports extraction"| node_extraction
node_async_crawler -.->|"supports chunking"| node_chunking
node_async_crawler -.->|"supports tables"| node_tables
node_async_crawler -.->|"supports PDFs"| node_pdf
node_async_crawler -.->|"loads pages"| node_web
node_async_crawler -.->|"returns results"| node_results
node_user -->|"calls"| node_docker_client
node_docker_client -.->|"sends requests"| node_docker_server
node_user -->|"requests API"| node_docker_server
node_docker_server -->|"uses pool"| node_pool
node_pool -->|"reports status"| node_monitor
node_monitor_routes -->|"reads metrics"| node_monitor
node_docker_server -->|"uses broker"| node_egress
node_docker_server -.->|"uses broker"| node_llm_broker
node_user -.->|"requests search"| node_search
node_cloud_service -.->|"searches live web"| node_web
node_extraction -.->|"uses provider"| node_llm_provider
node_cloud_service -->|"serves API"| node_user

click node_async_crawler "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_webcrawler.py"
click node_configs "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_configs.py"
click node_docker_server "https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/server.py"
click node_docker_client "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/docker_client.py"
click node_pool "https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/crawler_pool.py"
click node_browser_manager "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py"
click node_browser_adapter "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_adapter.py"
click node_browser_profiler "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_profiler.py"
click node_deep_crawl "https://github.com/unclecode/crawl4ai/tree/main/crawl4ai/deep_crawling"
click node_url_seeder "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_url_seeder.py"
click node_link_preview "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/link_preview.py"
click node_scraping "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/content_scraping_strategy.py"
click node_markdown "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/markdown_generation_strategy.py"
click node_html2text "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/html2text/__init__.py"
click node_filters "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/content_filter_strategy.py"
click node_extraction "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py"
click node_chunking "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/chunking_strategy.py"
click node_tables "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/table_extraction.py"
click node_pdf "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/processors/pdf/processor.py"
click node_results "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/models.py"
click node_search "https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/crawlers/google_search/crawler.py"
click node_egress "https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/egress_broker.py"
click node_llm_broker "https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/llm_broker.py"
click node_monitor "https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/monitor.py"
click node_monitor_routes "https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/monitor_routes.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_async_crawler,node_configs,node_docker_client,node_search toneBlue
class node_browser_manager,node_browser_adapter,node_browser_profiler,node_deep_crawl,node_url_seeder,node_link_preview toneAmber
class node_scraping,node_markdown,node_html2text,node_filters,node_extraction,node_chunking,node_tables,node_pdf,node_results toneMint
class node_docker_server,node_pool,node_egress,node_llm_broker,node_monitor,node_monitor_routes toneRose
class node_user,node_cloud_service,node_web,node_llm_provider toneIndigo
```

## Components

### Entry points

- [Async crawler](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_webcrawler.py): `crawl4ai/async_webcrawler.py`
- [Crawl configuration](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_configs.py): `crawl4ai/async_configs.py`
- [Docker client](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/docker_client.py): `crawl4ai/docker_client.py`
- [Web search](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/crawlers/google_search/crawler.py): `crawl4ai/crawlers/google_search/crawler.py`

### Crawling and browsers

- [Browser manager](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_manager.py): `crawl4ai/browser_manager.py`
- [Browser adapters](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_adapter.py): `crawl4ai/browser_adapter.py`
- [Browser profiler](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/browser_profiler.py): `crawl4ai/browser_profiler.py`
- [Deep crawling](https://github.com/unclecode/crawl4ai/tree/main/crawl4ai/deep_crawling): `crawl4ai/deep_crawling`
- [URL seeding](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/async_url_seeder.py): `crawl4ai/async_url_seeder.py`
- [Link previews](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/link_preview.py): `crawl4ai/link_preview.py`

### Content processing

- [Content scraping](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/content_scraping_strategy.py): `crawl4ai/content_scraping_strategy.py`
- [Markdown generation](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/markdown_generation_strategy.py): `crawl4ai/markdown_generation_strategy.py`
- [HTML to Markdown](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/html2text/__init__.py): `crawl4ai/html2text/__init__.py`
- [Content filters](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/content_filter_strategy.py): `crawl4ai/content_filter_strategy.py`
- [Structured extraction](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/extraction_strategy.py): `crawl4ai/extraction_strategy.py`
- [Text chunking](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/chunking_strategy.py): `crawl4ai/chunking_strategy.py`
- [Table extraction](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/table_extraction.py): `crawl4ai/table_extraction.py`
- [PDF processing](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/processors/pdf/processor.py): `crawl4ai/processors/pdf/processor.py`
- [Crawl results](https://github.com/unclecode/crawl4ai/blob/main/crawl4ai/models.py): `crawl4ai/models.py`

### Server and integrations

- [Docker API](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/server.py): `deploy/docker/server.py`
- [Browser pool](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/crawler_pool.py): `deploy/docker/crawler_pool.py`
- [Egress broker](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/egress_broker.py): `deploy/docker/egress_broker.py`
- [LLM broker](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/llm_broker.py): `deploy/docker/llm_broker.py`
- [Server monitoring](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/monitor.py): `deploy/docker/monitor.py`
- [Monitor API](https://github.com/unclecode/crawl4ai/blob/main/deploy/docker/monitor_routes.py): `deploy/docker/monitor_routes.py`

### Other components

- **Caller**
- **Crawl4AI Cloud**
- **Target websites**
- **LLM provider**

## Connections

- Caller → Async crawler: calls
- Async crawler → Crawl configuration: uses settings (evidence: `crawl4ai/async_configs.py`)
- Async crawler → Browser manager: crawls pages
- Browser manager → Browser adapters: uses adapter (evidence: `crawl4ai/browser_manager.py`)
- Browser manager → Browser profiler: profiles browser (evidence: `crawl4ai/browser_manager.py`)
- Async crawler → Deep crawling: supports exploration
- Async crawler → URL seeding: supports seeding
- Link previews → URL seeding: extracts link heads (evidence: `crawl4ai/link_preview.py`)
- Async crawler → Content scraping: processes page
- Content scraping → Markdown generation: generates output
- Markdown generation → HTML to Markdown: converts HTML (evidence: `crawl4ai/html2text/__init__.py`)
- Markdown generation → Content filters: supports filtering
- Async crawler → Structured extraction: supports extraction
- Async crawler → Text chunking: supports chunking
- Async crawler → Table extraction: supports tables (evidence: `crawl4ai/__init__.py`)
- Async crawler → PDF processing: supports PDFs
- Async crawler → Target websites: loads pages
- Async crawler → Crawl results: returns results
- Caller → Docker client: calls (evidence: `crawl4ai/docker_client.py`)
- Docker client → Docker API: sends requests
- Caller → Docker API: requests API
- Docker API → Browser pool: uses pool (evidence: `deploy/docker/server.py`)
- Browser pool → Server monitoring: reports status (evidence: `deploy/docker/crawler_pool.py`)
- Monitor API → Server monitoring: reads metrics (evidence: `deploy/docker/monitor_routes.py`)
- Docker API → Egress broker: uses broker (evidence: `deploy/docker/server.py`)
- Docker API → LLM broker: uses broker
- Caller → Web search: requests search
- Crawl4AI Cloud → Target websites: searches live web
- Structured extraction → LLM provider: uses provider
- Crawl4AI Cloud → Caller: serves API
