# step0ne/crawl4ai architecture

> Architecture diagram of the GitHub repository step0ne/crawl4ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/step0ne/crawl4ai
- Repository: https://github.com/step0ne/crawl4ai

## Overview

##### Project type and purpose
- Crawl4AI is an open-source web-crawling and scraping toolkit optimized for LLMs.
- It ships as:
- A Python library (`crawl4ai/`) with async crawlers, extraction & markdown generation strategies, browser management, caching, dispatchers, and logging.
- A CLI (`crawl4ai/cli.py` & entry script `crwl`) for one-off crawls.
- A Dockerized FastAPI service (under `deploy/docker/`) exposing HTTP endpoints, an interactive playground UI, and an MCP SSE interface.

##### File structure
- Top-level:
- `crawl4ai/`: core modules (async crawler, browser manager, strategies, components, utils, models, migrations).
- `deploy/docker/`: FastAPI server (`server.py`), API handlers, playground static assets, supervisory config, Dockerfile artifacts.
- `tests/`: unit and integration tests organized by feature (async, browser, cli, docker, general).
- Infrastructure: `Dockerfile`, `docker-compose.yml`, CI workflows, docs, mkdocs config.
- Architectural patterns:
- Strategy Pattern for crawling and extraction.
- Pipeline: fetch → parse → filter → markdown/extraction → output.
- Async concurrency with a memory-adaptive dispatcher.
- Browser pooling for pre-warmed Playwright instances.
- Modular components (crawler, dispatcher, database/cache, logger).

##### Extract insights from README
- Highlights: LLM-driven extraction, CSS/JSON schema scraping, deep-crawl strategies (BFS/DFS/BFF), browser profiles, caching, screenshots, network capture, Docker deployment with FastAPI and MCP integration.
- The library is agnostic to LLM provider via `LLMConfig`, uses Pydantic schemas for structured extraction.

a. Main components
1. Client Layer
- Python code or CLI (`crwl`)
- HTTP clients hitting the Dockerized API
2. API Layer (Docker service)
- FastAPI server with endpoints: `/crawl`, `/task/{id}`, `/mcp/sse`, `/playground`
- Authentication (JWT)
- Supervisord for process monitoring
3. Core Library
- AsyncDispatcher: controls concurrency & memory usage
- AsyncWebCrawler: orchestrates crawling sessions
- BrowserManager & BrowserProfiler: manage Playwright sessions & pools
- Crawling Strategies: HTTP-only, browser-based, deep-crawl strategies (BFS/DFS/BFF)
- Extraction Strategies: CSS/JSON schema, LLM extraction via external API
- MarkdownGenerationStrategy: BM25, pruning, citation generation
- Cache/Database: optional result caching layer
- Logger: async logger for tracing and debug
4. External Services
- Playwright (Chromium, Firefox, WebKit)
- LLM providers (OpenAI, local models via Litellm)
- Proxy servers
5. Data Stores & Outputs
- In-memory or on-disk cache
- Final outputs: Markdown, JSON, screenshots, PDF, MHTML

b. Relationships and data flows
1. User invokes crawl (CLI or HTTP).
2. Request enters FastAPI → validated → pushed to Dispatcher.
3. Dispatcher allocates a browser from BrowserPool.
4. Browser fetches URL(s), runs JS hooks, captures network/console.
5. Raw HTML → ExtractionStrategy → optional LLM API call → structured JSON.
6. MarkdownGenerator filters content → produces LLM-friendly Markdown.
7. Results are returned to user or stored in cache/database.
8. For MCP: results pushed over SSE to AI tools.

c. Architectural patterns & design principles
- Strategy Pattern for crawling/extraction/markdown generation
- Async pipeline to maximize throughput and real-time performance
- Resource pooling (browser instances) for low-latency responses
- Separation of concerns: core library vs. API service vs. CLI
- Extensibility hooks at every stage (pre/post fetch, extraction, filtering)

d. Key technologies & frameworks
- Python 3.9+, Asyncio, Playwright (and Selenium for sync mode)
- FastAPI for HTTP API & Uvicorn/Gunicorn
- Docker multi-stage builds, docker-compose
- Pydantic for config & schemas, BM25 for content filtering
- MCP SSE for push integration with AI tools
- Supervisord for process management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client Layer
    subgraph "Client Layer"
        direction TB
        CLI["CLI (crwl)"]:::client
        HTTPClient["HTTP Client"]:::client
    end

    %% API Service Layer
    subgraph "API Service (Docker)" 
        direction TB
        FastAPIServer["FastAPI Server"]:::api
        subgraph "API Handlers"
            direction TB
            Routes["Main Routes (/crawl, /task/{id})"]:::api
            Auth["Authentication (JWT)"]:::api
            MCPBridge["MCP SSE Bridge"]:::api
        end
        BrowserPool["Browser Pool Manager"]:::api
        Supervisord["Supervisord"]:::api
    end

    %% Core Library Layer
    subgraph "Core Library" 
        direction TB
        AsyncDispatcher["AsyncDispatcher"]:::library
        AsyncWebCrawler["AsyncWebCrawler"]:::library
        BrowserManager["BrowserManager"]:::library
        BrowserProfiler["BrowserProfiler"]:::library

        subgraph "Crawling Strategies"
            direction TB
            HTTPOnly["HTTP-only Strategy"]:::library
            BFS["BFS Strategy"]:::library
            DFS["DFS Strategy"]:::library
            BFF["BFF Strategy"]:::library
        end

        ExtractionStrategy["ExtractionStrategy"]:::library
        MarkdownGen["MarkdownGenerationStrategy"]:::library

        subgraph "Cache/Database"
            direction TB
            AsyncDB["Async Database Interface"]:::datastore
            CacheCtx["Cache Context Manager"]:::datastore
        end

        AsyncLogger["Async Logger"]:::library
        LLMConfig["LLM Config & Schemas"]:::library
        ModelLoader["Model Loader & Types"]:::library
        ProxyStrategy["Proxy Strategy"]:::library
        DispatcherMonitor["Dispatcher Monitor"]:::library
    end

    %% External Services & Data Stores
    subgraph "External Systems"
        direction TB
        Playwright["Playwright (Chromium, Firefox, WebKit)"]:::external
        LLMProviders["LLM Providers"]:::external
        ProxyServer["Proxy Servers"]:::external
    end

    subgraph "Data Stores"
        direction TB
        CacheDB["In-memory/On-disk Cache"]:::datastore
    end

    %% Connections: Client to Service/Core
    CLI -->|"invokes crawl"| AsyncWebCrawler
    CLI -->|"invokes crawl via API"| FastAPIServer
    HTTPClient -->|"HTTP request"| FastAPIServer

    %% API to Core
    FastAPIServer -->|"validate & enqueue"| AsyncDispatcher
    Routes --> FastAPIServer
    Auth --> FastAPIServer
    MCPBridge --> FastAPIServer
    BrowserPool --> AsyncDispatcher

    %% Core Workflow
    AsyncWebCrawler --> AsyncDispatcher
    AsyncDispatcher --> BrowserManager
    BrowserManager --> BrowserProfiler
    BrowserProfiler --> Playwright
    BrowserManager --> HTTPOnly
    BrowserManager --> BFS
    BrowserManager --> DFS
    BrowserManager --> BFF
    HTTPOnly --> ExtractionStrategy
    BFS --> ExtractionStrategy
    DFS --> ExtractionStrategy
    BFF --> ExtractionStrategy
    ExtractionStrategy --> LLMProviders
    ExtractionStrategy --> ProxyServer
    ExtractionStrategy --> CacheDB
    ExtractionStrategy --> AsyncDB
    ExtractionStrategy --> MarkdownGen
    MarkdownGen --> CacheDB
    MarkdownGen --> AsyncDB
    MarkdownGen --> AsyncLogger
    DispatcherMonitor --> AsyncDispatcher

    %% API Response
    AsyncDispatcher -->|"results"| FastAPIServer
    FastAPIServer -->|"response"| HTTPClient

    %% SSE Push
    AsyncDispatcher -.->|"SSE push"| MCPBridge

    %% Data Stores connections
    AsyncDB --> CacheDB

    %% Click Events
    click AsyncDispatcher "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/async_dispatcher.py"
    click AsyncWebCrawler "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/async_webcrawler.py"
    click BrowserManager "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/browser_manager.py"
    click BrowserProfiler "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/browser_profiler.py"
    click HTTPOnly "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/content_scraping_strategy.py"
    click BFS "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/deep_crawling/bfs_strategy.py"
    click DFS "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/deep_crawling/dfs_strategy.py"
    click BFF "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/deep_crawling/bff_strategy.py"
    click ExtractionStrategy "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/extraction_strategy.py"
    click MarkdownGen "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/markdown_generation_strategy.py"
    click AsyncDB "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/async_database.py"
    click CacheCtx "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/cache_context.py"
    click AsyncLogger "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/async_logger.py"
    click CLI "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/cli.py"
    click FastAPIServer "https://github.com/step0ne/crawl4ai/blob/main/deploy/docker/server.py"
    click Routes "https://github.com/step0ne/crawl4ai/blob/main/deploy/docker/api.py"
    click Auth "https://github.com/step0ne/crawl4ai/blob/main/deploy/docker/auth.py"
    click MCPBridge "https://github.com/step0ne/crawl4ai/blob/main/deploy/docker/mcp_bridge.py"
    click BrowserPool "https://github.com/step0ne/crawl4ai/blob/main/deploy/docker/crawler_pool.py"
    click Supervisord "https://github.com/step0ne/crawl4ai/blob/main/deploy/docker/supervisord.conf"
    click LLMConfig "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/config.py"
    click ModelLoader "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/model_loader.py"
    click DispatcherMonitor "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/components/crawler_monitor.py"
    click ProxyStrategy "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/proxy_strategy.py"
    click CacheDB "https://github.com/step0ne/crawl4ai/blob/main/crawl4ai/cache_context.py"

    %% Styles
    classDef library fill:#D0E8FF,stroke:#0077CC,color:#000
    classDef api fill:#DFFFE0,stroke:#33AA00,color:#000
    classDef client fill:#E0FFE0,stroke:#33AA00,color:#000
    classDef external fill:#F0F0F0,stroke:#888,color:#000
    classDef datastore fill:#FFE8C0,stroke:#CC7700,color:#000
```
