How step0ne/crawl4ai works
Project type and purpose
• Crawl4AI is an open-source web-crawling and scraping toolkit optimized for LLMs.
• It ships as:
• A Python library (crawl4ai/) with async crawlers, extraction & markdown generation strategies, browser management, caching, dispatchers, and logging.
• A CLI (crawl4ai/cli.py & entry script crwl) for one-off crawls.
• A Dockerized FastAPI service (under deploy/docker/) exposing HTTP endpoints, an interactive playground UI, and an MCP SSE interface.
File structure
• Top-level:
• crawl4ai/: core modules (async crawler, browser manager, strategies, components, utils, models, migrations).
• deploy/docker/: FastAPI server (server.py), API handlers, playground static assets, supervisory config, Dockerfile artifacts.
• tests/: unit and integration tests organized by feature (async, browser, cli, docker, general).
• Infrastructure: Dockerfile, docker-compose.yml, CI workflows, docs, mkdocs config.
• Architectural patterns:
• Strategy Pattern for crawling and extraction.
• Pipeline: fetch → parse → filter → markdown/extraction → output.
• Async concurrency with a memory-adaptive dispatcher.
• Browser pooling for pre-warmed Playwright instances.
• Modular components (crawler, dispatcher, database/cache, logger).
Extract insights from README
• Highlights: LLM-driven extraction, CSS/JSON schema scraping, deep-crawl strategies (BFS/DFS/BFF), browser profiles, caching, screenshots, network capture, Docker deployment with FastAPI and MCP integration.
• The library is agnostic to LLM provider via LLMConfig, uses Pydantic schemas for structured extraction.
a. Main components
1. Client Layer
• Python code or CLI (crwl)
• HTTP clients hitting the Dockerized API
2. API Layer (Docker service)
• FastAPI server with endpoints: /crawl, /task/{id}, /mcp/sse, /playground
• Authentication (JWT)
• Supervisord for process monitoring
3. Core Library
• AsyncDispatcher: controls concurrency & memory usage
• AsyncWebCrawler: orchestrates crawling sessions
• BrowserManager & BrowserProfiler: manage Playwright sessions & pools
• Crawling Strategies: HTTP-only, browser-based, deep-crawl strategies (BFS/DFS/BFF)
• Extraction Strategies: CSS/JSON schema, LLM extraction via external API
• MarkdownGenerationStrategy: BM25, pruning, citation generation
• Cache/Database: optional result caching layer
• Logger: async logger for tracing and debug
4. External Services
• Playwright (Chromium, Firefox, WebKit)
• LLM providers (OpenAI, local models via Litellm)
• Proxy servers
5. Data Stores & Outputs
• In-memory or on-disk cache
• Final outputs: Markdown, JSON, screenshots, PDF, MHTML
b. Relationships and data flows
1. User invokes crawl (CLI or HTTP).
2. Request enters FastAPI → validated → pushed to Dispatcher.
3. Dispatcher allocates a browser from BrowserPool.
4. Browser fetches URL(s), runs JS hooks, captures network/console.
5. Raw HTML → ExtractionStrategy → optional LLM API call → structured JSON.
6. MarkdownGenerator filters content → produces LLM-friendly Markdown.
7. Results are returned to user or stored in cache/database.
8. For MCP: results pushed over SSE to AI tools.
c. Architectural patterns & design principles
• Strategy Pattern for crawling/extraction/markdown generation
• Async pipeline to maximize throughput and real-time performance
• Resource pooling (browser instances) for low-latency responses
• Separation of concerns: core library vs. API service vs. CLI
• Extensibility hooks at every stage (pre/post fetch, extraction, filtering)
d. Key technologies & frameworks
• Python 3.9+, Asyncio, Playwright (and Selenium for sync mode)
• FastAPI for HTTP API & Uvicorn/Gunicorn
• Docker multi-stage builds, docker-compose
• Pydantic for config & schemas, BM25 for content filtering
• MCP SSE for push integration with AI tools
• Supervisord for process management