How unclecode/crawl4ai works
Crawl4AI is a web-crawling and extraction system, available as a Python library and self-hosted Docker API; the README also describes hosted Cloud endpoints. Its principal flow accepts a URL, loads it in a browser, processes page content into Markdown or structured data, and returns results. The graph preserves documented search, deep crawling, link previews, PDF, browser-control, and server capabilities. Source excerpts are partial, so unsampled runtime wiring is omitted or shown only where documented.
Main entry points: async_webcrawler.py Async crawler · server.py Docker API · browser_manager.py Browser manager · markdown_generation_strategy.py Markdown generation
Components 29
Entry points
- Async crawler
crawl4ai/async_webcrawler.py - Crawl configuration
crawl4ai/async_configs.py - Docker client
crawl4ai/docker_client.py - Web search
crawl4ai/crawlers/google_search/crawler.py
Crawling and browsers
- Browser manager
crawl4ai/browser_manager.py - Browser adapters
crawl4ai/browser_adapter.py - Browser profiler
crawl4ai/browser_profiler.py - Deep crawling
crawl4ai/deep_crawling - URL seeding
crawl4ai/async_url_seeder.py - Link previews
crawl4ai/link_preview.py
Content processing
- Content scraping
crawl4ai/content_scraping_strategy.py - Markdown generation
crawl4ai/markdown_generation_strategy.py - HTML to Markdown
crawl4ai/html2text/__init__.py - Content filters
crawl4ai/content_filter_strategy.py - Structured extraction
crawl4ai/extraction_strategy.py - Text chunking
crawl4ai/chunking_strategy.py - Table extraction
crawl4ai/table_extraction.py - PDF processing
crawl4ai/processors/pdf/processor.py - Crawl results
crawl4ai/models.py
Server and integrations
- Docker API
deploy/docker/server.py - Browser pool
deploy/docker/crawler_pool.py - Egress broker
deploy/docker/egress_broker.py - LLM broker
deploy/docker/llm_broker.py - Server monitoring
deploy/docker/monitor.py - Monitor API
deploy/docker/monitor_routes.py
Other components
- Caller
- Crawl4AI Cloud
- Target websites
- LLM provider
Connections 30
- Caller → Async crawler: calls
- Async crawler → Crawl configuration: uses settings
crawl4ai/async_configs.py - Async crawler → Browser manager: crawls pages
- Browser manager → Browser adapters: uses adapter
crawl4ai/browser_manager.py - Browser manager → Browser profiler: profiles browser
crawl4ai/browser_manager.py - Async crawler → Deep crawling: supports exploration
- Async crawler → URL seeding: supports seeding
- Link previews → URL seeding: extracts link heads
crawl4ai/link_preview.py - Async crawler → Content scraping: processes page
- Content scraping → Markdown generation: generates output
- Markdown generation → HTML to Markdown: converts HTML
crawl4ai/html2text/__init__.py - Markdown generation → Content filters: supports filtering
- Async crawler → Structured extraction: supports extraction
- Async crawler → Text chunking: supports chunking
- Async crawler → Table extraction: supports tables
crawl4ai/__init__.py - Async crawler → PDF processing: supports PDFs
- Async crawler → Target websites: loads pages
- Async crawler → Crawl results: returns results
- Caller → Docker client: calls
crawl4ai/docker_client.py - Docker client → Docker API: sends requests
- Caller → Docker API: requests API
- Docker API → Browser pool: uses pool
deploy/docker/server.py - Browser pool → Server monitoring: reports status
deploy/docker/crawler_pool.py - Monitor API → Server monitoring: reads metrics
deploy/docker/monitor_routes.py - Docker API → Egress broker: uses broker
deploy/docker/server.py - Docker API → LLM broker: uses broker
- Caller → Web search: requests search
- Crawl4AI Cloud → Target websites: searches live web
- Structured extraction → LLM provider: uses provider
- Crawl4AI Cloud → Caller: serves API