How jamesbmour/crawl4ai_webui works
Crawl4AI Studio is a self-hosted visual interface for configuring web crawls, extraction, and related workflows. Users choose a mode in the React UI, submit crawl or discovery requests to FastAPI, and follow job progress and results; jobs and artifacts can be revisited later. The backend maps UI options to Crawl4AI, runs crawl jobs, and persists results, schemas, and profiles. The README documents additional output, adaptive, Q&A, and export capabilities. Some backend routing and execution details are supported only by the source index or README, so those relationships are cited at that level.
Main entry points: main.py FastAPI app · App.tsx Page router · api.ts API client · mapping.py Config mapping
Components 30
Studio interface
- Page router
frontend/src/App.tsx - Crawl modes
frontend/src/pages - URL discovery
frontend/src/pages/DiscoveryPage.tsx - Extraction UI
frontend/src/pages/ExtractionPage.tsx - Ask interface
frontend/src/pages/AskPage.tsx - Job history
frontend/src/pages/JobsPage.tsx - Settings UI
frontend/src/pages/SettingsPage.tsx - Crawl configuration
frontend/src/components/ConfigPanel.tsx - Schema editor
frontend/src/components/ExtractionEditor.tsx - Job event state
frontend/src/components/useJob.ts - Result viewer
frontend/src/components/ResultTabs.tsx - API client
frontend/src/lib/api.ts
HTTP API
- FastAPI app
backend/app/main.py - Crawl endpoints
backend/app/api/crawl.py - Seed endpoints
backend/app/api/seed.py - Extraction endpoints
backend/app/api/extraction.py - Ask endpoints
backend/app/api/ask.py - Job endpoints
backend/app/api/jobs_api.py - Schema library API
backend/app/api/library.py - Settings endpoints
backend/app/api/settings_api.py
Crawl runtime
- Request models
backend/app/models/schemas.py - Config mapping
backend/app/core/mapping.py - Crawl runners
backend/app/core/runners.py - Browser pool
backend/app/core/crawler_pool.py - Result processing
backend/app/core/results.py - Job manager
backend/app/core/jobs.py
Persistence and settings
- SQLite and artifacts
backend/app/core/storage.py - App settings
backend/app/core/settings.py
Other components
- Browser user
- Crawl4AI library
Connections 48
- Browser user → Page router: uses
- Page router → Crawl modes: routes to
frontend/src/App.tsx - Page router → URL discovery: routes to
frontend/src/App.tsx - Page router → Extraction UI: routes to
frontend/src/App.tsx - Page router → Ask interface: routes to
frontend/src/App.tsx - Page router → Job history: routes to
frontend/src/App.tsx - Page router → Settings UI: routes to
frontend/src/App.tsx - Crawl modes → Crawl configuration: configures with
frontend/src/pages/DeepCrawlPage.tsx - Crawl modes → Job event state: tracks via
frontend/src/pages/DeepCrawlPage.tsx - Crawl modes → Result viewer: displays with
frontend/src/pages/DeepCrawlPage.tsx - Extraction UI → Schema editor: edits with
frontend/src/pages/ExtractionPage.tsx - Extraction UI → Job event state: tracks via
frontend/src/pages/ExtractionPage.tsx - Extraction UI → Result viewer: displays with
frontend/src/pages/ExtractionPage.tsx - URL discovery → API client: requests through
frontend/src/pages/DiscoveryPage.tsx - Ask interface → API client: requests through
frontend/src/pages/AskPage.tsx - Job history → API client: requests through
frontend/src/pages/JobsPage.tsx - Settings UI → API client: requests through
frontend/src/pages/SettingsPage.tsx - Crawl configuration → API client: loads schemas
frontend/src/components/ConfigPanel.tsx - Job event state → API client: starts and streams
frontend/src/components/useJob.ts - Result viewer → API client: loads artifacts
frontend/src/components/ResultTabs.tsx - API client → FastAPI app: HTTP and SSE
frontend/src/lib/api.ts - FastAPI app → Crawl endpoints: mounts router
backend/app/main.py - FastAPI app → Seed endpoints: mounts router
backend/app/main.py - FastAPI app → Extraction endpoints: mounts router
backend/app/main.py - FastAPI app → Ask endpoints: mounts router
backend/app/main.py - FastAPI app → Job endpoints: mounts router
backend/app/main.py - FastAPI app → Schema library API: mounts router
backend/app/main.py - FastAPI app → Settings endpoints: mounts router
backend/app/main.py - Crawl endpoints → Request models: validates payloads
backend/app/api/crawl.py - Crawl endpoints → Config mapping: maps config
backend/app/api/crawl.py - Crawl endpoints → Crawl runners: dispatches crawl
backend/app/api/crawl.py - Crawl endpoints → Job manager: manages jobs
backend/app/api/crawl.py - Seed endpoints → Config mapping: maps options
backend/app/api/seed.py - Extraction endpoints → Browser pool: uses crawler
backend/app/api/extraction.py - Ask endpoints → Browser pool: uses crawler
backend/app/api/ask.py - Ask endpoints → SQLite and artifacts: reads job data
backend/app/api/ask.py - Job endpoints → Job manager: controls jobs
backend/app/api/jobs_api.py - Job endpoints → SQLite and artifacts: reads persisted jobs
backend/app/api/jobs_api.py - Schema library API → SQLite and artifacts: manages schemas
backend/app/api/library.py - Settings endpoints → App settings: reads and writes
backend/app/api/settings_api.py - Settings endpoints → Config mapping: maps config
backend/app/api/settings_api.py - Config mapping → Request models: translates models
backend/app/core/mapping.py - Crawl runners → Config mapping: builds configs
backend/app/core/runners.py - Crawl runners → Browser pool: runs crawlers
backend/app/core/runners.py - Crawl runners → Result processing: processes output
backend/app/core/runners.py - Crawl runners → SQLite and artifacts: persists results
backend/app/core/runners.py - Job manager → SQLite and artifacts: persists job state
backend/app/core/jobs.py - Config mapping → Crawl4AI library: creates SDK configs
backend/app/core/mapping.py