# jamesbmour/crawl4ai_webui architecture

> Architecture diagram of the GitHub repository jamesbmour/crawl4ai_webui, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/jamesbmour/crawl4ai_webui
- Repository: https://github.com/jamesbmour/crawl4ai_webui

## Overview

Crawl4AI Studio is a self-hosted visual interface for configuring web crawls, extraction, and related workflows. Users choose a mode in the React UI, submit crawl or discovery requests to FastAPI, and follow job progress and results; jobs and artifacts can be revisited later. The backend maps UI options to Crawl4AI, runs crawl jobs, and persists results, schemas, and profiles. The README documents additional output, adaptive, Q&A, and export capabilities. Some backend routing and execution details are supported only by the source index or README, so those relationships are cited at that level.

Main entry points: [`backend/app/main.py`](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/main.py) (FastAPI app), [`frontend/src/App.tsx`](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/App.tsx) (Page router), [`frontend/src/lib/api.ts`](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/lib/api.ts) (API client), [`backend/app/core/mapping.py`](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/mapping.py) (Config mapping)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["Studio interface"]
  node_app["Page router<br/>[App.tsx]"]
  node_crawl_pages["Crawl modes"]
  node_discovery_page["URL discovery<br/>[DiscoveryPage.tsx]"]
  node_extract_page["Extraction UI<br/>[ExtractionPage.tsx]"]
  node_ask_page["Ask interface<br/>[AskPage.tsx]"]
  node_job_pages["Job history<br/>[JobsPage.tsx]"]
  node_settings_page["Settings UI<br/>[SettingsPage.tsx]"]
  node_config_ui["Crawl configuration<br/>[ConfigPanel.tsx]"]
  node_schema_ui["Schema editor"]
  node_job_state["Job event state<br/>[useJob.ts]"]
  node_result_ui["Result viewer<br/>[ResultTabs.tsx]"]
  node_api_client["API client<br/>[api.ts]"]
end

subgraph group_api["HTTP API"]
  node_server["FastAPI app<br/>[main.py]"]
  node_crawl_api["Crawl endpoints<br/>[crawl.py]"]
  node_discovery_api["Seed endpoints<br/>[seed.py]"]
  node_extract_api["Extraction endpoints<br/>[extraction.py]"]
  node_ask_api["Ask endpoints<br/>[ask.py]"]
  node_jobs_api["Job endpoints<br/>[jobs_api.py]"]
  node_library_api["Schema library API<br/>[library.py]"]
  node_settings_api["Settings endpoints<br/>[settings_api.py]"]
end

subgraph group_runtime["Crawl runtime"]
  node_schemas["Request models<br/>[schemas.py]"]
  node_mapping["Config mapping<br/>[mapping.py]"]
  node_runners["Crawl runners<br/>[runners.py]"]
  node_crawler_pool["Browser pool<br/>[crawler_pool.py]"]
  node_results["Result processing<br/>[results.py]"]
  node_jobs["Job manager<br/>[jobs.py]"]
end

subgraph group_state["Persistence and settings"]
  node_storage[("SQLite and artifacts<br/>[storage.py]")]
  node_settings["App settings<br/>[settings.py]"]
end

node_browser(("Browser user"))
node_crawl4ai["Crawl4AI library"]

node_browser -->|"uses"| node_app
node_app -->|"routes to"| node_crawl_pages
node_app -->|"routes to"| node_discovery_page
node_app -->|"routes to"| node_extract_page
node_app -->|"routes to"| node_ask_page
node_app -->|"routes to"| node_job_pages
node_app -->|"routes to"| node_settings_page
node_crawl_pages -->|"configures with"| node_config_ui
node_crawl_pages -->|"tracks via"| node_job_state
node_crawl_pages -->|"displays with"| node_result_ui
node_extract_page -->|"edits with"| node_schema_ui
node_extract_page -->|"tracks via"| node_job_state
node_extract_page -->|"displays with"| node_result_ui
node_discovery_page -->|"requests through"| node_api_client
node_ask_page -->|"requests through"| node_api_client
node_job_pages -->|"requests through"| node_api_client
node_settings_page -->|"requests through"| node_api_client
node_config_ui -->|"loads schemas"| node_api_client
node_job_state -->|"starts and streams"| node_api_client
node_result_ui -->|"loads artifacts"| node_api_client
node_api_client -->|"HTTP and SSE"| node_server
node_server -->|"mounts router"| node_crawl_api
node_server -->|"mounts router"| node_discovery_api
node_server -->|"mounts router"| node_extract_api
node_server -->|"mounts router"| node_ask_api
node_server -->|"mounts router"| node_jobs_api
node_server -->|"mounts router"| node_library_api
node_server -->|"mounts router"| node_settings_api
node_crawl_api -->|"validates payloads"| node_schemas
node_crawl_api -->|"maps config"| node_mapping
node_crawl_api -->|"dispatches crawl"| node_runners
node_crawl_api -->|"manages jobs"| node_jobs
node_discovery_api -->|"maps options"| node_mapping
node_extract_api -->|"uses crawler"| node_crawler_pool
node_ask_api -->|"uses crawler"| node_crawler_pool
node_ask_api -->|"reads job data"| node_storage
node_jobs_api -->|"controls jobs"| node_jobs
node_jobs_api -->|"reads persisted jobs"| node_storage
node_library_api -->|"manages schemas"| node_storage
node_settings_api -->|"reads and writes"| node_settings
node_settings_api -->|"maps config"| node_mapping
node_mapping -->|"translates models"| node_schemas
node_runners -->|"builds configs"| node_mapping
node_runners -->|"runs crawlers"| node_crawler_pool
node_runners -->|"processes output"| node_results
node_runners -->|"persists results"| node_storage
node_jobs -->|"persists job state"| node_storage
node_mapping -->|"creates SDK configs"| node_crawl4ai

click node_app "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/App.tsx"
click node_crawl_pages "https://github.com/jamesbmour/crawl4ai_webui/tree/main/frontend/src/pages"
click node_discovery_page "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/DiscoveryPage.tsx"
click node_extract_page "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/ExtractionPage.tsx"
click node_ask_page "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/AskPage.tsx"
click node_job_pages "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/JobsPage.tsx"
click node_settings_page "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/SettingsPage.tsx"
click node_config_ui "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/ConfigPanel.tsx"
click node_schema_ui "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/ExtractionEditor.tsx"
click node_job_state "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/useJob.ts"
click node_result_ui "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/ResultTabs.tsx"
click node_api_client "https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/lib/api.ts"
click node_server "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/main.py"
click node_crawl_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/crawl.py"
click node_discovery_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/seed.py"
click node_extract_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/extraction.py"
click node_ask_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/ask.py"
click node_jobs_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/jobs_api.py"
click node_library_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/library.py"
click node_settings_api "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/settings_api.py"
click node_schemas "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/models/schemas.py"
click node_mapping "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/mapping.py"
click node_runners "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/runners.py"
click node_crawler_pool "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/crawler_pool.py"
click node_results "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/results.py"
click node_jobs "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/jobs.py"
click node_storage "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/storage.py"
click node_settings "https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/settings.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_crawl_pages,node_discovery_page,node_extract_page,node_ask_page,node_job_pages,node_settings_page,node_config_ui,node_schema_ui,node_job_state,node_result_ui,node_api_client,node_browser toneBlue
class node_server,node_crawl_api,node_discovery_api,node_extract_api,node_ask_api,node_jobs_api,node_library_api,node_settings_api toneAmber
class node_schemas,node_mapping,node_runners,node_crawler_pool,node_results,node_jobs toneMint
class node_storage,node_settings toneRose
class node_crawl4ai toneIndigo
```

## Components

### Studio interface

- [Page router](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Crawl modes](https://github.com/jamesbmour/crawl4ai_webui/tree/main/frontend/src/pages): `frontend/src/pages`
- [URL discovery](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/DiscoveryPage.tsx): `frontend/src/pages/DiscoveryPage.tsx`
- [Extraction UI](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/ExtractionPage.tsx): `frontend/src/pages/ExtractionPage.tsx`
- [Ask interface](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/AskPage.tsx): `frontend/src/pages/AskPage.tsx`
- [Job history](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/JobsPage.tsx): `frontend/src/pages/JobsPage.tsx`
- [Settings UI](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/pages/SettingsPage.tsx): `frontend/src/pages/SettingsPage.tsx`
- [Crawl configuration](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/ConfigPanel.tsx): `frontend/src/components/ConfigPanel.tsx`
- [Schema editor](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/ExtractionEditor.tsx): `frontend/src/components/ExtractionEditor.tsx`
- [Job event state](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/useJob.ts): `frontend/src/components/useJob.ts`
- [Result viewer](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/components/ResultTabs.tsx): `frontend/src/components/ResultTabs.tsx`
- [API client](https://github.com/jamesbmour/crawl4ai_webui/blob/main/frontend/src/lib/api.ts): `frontend/src/lib/api.ts`

### HTTP API

- [FastAPI app](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/main.py): `backend/app/main.py`
- [Crawl endpoints](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/crawl.py): `backend/app/api/crawl.py`
- [Seed endpoints](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/seed.py): `backend/app/api/seed.py`
- [Extraction endpoints](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/extraction.py): `backend/app/api/extraction.py`
- [Ask endpoints](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/ask.py): `backend/app/api/ask.py`
- [Job endpoints](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/jobs_api.py): `backend/app/api/jobs_api.py`
- [Schema library API](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/library.py): `backend/app/api/library.py`
- [Settings endpoints](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/api/settings_api.py): `backend/app/api/settings_api.py`

### Crawl runtime

- [Request models](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/models/schemas.py): `backend/app/models/schemas.py`
- [Config mapping](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/mapping.py): `backend/app/core/mapping.py`
- [Crawl runners](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/runners.py): `backend/app/core/runners.py`
- [Browser pool](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/crawler_pool.py): `backend/app/core/crawler_pool.py`
- [Result processing](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/results.py): `backend/app/core/results.py`
- [Job manager](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/jobs.py): `backend/app/core/jobs.py`

### Persistence and settings

- [SQLite and artifacts](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/storage.py): `backend/app/core/storage.py`
- [App settings](https://github.com/jamesbmour/crawl4ai_webui/blob/main/backend/app/core/settings.py): `backend/app/core/settings.py`

### Other components

- **Browser user**
- **Crawl4AI library**

## Connections

- Browser user → Page router: uses
- Page router → Crawl modes: routes to (evidence: `frontend/src/App.tsx`)
- Page router → URL discovery: routes to (evidence: `frontend/src/App.tsx`)
- Page router → Extraction UI: routes to (evidence: `frontend/src/App.tsx`)
- Page router → Ask interface: routes to (evidence: `frontend/src/App.tsx`)
- Page router → Job history: routes to (evidence: `frontend/src/App.tsx`)
- Page router → Settings UI: routes to (evidence: `frontend/src/App.tsx`)
- Crawl modes → Crawl configuration: configures with (evidence: `frontend/src/pages/DeepCrawlPage.tsx`)
- Crawl modes → Job event state: tracks via (evidence: `frontend/src/pages/DeepCrawlPage.tsx`)
- Crawl modes → Result viewer: displays with (evidence: `frontend/src/pages/DeepCrawlPage.tsx`)
- Extraction UI → Schema editor: edits with (evidence: `frontend/src/pages/ExtractionPage.tsx`)
- Extraction UI → Job event state: tracks via (evidence: `frontend/src/pages/ExtractionPage.tsx`)
- Extraction UI → Result viewer: displays with (evidence: `frontend/src/pages/ExtractionPage.tsx`)
- URL discovery → API client: requests through (evidence: `frontend/src/pages/DiscoveryPage.tsx`)
- Ask interface → API client: requests through (evidence: `frontend/src/pages/AskPage.tsx`)
- Job history → API client: requests through (evidence: `frontend/src/pages/JobsPage.tsx`)
- Settings UI → API client: requests through (evidence: `frontend/src/pages/SettingsPage.tsx`)
- Crawl configuration → API client: loads schemas (evidence: `frontend/src/components/ConfigPanel.tsx`)
- Job event state → API client: starts and streams (evidence: `frontend/src/components/useJob.ts`)
- Result viewer → API client: loads artifacts (evidence: `frontend/src/components/ResultTabs.tsx`)
- API client → FastAPI app: HTTP and SSE (evidence: `frontend/src/lib/api.ts`)
- FastAPI app → Crawl endpoints: mounts router (evidence: `backend/app/main.py`)
- FastAPI app → Seed endpoints: mounts router (evidence: `backend/app/main.py`)
- FastAPI app → Extraction endpoints: mounts router (evidence: `backend/app/main.py`)
- FastAPI app → Ask endpoints: mounts router (evidence: `backend/app/main.py`)
- FastAPI app → Job endpoints: mounts router (evidence: `backend/app/main.py`)
- FastAPI app → Schema library API: mounts router (evidence: `backend/app/main.py`)
- FastAPI app → Settings endpoints: mounts router (evidence: `backend/app/main.py`)
- Crawl endpoints → Request models: validates payloads (evidence: `backend/app/api/crawl.py`)
- Crawl endpoints → Config mapping: maps config (evidence: `backend/app/api/crawl.py`)
- Crawl endpoints → Crawl runners: dispatches crawl (evidence: `backend/app/api/crawl.py`)
- Crawl endpoints → Job manager: manages jobs (evidence: `backend/app/api/crawl.py`)
- Seed endpoints → Config mapping: maps options (evidence: `backend/app/api/seed.py`)
- Extraction endpoints → Browser pool: uses crawler (evidence: `backend/app/api/extraction.py`)
- Ask endpoints → Browser pool: uses crawler (evidence: `backend/app/api/ask.py`)
- Ask endpoints → SQLite and artifacts: reads job data (evidence: `backend/app/api/ask.py`)
- Job endpoints → Job manager: controls jobs (evidence: `backend/app/api/jobs_api.py`)
- Job endpoints → SQLite and artifacts: reads persisted jobs (evidence: `backend/app/api/jobs_api.py`)
- Schema library API → SQLite and artifacts: manages schemas (evidence: `backend/app/api/library.py`)
- Settings endpoints → App settings: reads and writes (evidence: `backend/app/api/settings_api.py`)
- Settings endpoints → Config mapping: maps config (evidence: `backend/app/api/settings_api.py`)
- Config mapping → Request models: translates models (evidence: `backend/app/core/mapping.py`)
- Crawl runners → Config mapping: builds configs (evidence: `backend/app/core/runners.py`)
- Crawl runners → Browser pool: runs crawlers (evidence: `backend/app/core/runners.py`)
- Crawl runners → Result processing: processes output (evidence: `backend/app/core/runners.py`)
- Crawl runners → SQLite and artifacts: persists results (evidence: `backend/app/core/runners.py`)
- Job manager → SQLite and artifacts: persists job state (evidence: `backend/app/core/jobs.py`)
- Config mapping → Crawl4AI library: creates SDK configs (evidence: `backend/app/core/mapping.py`)
