# starlife-creator/omnicrawler architecture

> Architecture diagram of the GitHub repository starlife-creator/omnicrawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/starlife-creator/omnicrawler
- Repository: https://github.com/starlife-creator/omnicrawler

## Overview

OmniCrawler is a desktop-oriented data collection platform with CLI and GUI entry points. Its primary workflow turns a user’s URL or task description into a configured, reviewable crawl; plans and runs collection; parses and extracts structured data; checks quality; and delivers results. The README documents the pipeline stages and major feature areas, while sampled source confirms selected GUI, CLI, storage, and plugin behavior. Some subsystem links below reflect documented architecture rather than inspected call sites.

Main entry points: [`src/omnicrawler/pipeline/core.py`](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline/core.py) (Pipeline Orchestrator), [`src/omnicrawler/gui/main.py`](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/gui/main.py) (Desktop GUI), [`src/omnicrawler/cli/_handlers.py`](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/cli/_handlers.py) (CLI Commands), [`src/omnicrawler/fetching/async_fetcher.py`](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/fetching/async_fetcher.py) (Web Fetching)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interaction["User Interfaces"]
  node_gui["Desktop GUI<br/>[main.py]"]
  node_cli["CLI Commands<br/>[_handlers.py]"]
end

subgraph group_workflow["Collection Workflow"]
  node_task_design["Task Design"]
  node_templates["Templates"]
  node_plan["Plan and Preflight<br/>[plan_compiler.py]"]
  node_pipeline["Pipeline Orchestrator<br/>[core.py]"]
  node_fetch["Web Fetching<br/>[async_fetcher.py]"]
  node_browser["Browser Sessions<br/>[browser_fetcher.py]"]
  node_sources["Source Adapters<br/>[sources.py]"]
end

subgraph group_data["Documents and Data"]
  node_extract["Structured Extraction<br/>[extractors.py]"]
  node_documents["Document Parsing<br/>[parsers.py]"]
  node_pdf["PDF Processing<br/>[service.py]"]
  node_quality["Quality Checks<br/>[quality.py]"]
  node_review["Human Review"]
  node_delivery["Export and Sinks<br/>[exporters.py]"]
  node_convert["Format Conversion<br/>[_core.py]"]
end

subgraph group_platform["Platform Services"]
  node_state[("Run State<br/>[state_store.py]")]
  node_runtime["Scheduling and Recovery<br/>[scheduler.py]"]
  node_security["Egress and Policy<br/>[policy.py]"]
  node_monitor["Change Monitoring<br/>[change_detector.py]"]
end

subgraph group_extensions["Ecosystem"]
  node_plugins["Plugin Runtime<br/>[plugin_runtime.py]"]
  node_market["Plugin Market<br/>[market_client.py]"]
end

node_user(("Operator"))
node_web(("Websites and APIs"))
node_ai(("AI Provider"))
node_object_store[("Object Storage")]
node_market_service(("Market Catalog"))

node_user -->|"uses"| node_gui
node_user -->|"runs"| node_cli
node_gui -->|"drafts tasks"| node_task_design
node_cli -->|"compiles request"| node_task_design
node_gui -->|"applies template"| node_templates
node_cli -->|"manages templates"| node_templates
node_task_design -->|"creates draft"| node_plan
node_templates -->|"configures task"| node_plan
node_gui -->|"starts run"| node_pipeline
node_cli -->|"dispatches run"| node_pipeline
node_pipeline -->|"plans"| node_plan
node_pipeline -->|"checks policy"| node_security
node_pipeline -->|"fetches pages"| node_fetch
node_pipeline -->|"extracts fields"| node_extract
node_pipeline -->|"parses documents"| node_documents
node_pipeline -->|"checks results"| node_quality
node_pipeline -->|"exports results"| node_delivery
node_fetch -->|"requests content"| node_web
node_fetch -->|"uses sessions"| node_browser
node_fetch -->|"uses adapters"| node_sources
node_browser -->|"loads pages"| node_web
node_documents -->|"delegates PDF"| node_pdf
node_pipeline -->|"records run"| node_state
node_runtime -->|"dispatches work"| node_pipeline
node_runtime -->|"tracks recovery"| node_state
node_delivery -.->|"stores objects"| node_object_store
node_gui -->|"opens review"| node_review
node_review -->|"reviews evidence"| node_quality
node_gui -->|"opens converter"| node_convert
node_cli -->|"converts files"| node_convert
node_monitor -->|"checks changes"| node_web
node_extract -.->|"requests extraction"| node_ai
node_pipeline -.->|"loads extensions"| node_plugins
node_gui -.->|"browses catalog"| node_market
node_market -.->|"fetches catalog"| node_market_service

click node_gui "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/gui/main.py"
click node_cli "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/cli/_handlers.py"
click node_task_design "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/services/natural_language_task.py"
click node_templates "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/templates/template_application.py"
click node_plan "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline_ops/plan_compiler.py"
click node_pipeline "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline/core.py"
click node_fetch "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/fetching/async_fetcher.py"
click node_browser "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/fetching/browser_fetcher.py"
click node_sources "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/sources/sources.py"
click node_extract "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/extraction/extractors.py"
click node_documents "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/document_ir/parsers.py"
click node_pdf "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pdfx/service.py"
click node_quality "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/quality/quality.py"
click node_review "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/review/review_workbench.py"
click node_delivery "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline/exporters.py"
click node_convert "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/convertx/_core.py"
click node_state "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/state/state_store.py"
click node_runtime "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/runtime/scheduler.py"
click node_security "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/security/policy.py"
click node_monitor "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/scheduling/change_detector.py"
click node_plugins "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/plugins/plugin_runtime.py"
click node_market "https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/plugins/market_client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gui,node_cli toneBlue
class node_task_design,node_templates,node_plan,node_pipeline,node_fetch,node_browser,node_sources,node_object_store toneAmber
class node_extract,node_documents,node_pdf,node_quality,node_review,node_delivery,node_convert,node_web toneMint
class node_state,node_runtime,node_security,node_monitor toneRose
class node_plugins,node_market,node_user,node_ai,node_market_service toneIndigo
```

## Components

### User Interfaces

- [Desktop GUI](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/gui/main.py): `src/omnicrawler/gui/main.py`
- [CLI Commands](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/cli/_handlers.py): `src/omnicrawler/cli/_handlers.py`

### Collection Workflow

- [Task Design](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/services/natural_language_task.py): `src/omnicrawler/services/natural_language_task.py`
- [Templates](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/templates/template_application.py): `src/omnicrawler/templates/template_application.py`
- [Plan and Preflight](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline_ops/plan_compiler.py): `src/omnicrawler/pipeline_ops/plan_compiler.py`
- [Pipeline Orchestrator](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline/core.py): `src/omnicrawler/pipeline/core.py`
- [Web Fetching](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/fetching/async_fetcher.py): `src/omnicrawler/fetching/async_fetcher.py`
- [Browser Sessions](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/fetching/browser_fetcher.py): `src/omnicrawler/fetching/browser_fetcher.py`
- [Source Adapters](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/sources/sources.py): `src/omnicrawler/sources/sources.py`

### Documents and Data

- [Structured Extraction](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/extraction/extractors.py): `src/omnicrawler/extraction/extractors.py`
- [Document Parsing](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/document_ir/parsers.py): `src/omnicrawler/document_ir/parsers.py`
- [PDF Processing](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pdfx/service.py): `src/omnicrawler/pdfx/service.py`
- [Quality Checks](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/quality/quality.py): `src/omnicrawler/quality/quality.py`
- [Human Review](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/review/review_workbench.py): `src/omnicrawler/review/review_workbench.py`
- [Export and Sinks](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/pipeline/exporters.py): `src/omnicrawler/pipeline/exporters.py`
- [Format Conversion](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/convertx/_core.py): `src/omnicrawler/convertx/_core.py`

### Platform Services

- [Run State](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/state/state_store.py): `src/omnicrawler/state/state_store.py`
- [Scheduling and Recovery](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/runtime/scheduler.py): `src/omnicrawler/runtime/scheduler.py`
- [Egress and Policy](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/security/policy.py): `src/omnicrawler/security/policy.py`
- [Change Monitoring](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/scheduling/change_detector.py): `src/omnicrawler/scheduling/change_detector.py`

### Ecosystem

- [Plugin Runtime](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/plugins/plugin_runtime.py): `src/omnicrawler/plugins/plugin_runtime.py`
- [Plugin Market](https://github.com/starlife-creator/omnicrawler/blob/main/src/omnicrawler/plugins/market_client.py): `src/omnicrawler/plugins/market_client.py`

### Other components

- **Operator**
- **Websites and APIs**
- **AI Provider**
- **Object Storage**
- **Market Catalog**

## Connections

- Operator → Desktop GUI: uses
- Operator → CLI Commands: runs
- Desktop GUI → Task Design: drafts tasks
- CLI Commands → Task Design: compiles request
- Desktop GUI → Templates: applies template
- CLI Commands → Templates: manages templates
- Task Design → Plan and Preflight: creates draft
- Templates → Plan and Preflight: configures task
- Desktop GUI → Pipeline Orchestrator: starts run
- CLI Commands → Pipeline Orchestrator: dispatches run
- Pipeline Orchestrator → Plan and Preflight: plans
- Pipeline Orchestrator → Egress and Policy: checks policy
- Pipeline Orchestrator → Web Fetching: fetches pages
- Pipeline Orchestrator → Structured Extraction: extracts fields
- Pipeline Orchestrator → Document Parsing: parses documents
- Pipeline Orchestrator → Quality Checks: checks results
- Pipeline Orchestrator → Export and Sinks: exports results
- Web Fetching → Websites and APIs: requests content
- Web Fetching → Browser Sessions: uses sessions
- Web Fetching → Source Adapters: uses adapters
- Browser Sessions → Websites and APIs: loads pages
- Document Parsing → PDF Processing: delegates PDF
- Pipeline Orchestrator → Run State: records run
- Scheduling and Recovery → Pipeline Orchestrator: dispatches work
- Scheduling and Recovery → Run State: tracks recovery
- Export and Sinks → Object Storage: stores objects
- Desktop GUI → Human Review: opens review
- Human Review → Quality Checks: reviews evidence
- Desktop GUI → Format Conversion: opens converter
- CLI Commands → Format Conversion: converts files
- Change Monitoring → Websites and APIs: checks changes
- Structured Extraction → AI Provider: requests extraction
- Pipeline Orchestrator → Plugin Runtime: loads extensions
- Desktop GUI → Plugin Market: browses catalog
- Plugin Market → Market Catalog: fetches catalog