How starlife-creator/omnicrawler works
OmniCrawler is a desktop-oriented data collection platform with CLI and GUI entry points. Its primary workflow turns a user’s URL or task description into a configured, reviewable crawl; plans and runs collection; parses and extracts structured data; checks quality; and delivers results. The README documents the pipeline stages and major feature areas, while sampled source confirms selected GUI, CLI, storage, and plugin behavior. Some subsystem links below reflect documented architecture rather than inspected call sites.
Main entry points: core.py Pipeline Orchestrator · main.py Desktop GUI · _handlers.py CLI Commands · async_fetcher.py Web Fetching
Components 27
User Interfaces
- Desktop GUI
src/omnicrawler/gui/main.py - CLI Commands
src/omnicrawler/cli/_handlers.py
Collection Workflow
- Task Design
src/omnicrawler/services/natural_language_task.py - Templates
src/omnicrawler/templates/template_application.py - Plan and Preflight
src/omnicrawler/pipeline_ops/plan_compiler.py - Pipeline Orchestrator
src/omnicrawler/pipeline/core.py - Web Fetching
src/omnicrawler/fetching/async_fetcher.py - Browser Sessions
src/omnicrawler/fetching/browser_fetcher.py - Source Adapters
src/omnicrawler/sources/sources.py
Documents and Data
- Structured Extraction
src/omnicrawler/extraction/extractors.py - Document Parsing
src/omnicrawler/document_ir/parsers.py - PDF Processing
src/omnicrawler/pdfx/service.py - Quality Checks
src/omnicrawler/quality/quality.py - Human Review
src/omnicrawler/review/review_workbench.py - Export and Sinks
src/omnicrawler/pipeline/exporters.py - Format Conversion
src/omnicrawler/convertx/_core.py
Platform Services
- Run State
src/omnicrawler/state/state_store.py - Scheduling and Recovery
src/omnicrawler/runtime/scheduler.py - Egress and Policy
src/omnicrawler/security/policy.py - Change Monitoring
src/omnicrawler/scheduling/change_detector.py
Ecosystem
- Plugin Runtime
src/omnicrawler/plugins/plugin_runtime.py - Plugin Market
src/omnicrawler/plugins/market_client.py
Other components
- Operator
- Websites and APIs
- AI Provider
- Object Storage
- Market Catalog
Connections 35
- Operator → Desktop GUI: uses
- Operator → CLI Commands: runs
- Desktop GUI → Task Design: drafts tasks
- CLI Commands → Task Design: compiles request
- Desktop GUI → Templates: applies template
- CLI Commands → Templates: manages templates
- Task Design → Plan and Preflight: creates draft
- Templates → Plan and Preflight: configures task
- Desktop GUI → Pipeline Orchestrator: starts run
- CLI Commands → Pipeline Orchestrator: dispatches run
- Pipeline Orchestrator → Plan and Preflight: plans
- Pipeline Orchestrator → Egress and Policy: checks policy
- Pipeline Orchestrator → Web Fetching: fetches pages
- Pipeline Orchestrator → Structured Extraction: extracts fields
- Pipeline Orchestrator → Document Parsing: parses documents
- Pipeline Orchestrator → Quality Checks: checks results
- Pipeline Orchestrator → Export and Sinks: exports results
- Web Fetching → Websites and APIs: requests content
- Web Fetching → Browser Sessions: uses sessions
- Web Fetching → Source Adapters: uses adapters
- Browser Sessions → Websites and APIs: loads pages
- Document Parsing → PDF Processing: delegates PDF
- Pipeline Orchestrator → Run State: records run
- Scheduling and Recovery → Pipeline Orchestrator: dispatches work
- Scheduling and Recovery → Run State: tracks recovery
- Export and Sinks → Object Storage: stores objects
- Desktop GUI → Human Review: opens review
- Human Review → Quality Checks: reviews evidence
- Desktop GUI → Format Conversion: opens converter
- CLI Commands → Format Conversion: converts files
- Change Monitoring → Websites and APIs: checks changes
- Structured Extraction → AI Provider: requests extraction
- Pipeline Orchestrator → Plugin Runtime: loads extensions
- Desktop GUI → Plugin Market: browses catalog
- Plugin Market → Market Catalog: fetches catalog