How olehbozhok/web_extractor works
web_extractor is a local web-fetching and reading service for LLM agents, exposed over MCP. Agents request page text or URL checks; the service validates and normalizes URLs, applies cache/coalescing/domain controls, fetches static or rendered content, extracts text and links, and returns paginated or searchable results. It also records request events and provides a web UI for dashboard, event, agent, and cache inspection. The graph follows documented capabilities and indexed references; some wiring is inferred only from those references, so unsampled implementation details are not asserted.
Main entry points: pipeline.py Fetch pipeline · app.py App composition · tools.py MCP tools · database.py SQLite database
Components 24
Interfaces
- MCP tools
src/web_extractor/mcp_api/tools.py - App composition
src/web_extractor/mcp_api/app.py - Web routes
src/web_extractor/web/routes.py
Fetch and extraction
- Fetch pipeline
src/web_extractor/service/pipeline.py - URL rules
src/web_extractor/domain/urls.py - SSRF rules
src/web_extractor/domain/ssrf.py - HTTP fetcher
src/web_extractor/fetchers/http_fetcher.py - Browser fetcher
src/web_extractor/fetchers/browser_fetcher.py - Extractor dispatch
src/web_extractor/extractors/dispatch.py - Text extraction
src/web_extractor/extractors/trafilatura_extractor.py - PDF extraction
src/web_extractor/extractors/pdf_extractor.py - Listing links
src/web_extractor/extractors/listing_links.py
State and records
- Page cache
src/web_extractor/storage/cache_repository.py - Read cursors
src/web_extractor/service/cursor_service.py - Cursor records
src/web_extractor/storage/cursor_repository.py - Event journal
src/web_extractor/storage/event_repository.py - SQLite database
src/web_extractor/storage/database.py - Page search
src/web_extractor/service/search.py - Domain records
src/web_extractor/domain/records.py
Monitoring UI
- Usage statistics
src/web_extractor/storage/stats.py - Event queries
src/web_extractor/storage/event_query.py - HTML views
src/web_extractor/web/view.py
Other components
- LLM agent
- Crawl4AI connector
Connections 34
- LLM agent → MCP tools: calls tools
- App composition → MCP tools: mounts
src/web_extractor/mcp_api/app.py - App composition → Fetch pipeline: wires
src/web_extractor/mcp_api/app.py - App composition → HTTP fetcher: configures
src/web_extractor/mcp_api/app.py - App composition → Browser fetcher: configures
src/web_extractor/mcp_api/app.py - App composition → Extractor dispatch: configures
src/web_extractor/mcp_api/app.py - MCP tools → Fetch pipeline: requests fetch
src/web_extractor/mcp_api/tools.py - MCP tools → Read cursors: reads more
src/web_extractor/mcp_api/tools.py - MCP tools → Page search: searches text
src/web_extractor/mcp_api/tools.py - Fetch pipeline → URL rules: normalizes
src/web_extractor/service/pipeline.py - Fetch pipeline → HTTP fetcher: fetches static
src/web_extractor/service/pipeline.py - Fetch pipeline → Browser fetcher: fetches rendered
src/web_extractor/service/pipeline.py - Fetch pipeline → Extractor dispatch: extracts content
src/web_extractor/service/pipeline.py - Fetch pipeline → Page cache: reads and writes
src/web_extractor/service/pipeline.py - Fetch pipeline → Event journal: appends events
src/web_extractor/service/pipeline.py - Fetch pipeline → Domain records: creates records
src/web_extractor/service/pipeline.py - HTTP fetcher → SSRF rules: checks targets
- Browser fetcher → SSRF rules: checks targets
src/web_extractor/fetchers/browser_fetcher.py - Browser fetcher → Crawl4AI connector: renders via
- Extractor dispatch → Text extraction: selects extractor
- Extractor dispatch → PDF extraction: selects extractor
- Fetch pipeline → Listing links: collects links
src/web_extractor/service/pipeline.py - Read cursors → Page cache: reads cached text
src/web_extractor/service/cursor_service.py - Read cursors → Cursor records: issues and resolves
src/web_extractor/service/cursor_service.py - Read cursors → Text extraction: re-extracts HTML
src/web_extractor/service/cursor_service.py - Page cache → SQLite database: queries
src/web_extractor/storage/cache_repository.py - Event journal → SQLite database: writes journal
src/web_extractor/storage/event_repository.py - Cursor records → SQLite database: stores cursors
src/web_extractor/storage/cursor_repository.py - Web routes → Usage statistics: loads statistics
src/web_extractor/web/routes.py - Web routes → Event queries: filters events
src/web_extractor/web/routes.py - Web routes → Page cache: manages cache
src/web_extractor/web/routes.py - Web routes → HTML views: renders pages
src/web_extractor/web/routes.py - Event queries → SQLite database: queries events
src/web_extractor/storage/event_query.py - Usage statistics → SQLite database: aggregates usage