# olehbozhok/web_extractor architecture

> Architecture diagram of the GitHub repository olehbozhok/web_extractor, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/olehbozhok/web_extractor
- Repository: https://github.com/olehbozhok/web_extractor

## Overview

web_extractor is a local web-fetching and reading service for LLM agents, exposed over MCP. Agents request page text or URL checks; the service validates and normalizes URLs, applies cache/coalescing/domain controls, fetches static or rendered content, extracts text and links, and returns paginated or searchable results. It also records request events and provides a web UI for dashboard, event, agent, and cache inspection. The graph follows documented capabilities and indexed references; some wiring is inferred only from those references, so unsampled implementation details are not asserted.

Main entry points: [`src/web_extractor/service/pipeline.py`](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/pipeline.py) (Fetch pipeline), [`src/web_extractor/mcp_api/app.py`](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/mcp_api/app.py) (App composition), [`src/web_extractor/mcp_api/tools.py`](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/mcp_api/tools.py) (MCP tools), [`src/web_extractor/storage/database.py`](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/database.py) (SQLite database)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["Interfaces"]
  node_mcp["MCP tools<br/>[tools.py]"]
  node_composition["App composition<br/>[app.py]"]
  node_webui["Web routes<br/>[routes.py]"]
end

subgraph group_runtime["Fetch and extraction"]
  node_pipeline["Fetch pipeline<br/>[pipeline.py]"]
  node_urlrules["URL rules<br/>[urls.py]"]
  node_ssrf["SSRF rules<br/>[ssrf.py]"]
  node_httpfetch["HTTP fetcher<br/>[http_fetcher.py]"]
  node_browserfetch["Browser fetcher<br/>[browser_fetcher.py]"]
  node_dispatch["Extractor dispatch<br/>[dispatch.py]"]
  node_trafilatura["Text extraction"]
  node_pdf["PDF extraction<br/>[pdf_extractor.py]"]
  node_listing["Listing links<br/>[listing_links.py]"]
end

subgraph group_state["State and records"]
  node_cache[("Page cache")]
  node_cursors["Read cursors<br/>[cursor_service.py]"]
  node_cursorstore[("Cursor records")]
  node_events[("Event journal")]
  node_database[("SQLite database<br/>[database.py]")]
  node_search["Page search<br/>[search.py]"]
  node_types["Domain records<br/>[records.py]"]
end

subgraph group_observability["Monitoring UI"]
  node_stats["Usage statistics<br/>[stats.py]"]
  node_eventquery["Event queries<br/>[event_query.py]"]
  node_view["HTML views<br/>[view.py]"]
end

node_agent(("LLM agent"))
node_browser["Crawl4AI connector"]

node_agent -->|"calls tools"| node_mcp
node_composition -->|"mounts"| node_mcp
node_composition -->|"wires"| node_pipeline
node_composition -->|"configures"| node_httpfetch
node_composition -->|"configures"| node_browserfetch
node_composition -->|"configures"| node_dispatch
node_mcp -->|"requests fetch"| node_pipeline
node_mcp -->|"reads more"| node_cursors
node_mcp -->|"searches text"| node_search
node_pipeline -->|"normalizes"| node_urlrules
node_pipeline -->|"fetches static"| node_httpfetch
node_pipeline -->|"fetches rendered"| node_browserfetch
node_pipeline -->|"extracts content"| node_dispatch
node_pipeline -->|"reads and writes"| node_cache
node_pipeline -->|"appends events"| node_events
node_pipeline -->|"creates records"| node_types
node_httpfetch -.->|"checks targets"| node_ssrf
node_browserfetch -->|"checks targets"| node_ssrf
node_browserfetch -->|"renders via"| node_browser
node_dispatch -.->|"selects extractor"| node_trafilatura
node_dispatch -.->|"selects extractor"| node_pdf
node_pipeline -->|"collects links"| node_listing
node_cursors -->|"reads cached text"| node_cache
node_cursors -->|"issues and resolves"| node_cursorstore
node_cursors -->|"re-extracts HTML"| node_trafilatura
node_cache -->|"queries"| node_database
node_events -->|"writes journal"| node_database
node_cursorstore -->|"stores cursors"| node_database
node_webui -->|"loads statistics"| node_stats
node_webui -->|"filters events"| node_eventquery
node_webui -->|"manages cache"| node_cache
node_webui -->|"renders pages"| node_view
node_eventquery -->|"queries events"| node_database
node_stats -.->|"aggregates usage"| node_database

click node_mcp "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/mcp_api/tools.py"
click node_composition "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/mcp_api/app.py"
click node_webui "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/web/routes.py"
click node_pipeline "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/pipeline.py"
click node_urlrules "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/domain/urls.py"
click node_ssrf "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/domain/ssrf.py"
click node_httpfetch "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/fetchers/http_fetcher.py"
click node_browserfetch "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/fetchers/browser_fetcher.py"
click node_dispatch "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/dispatch.py"
click node_trafilatura "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/trafilatura_extractor.py"
click node_pdf "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/pdf_extractor.py"
click node_listing "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/listing_links.py"
click node_cache "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/cache_repository.py"
click node_cursors "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/cursor_service.py"
click node_cursorstore "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/cursor_repository.py"
click node_events "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/event_repository.py"
click node_database "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/database.py"
click node_search "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/search.py"
click node_stats "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/stats.py"
click node_eventquery "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/event_query.py"
click node_view "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/web/view.py"
click node_types "https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/domain/records.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mcp,node_composition,node_webui toneBlue
class node_pipeline,node_urlrules,node_ssrf,node_httpfetch,node_browserfetch,node_dispatch,node_trafilatura,node_pdf,node_listing toneAmber
class node_cache,node_cursors,node_cursorstore,node_events,node_database,node_search,node_types toneMint
class node_stats,node_eventquery,node_view toneRose
class node_agent,node_browser toneIndigo
```

## Components

### Interfaces

- [MCP tools](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/mcp_api/tools.py): `src/web_extractor/mcp_api/tools.py`
- [App composition](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/mcp_api/app.py): `src/web_extractor/mcp_api/app.py`
- [Web routes](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/web/routes.py): `src/web_extractor/web/routes.py`

### Fetch and extraction

- [Fetch pipeline](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/pipeline.py): `src/web_extractor/service/pipeline.py`
- [URL rules](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/domain/urls.py): `src/web_extractor/domain/urls.py`
- [SSRF rules](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/domain/ssrf.py): `src/web_extractor/domain/ssrf.py`
- [HTTP fetcher](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/fetchers/http_fetcher.py): `src/web_extractor/fetchers/http_fetcher.py`
- [Browser fetcher](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/fetchers/browser_fetcher.py): `src/web_extractor/fetchers/browser_fetcher.py`
- [Extractor dispatch](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/dispatch.py): `src/web_extractor/extractors/dispatch.py`
- [Text extraction](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/trafilatura_extractor.py): `src/web_extractor/extractors/trafilatura_extractor.py`
- [PDF extraction](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/pdf_extractor.py): `src/web_extractor/extractors/pdf_extractor.py`
- [Listing links](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/extractors/listing_links.py): `src/web_extractor/extractors/listing_links.py`

### State and records

- [Page cache](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/cache_repository.py): `src/web_extractor/storage/cache_repository.py`
- [Read cursors](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/cursor_service.py): `src/web_extractor/service/cursor_service.py`
- [Cursor records](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/cursor_repository.py): `src/web_extractor/storage/cursor_repository.py`
- [Event journal](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/event_repository.py): `src/web_extractor/storage/event_repository.py`
- [SQLite database](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/database.py): `src/web_extractor/storage/database.py`
- [Page search](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/service/search.py): `src/web_extractor/service/search.py`
- [Domain records](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/domain/records.py): `src/web_extractor/domain/records.py`

### Monitoring UI

- [Usage statistics](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/stats.py): `src/web_extractor/storage/stats.py`
- [Event queries](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/storage/event_query.py): `src/web_extractor/storage/event_query.py`
- [HTML views](https://github.com/olehbozhok/web_extractor/blob/main/src/web_extractor/web/view.py): `src/web_extractor/web/view.py`

### Other components

- **LLM agent**
- **Crawl4AI connector**

## Connections

- LLM agent → MCP tools: calls tools
- App composition → MCP tools: mounts (evidence: `src/web_extractor/mcp_api/app.py`)
- App composition → Fetch pipeline: wires (evidence: `src/web_extractor/mcp_api/app.py`)
- App composition → HTTP fetcher: configures (evidence: `src/web_extractor/mcp_api/app.py`)
- App composition → Browser fetcher: configures (evidence: `src/web_extractor/mcp_api/app.py`)
- App composition → Extractor dispatch: configures (evidence: `src/web_extractor/mcp_api/app.py`)
- MCP tools → Fetch pipeline: requests fetch (evidence: `src/web_extractor/mcp_api/tools.py`)
- MCP tools → Read cursors: reads more (evidence: `src/web_extractor/mcp_api/tools.py`)
- MCP tools → Page search: searches text (evidence: `src/web_extractor/mcp_api/tools.py`)
- Fetch pipeline → URL rules: normalizes (evidence: `src/web_extractor/service/pipeline.py`)
- Fetch pipeline → HTTP fetcher: fetches static (evidence: `src/web_extractor/service/pipeline.py`)
- Fetch pipeline → Browser fetcher: fetches rendered (evidence: `src/web_extractor/service/pipeline.py`)
- Fetch pipeline → Extractor dispatch: extracts content (evidence: `src/web_extractor/service/pipeline.py`)
- Fetch pipeline → Page cache: reads and writes (evidence: `src/web_extractor/service/pipeline.py`)
- Fetch pipeline → Event journal: appends events (evidence: `src/web_extractor/service/pipeline.py`)
- Fetch pipeline → Domain records: creates records (evidence: `src/web_extractor/service/pipeline.py`)
- HTTP fetcher → SSRF rules: checks targets
- Browser fetcher → SSRF rules: checks targets (evidence: `src/web_extractor/fetchers/browser_fetcher.py`)
- Browser fetcher → Crawl4AI connector: renders via
- Extractor dispatch → Text extraction: selects extractor
- Extractor dispatch → PDF extraction: selects extractor
- Fetch pipeline → Listing links: collects links (evidence: `src/web_extractor/service/pipeline.py`)
- Read cursors → Page cache: reads cached text (evidence: `src/web_extractor/service/cursor_service.py`)
- Read cursors → Cursor records: issues and resolves (evidence: `src/web_extractor/service/cursor_service.py`)
- Read cursors → Text extraction: re-extracts HTML (evidence: `src/web_extractor/service/cursor_service.py`)
- Page cache → SQLite database: queries (evidence: `src/web_extractor/storage/cache_repository.py`)
- Event journal → SQLite database: writes journal (evidence: `src/web_extractor/storage/event_repository.py`)
- Cursor records → SQLite database: stores cursors (evidence: `src/web_extractor/storage/cursor_repository.py`)
- Web routes → Usage statistics: loads statistics (evidence: `src/web_extractor/web/routes.py`)
- Web routes → Event queries: filters events (evidence: `src/web_extractor/web/routes.py`)
- Web routes → Page cache: manages cache (evidence: `src/web_extractor/web/routes.py`)
- Web routes → HTML views: renders pages (evidence: `src/web_extractor/web/routes.py`)
- Event queries → SQLite database: queries events (evidence: `src/web_extractor/storage/event_query.py`)
- Usage statistics → SQLite database: aggregates usage