# sahenjp/rustxng architecture

> Architecture diagram of the GitHub repository sahenjp/rustxng, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sahenjp/rustxng
- Repository: https://github.com/sahenjp/rustxng

## Overview

RustXNG is a self-hosted metasearch service: a user submits a query through HTTP or MCP, the request is parsed and dispatched to configured external providers and/or selected local indexes, and results are merged and returned in the requested format. Separate capabilities include fetching readable page text, MCP page reading, autocomplete, plugins, and answerers. The graph reflects the sampled source and README; several subsystem links are documented rather than verified from implementation excerpts.

Main entry points: [`src/aggregator.rs`](https://github.com/sahenjp/rustxng/blob/main/src/aggregator.rs) (Search Orchestrator), [`src/web/handlers.rs`](https://github.com/sahenjp/rustxng/blob/main/src/web/handlers.rs) (HTTP API and UI), [`src/fetch/direct.rs`](https://github.com/sahenjp/rustxng/blob/main/src/fetch/direct.rs) (Page Fetching), [`src/mcp/mod.rs`](https://github.com/sahenjp/rustxng/blob/main/src/mcp/mod.rs) (MCP Tools)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User Interfaces"]
  node_http["HTTP API and UI<br/>[handlers.rs]"]
  node_mcp["MCP Tools<br/>[mod.rs]"]
  node_format["Response Formats<br/>[feeds.rs]"]
end

subgraph group_search["Search Orchestration"]
  node_query["Query Parsing<br/>[query.rs]"]
  node_aggregate["Search Orchestrator<br/>[aggregator.rs]"]
  node_results["Merge and Ranking<br/>[results.rs]"]
  node_plugins["Search Plugins<br/>[mod.rs]"]
  node_answerers["Answerers<br/>[mod.rs]"]
  node_autocomplete["Autocomplete<br/>[autocomplete.rs]"]
  node_bangs["External Bangs<br/>[external_bangs.rs]"]
end

subgraph group_sources["Search Sources"]
  node_registry["Engine Registry<br/>[mod.rs]"]
  node_providers["Provider Engines<br/>[mod.rs]"]
  node_provider_services(("External Providers"))
  node_localindex[("Local Index<br/>[localindex.rs]")]
  node_crawl["Local Crawling<br/>[crawl.rs]"]
  node_wikiindex["Wikipedia Index Import"]
end

subgraph group_page["Page Retrieval"]
  node_fetch["Page Fetching<br/>[direct.rs]"]
  node_content["Text Extraction<br/>[content.rs]"]
  node_renderer["Optional Rendering<br/>[render.rs]"]
end

subgraph group_runtime["Runtime Services"]
  node_network["HTTP Networking<br/>[network.rs]"]
  node_cache[("Response Cache<br/>[cache.rs]")]
  node_settings["Runtime Settings<br/>[settings.rs]"]
  node_metrics["Metrics<br/>[metrics.rs]"]
end

node_user(("Search Client"))

node_user -->|"submits search"| node_http
node_user -->|"uses tools"| node_mcp
node_http -->|"parses query"| node_query
node_http -->|"runs search"| node_aggregate
node_mcp -->|"runs search"| node_aggregate
node_mcp -->|"reads pages"| node_fetch
node_query -->|"provides request"| node_aggregate
node_aggregate -->|"selects engines"| node_registry
node_registry -->|"builds engines"| node_providers
node_aggregate -.->|"queries providers"| node_provider_services
node_aggregate -->|"searches index"| node_localindex
node_crawl -.->|"builds index"| node_localindex
node_wikiindex -.->|"imports dump"| node_localindex
node_aggregate -->|"merges results"| node_results
node_aggregate -->|"runs plugins"| node_plugins
node_aggregate -.->|"adds answers"| node_answerers
node_http -->|"serves suggestions"| node_autocomplete
node_query -->|"resolves bangs"| node_bangs
node_http -->|"formats response"| node_format
node_http -->|"fetches raw text"| node_fetch
node_fetch -->|"extracts text"| node_content
node_fetch -->|"optionally renders"| node_renderer
node_fetch -->|"requests pages"| node_network
node_aggregate -.->|"requests providers"| node_network
node_network -->|"reads and writes"| node_cache
node_settings -->|"configures engines"| node_registry
node_settings -->|"configures fetching"| node_fetch
node_http -.->|"exposes metrics"| node_metrics

click node_http "https://github.com/sahenjp/rustxng/blob/main/src/web/handlers.rs"
click node_mcp "https://github.com/sahenjp/rustxng/blob/main/src/mcp/mod.rs"
click node_format "https://github.com/sahenjp/rustxng/blob/main/src/web/feeds.rs"
click node_query "https://github.com/sahenjp/rustxng/blob/main/src/query.rs"
click node_aggregate "https://github.com/sahenjp/rustxng/blob/main/src/aggregator.rs"
click node_results "https://github.com/sahenjp/rustxng/blob/main/src/results.rs"
click node_plugins "https://github.com/sahenjp/rustxng/blob/main/src/plugins/mod.rs"
click node_registry "https://github.com/sahenjp/rustxng/blob/main/src/engine/mod.rs"
click node_providers "https://github.com/sahenjp/rustxng/blob/main/src/engines/mod.rs"
click node_localindex "https://github.com/sahenjp/rustxng/blob/main/src/localindex.rs"
click node_crawl "https://github.com/sahenjp/rustxng/blob/main/src/crawl.rs"
click node_wikiindex "https://github.com/sahenjp/rustxng/blob/main/src/bin/rustxng-wikidump.rs"
click node_fetch "https://github.com/sahenjp/rustxng/blob/main/src/fetch/direct.rs"
click node_content "https://github.com/sahenjp/rustxng/blob/main/src/content.rs"
click node_renderer "https://github.com/sahenjp/rustxng/blob/main/src/fetch/render.rs"
click node_network "https://github.com/sahenjp/rustxng/blob/main/src/network.rs"
click node_cache "https://github.com/sahenjp/rustxng/blob/main/src/cache.rs"
click node_settings "https://github.com/sahenjp/rustxng/blob/main/src/settings.rs"
click node_answerers "https://github.com/sahenjp/rustxng/blob/main/src/answerers/mod.rs"
click node_autocomplete "https://github.com/sahenjp/rustxng/blob/main/src/autocomplete.rs"
click node_bangs "https://github.com/sahenjp/rustxng/blob/main/src/external_bangs.rs"
click node_metrics "https://github.com/sahenjp/rustxng/blob/main/src/metrics.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_http,node_mcp,node_format,node_user toneBlue
class node_query,node_aggregate,node_results,node_plugins,node_answerers,node_autocomplete,node_bangs toneAmber
class node_registry,node_providers,node_provider_services,node_localindex,node_crawl,node_wikiindex toneMint
class node_fetch,node_content,node_renderer toneRose
class node_network,node_cache,node_settings,node_metrics toneIndigo
```

## Components

### User Interfaces

- [HTTP API and UI](https://github.com/sahenjp/rustxng/blob/main/src/web/handlers.rs): `src/web/handlers.rs`
- [MCP Tools](https://github.com/sahenjp/rustxng/blob/main/src/mcp/mod.rs): `src/mcp/mod.rs`
- [Response Formats](https://github.com/sahenjp/rustxng/blob/main/src/web/feeds.rs): `src/web/feeds.rs`

### Search Orchestration

- [Query Parsing](https://github.com/sahenjp/rustxng/blob/main/src/query.rs): `src/query.rs`
- [Search Orchestrator](https://github.com/sahenjp/rustxng/blob/main/src/aggregator.rs): `src/aggregator.rs`
- [Merge and Ranking](https://github.com/sahenjp/rustxng/blob/main/src/results.rs): `src/results.rs`
- [Search Plugins](https://github.com/sahenjp/rustxng/blob/main/src/plugins/mod.rs): `src/plugins/mod.rs`
- [Answerers](https://github.com/sahenjp/rustxng/blob/main/src/answerers/mod.rs): `src/answerers/mod.rs`
- [Autocomplete](https://github.com/sahenjp/rustxng/blob/main/src/autocomplete.rs): `src/autocomplete.rs`
- [External Bangs](https://github.com/sahenjp/rustxng/blob/main/src/external_bangs.rs): `src/external_bangs.rs`

### Search Sources

- [Engine Registry](https://github.com/sahenjp/rustxng/blob/main/src/engine/mod.rs): `src/engine/mod.rs`
- [Provider Engines](https://github.com/sahenjp/rustxng/blob/main/src/engines/mod.rs): `src/engines/mod.rs`
- **External Providers**
- [Local Index](https://github.com/sahenjp/rustxng/blob/main/src/localindex.rs): `src/localindex.rs`
- [Local Crawling](https://github.com/sahenjp/rustxng/blob/main/src/crawl.rs): `src/crawl.rs`
- [Wikipedia Index Import](https://github.com/sahenjp/rustxng/blob/main/src/bin/rustxng-wikidump.rs): `src/bin/rustxng-wikidump.rs`

### Page Retrieval

- [Page Fetching](https://github.com/sahenjp/rustxng/blob/main/src/fetch/direct.rs): `src/fetch/direct.rs`
- [Text Extraction](https://github.com/sahenjp/rustxng/blob/main/src/content.rs): `src/content.rs`
- [Optional Rendering](https://github.com/sahenjp/rustxng/blob/main/src/fetch/render.rs): `src/fetch/render.rs`

### Runtime Services

- [HTTP Networking](https://github.com/sahenjp/rustxng/blob/main/src/network.rs): `src/network.rs`
- [Response Cache](https://github.com/sahenjp/rustxng/blob/main/src/cache.rs): `src/cache.rs`
- [Runtime Settings](https://github.com/sahenjp/rustxng/blob/main/src/settings.rs): `src/settings.rs`
- [Metrics](https://github.com/sahenjp/rustxng/blob/main/src/metrics.rs): `src/metrics.rs`

### Other components

- **Search Client**

## Connections

- Search Client → HTTP API and UI: submits search (evidence: `README.md`)
- Search Client → MCP Tools: uses tools (evidence: `README.md`)
- HTTP API and UI → Query Parsing: parses query (evidence: `src/web/handlers.rs`)
- HTTP API and UI → Search Orchestrator: runs search (evidence: `src/web/handlers.rs`)
- MCP Tools → Search Orchestrator: runs search (evidence: `src/mcp/mod.rs`)
- MCP Tools → Page Fetching: reads pages (evidence: `src/mcp/mod.rs`)
- Query Parsing → Search Orchestrator: provides request (evidence: `src/aggregator.rs`)
- Search Orchestrator → Engine Registry: selects engines (evidence: `src/aggregator.rs`)
- Engine Registry → Provider Engines: builds engines (evidence: `src/engine/mod.rs`)
- Search Orchestrator → External Providers: queries providers (evidence: `README.md`)
- Search Orchestrator → Local Index: searches index (evidence: `src/aggregator.rs`)
- Local Crawling → Local Index: builds index (evidence: `README.md`)
- Wikipedia Index Import → Local Index: imports dump (evidence: `README.md`)
- Search Orchestrator → Merge and Ranking: merges results (evidence: `src/aggregator.rs`)
- Search Orchestrator → Search Plugins: runs plugins (evidence: `src/aggregator.rs`)
- Search Orchestrator → Answerers: adds answers (evidence: `src/aggregator.rs`)
- HTTP API and UI → Autocomplete: serves suggestions (evidence: `src/web/handlers.rs`)
- Query Parsing → External Bangs: resolves bangs (evidence: `src/query.rs`)
- HTTP API and UI → Response Formats: formats response (evidence: `src/web/handlers.rs`)
- HTTP API and UI → Page Fetching: fetches raw text (evidence: `src/web/handlers.rs`)
- Page Fetching → Text Extraction: extracts text (evidence: `src/fetch/direct.rs`)
- Page Fetching → Optional Rendering: optionally renders (evidence: `src/fetch/direct.rs`)
- Page Fetching → HTTP Networking: requests pages (evidence: `src/fetch/direct.rs`)
- Search Orchestrator → HTTP Networking: requests providers (evidence: `src/aggregator.rs`)
- HTTP Networking → Response Cache: reads and writes (evidence: `src/network.rs`)
- Runtime Settings → Engine Registry: configures engines (evidence: `src/engine/mod.rs`)
- Runtime Settings → Page Fetching: configures fetching (evidence: `src/fetch/mod.rs`)
- HTTP API and UI → Metrics: exposes metrics (evidence: `README.md`)