# mrkrsl/web-search-mcp architecture

> Architecture diagram of the GitHub repository mrkrsl/web-search-mcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mrkrsl/web-search-mcp
- Repository: https://github.com/mrkrsl/web-search-mcp

## Overview

#### Purpose and Entry Point
This is a Node.js/TypeScript stdio MCP server for local LLM clients. It exposes web-search and page-extraction tools without search API keys, combining browser automation and direct HTTP. `src/index.ts` initializes the MCP server, registers tools, validates inputs, and formats tool responses. The distributable entry is built from this source via `scripts/bundle.js`.

#### MCP Tool Orchestration
Three tool paths share the same services:
- `full-web-search`: search, then concurrently fetch/extract result pages.
- `get-web-search-summaries`: search only, returning result metadata/snippets.
- `get-single-web-page-content`: extract a specified URL directly.
The orchestration boundary is `src/index.ts`; shared result/request contracts are in `src/types.ts`.

#### Multi-Engine Search Pipeline
`src/search-engine.ts` implements engine selection, parsing, fallback, and optional relevance validation. Search prioritizes browser-driven Bing, then browser-driven Brave, then HTTP-based DuckDuckGo. It may evaluate multiple engines when configured, while normal operation stops after a satisfactory result set. External dependencies are Bing, Brave Search, and DuckDuckGo HTML endpoints; no authenticated search provider is required.

#### Browser Lifecycle and Isolation
`src/browser-pool.ts` owns Playwright browser provisioning, reuse/limits, per-engine isolation, and cleanup. Chromium and Firefox are material runtime dependencies, installed separately through Playwright. Browser pool behavior is controlled by environment settings such as maximum browser count, allowed browser types, headless mode, and browser-fallback threshold. This separates expensive, stateful automation from search and extraction policy.

#### Content Retrieval and Extraction
`src/content-extractor.ts` is the primary page-fetch/extract path, favoring Axios-based HTTP for speed and falling back to browser navigation after repeated failures or inaccessible pages. It handles timeout and protocol recovery, including HTTP/2-to-HTTP/1.1 fallback behavior. `src/enhanced-content-extractor.ts` provides the higher-level extraction path used to obtain cleaned main-page content, stripping navigation, ads, and other peripheral material.

#### Concurrency, Throttling, and Reliability
For full searches, page extraction is parallelized with timeout protection; one failed URL should not prevent other results from being returned. `src/rate-limiter.ts` constrains outbound activity, while `src/utils.ts` centralizes cross-cutting normalization and operational helpers. Environment-configured defaults govern request timeout, maximum extracted length, engine quality threshold, and whether all engines are queried.

#### Data Boundary and Deployment
`src/types.ts` defines the boundary between raw engine/page responses and structured MCP tool output: search result metadata, extracted content, and errors are normalized before returning to the client. The server runs under Node.js 18+ and communicates through MCP stdio, typically launched as compiled `dist/index.js` from an MCP client configuration (`mcp.json`). It depends on unrestricted outbound web access plus locally installed Playwright browser binaries.

Main entry points: [`src/search-engine.ts`](https://github.com/mrkrsl/web-search-mcp/blob/main/src/search-engine.ts) (Search pipeline), [`src/index.ts`](https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts) (MCP tool server), [`src/content-extractor.ts`](https://github.com/mrkrsl/web-search-mcp/blob/main/src/content-extractor.ts) (Content fetcher), [`src/browser-pool.ts`](https://github.com/mrkrsl/web-search-mcp/blob/main/src/browser-pool.ts) (Browser pool)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_mcp["MCP Server"]
  node_mcp_client(("Local LLM client<br/>MCP client"))
  node_bundle["Bundle build<br/>distribution build<br/>[bundle.js]"]
  node_server["MCP tool server<br/>stdio MCP server<br/>[index.ts]"]
  node_contracts["Result contracts<br/>TypeScript contracts<br/>[types.ts]"]
  node_full_search["full-web-search<br/>MCP tool<br/>[index.ts]"]
  node_summaries["get-web-search-summaries<br/>MCP tool<br/>[index.ts]"]
  node_single_page["get-single-web-page-content<br/>MCP tool<br/>[index.ts]"]
end

subgraph group_services["Search &amp; Extraction"]
  node_search_pipeline["Search pipeline<br/>engine selection &amp; fallback<br/>[search-engine.ts]"]
  node_browser_pool["Browser pool<br/>Playwright lifecycle<br/>[browser-pool.ts]"]
  node_content_fetcher["Content fetcher<br/>HTTP/browser retrieval"]
  node_content_cleaner["Content extractor<br/>main-content extraction"]
  node_rate_limiter["Rate limiter<br/>outbound throttling<br/>[rate-limiter.ts]"]
  node_utilities["Operational utilities<br/>shared helpers<br/>[utils.ts]"]
end

subgraph group_external["External Runtime"]
  node_bing(("Bing<br/>search endpoint"))
  node_brave(("Brave Search<br/>search endpoint"))
  node_duckduckgo(("DuckDuckGo HTML<br/>search endpoint"))
  node_web(("Target web pages<br/>HTTP destinations"))
  node_playwright{{"Chromium / Firefox<br/>Playwright browsers"}}
end

node_mcp_client -->|"stdio MCP"| node_server
node_bundle -->|"builds"| node_server
node_server -->|"exposes"| node_full_search
node_server -->|"exposes"| node_summaries
node_server -->|"exposes"| node_single_page
node_server -->|"formats"| node_contracts
node_full_search -->|"searches"| node_search_pipeline
node_full_search -->|"parallel extraction"| node_content_cleaner
node_summaries -->|"searches"| node_search_pipeline
node_single_page -->|"extracts"| node_content_cleaner
node_search_pipeline -->|"browser engines"| node_browser_pool
node_search_pipeline -->|"primary"| node_bing
node_search_pipeline -->|"fallback"| node_brave
node_search_pipeline -->|"HTTP fallback"| node_duckduckgo
node_content_cleaner -->|"retrieves"| node_content_fetcher
node_content_fetcher -.->|"recovery fallback"| node_browser_pool
node_content_fetcher -->|"HTTP fetch"| node_web
node_browser_pool -->|"provisions"| node_playwright
node_rate_limiter -.->|"constrains"| node_search_pipeline
node_rate_limiter -.->|"constrains"| node_content_fetcher
node_utilities -.->|"supports"| node_search_pipeline
node_utilities -.->|"supports"| node_content_fetcher

click node_bundle "https://github.com/mrkrsl/web-search-mcp/blob/main/scripts/bundle.js"
click node_server "https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts"
click node_contracts "https://github.com/mrkrsl/web-search-mcp/blob/main/src/types.ts"
click node_full_search "https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts"
click node_summaries "https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts"
click node_single_page "https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts"
click node_search_pipeline "https://github.com/mrkrsl/web-search-mcp/blob/main/src/search-engine.ts"
click node_browser_pool "https://github.com/mrkrsl/web-search-mcp/blob/main/src/browser-pool.ts"
click node_content_fetcher "https://github.com/mrkrsl/web-search-mcp/blob/main/src/content-extractor.ts"
click node_content_cleaner "https://github.com/mrkrsl/web-search-mcp/blob/main/src/enhanced-content-extractor.ts"
click node_rate_limiter "https://github.com/mrkrsl/web-search-mcp/blob/main/src/rate-limiter.ts"
click node_utilities "https://github.com/mrkrsl/web-search-mcp/blob/main/src/utils.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_mcp_client,node_bundle,node_server,node_contracts,node_full_search,node_summaries,node_single_page toneBlue
class node_search_pipeline,node_browser_pool,node_content_fetcher,node_content_cleaner,node_rate_limiter,node_utilities toneAmber
class node_bing,node_brave,node_duckduckgo,node_web,node_playwright toneMint
```

## Components

### MCP Server

Node.js stdio tool boundary.

- **Local LLM client**: MCP client
- [Bundle build](https://github.com/mrkrsl/web-search-mcp/blob/main/scripts/bundle.js): distribution build · Builds the distributable server entry. · `scripts/bundle.js`
- [MCP tool server](https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts): stdio MCP server · Registers tools, validates inputs, and formats responses. · `src/index.ts`
- [Result contracts](https://github.com/mrkrsl/web-search-mcp/blob/main/src/types.ts): TypeScript contracts · Normalizes search, content, and error boundaries. · `src/types.ts`
- [full-web-search](https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts): MCP tool · Searches then concurrently extracts result pages. · `src/index.ts`
- [get-web-search-summaries](https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts): MCP tool · Returns search metadata and snippets only. · `src/index.ts`
- [get-single-web-page-content](https://github.com/mrkrsl/web-search-mcp/blob/main/src/index.ts): MCP tool · Extracts content for one URL. · `src/index.ts`

### Search & Extraction

Shared retrieval services.

- [Search pipeline](https://github.com/mrkrsl/web-search-mcp/blob/main/src/search-engine.ts): engine selection & fallback · Parses engines and optionally validates relevance. · `src/search-engine.ts`
- [Browser pool](https://github.com/mrkrsl/web-search-mcp/blob/main/src/browser-pool.ts): Playwright lifecycle · Provides isolated, bounded browser automation. · `src/browser-pool.ts`
- [Content fetcher](https://github.com/mrkrsl/web-search-mcp/blob/main/src/content-extractor.ts): HTTP/browser retrieval · Uses Axios first, then browser recovery. · `src/content-extractor.ts`
- [Content extractor](https://github.com/mrkrsl/web-search-mcp/blob/main/src/enhanced-content-extractor.ts): main-content extraction · Cleans page content and removes peripheral material. · `src/enhanced-content-extractor.ts`
- [Rate limiter](https://github.com/mrkrsl/web-search-mcp/blob/main/src/rate-limiter.ts): outbound throttling · `src/rate-limiter.ts`
- [Operational utilities](https://github.com/mrkrsl/web-search-mcp/blob/main/src/utils.ts): shared helpers · Normalization and reliability helpers. · `src/utils.ts`

### External Runtime

Search endpoints and browser runtime.

- **Bing**: search endpoint
- **Brave Search**: search endpoint
- **DuckDuckGo HTML**: search endpoint
- **Target web pages**: HTTP destinations
- **Chromium / Firefox**: Playwright browsers · Locally installed browser binaries.

## Connections

- Local LLM client → MCP tool server: stdio MCP
- Bundle build → MCP tool server: builds
- MCP tool server → full-web-search: exposes
- MCP tool server → get-web-search-summaries: exposes
- MCP tool server → get-single-web-page-content: exposes
- MCP tool server → Result contracts: formats
- full-web-search → Search pipeline: searches
- full-web-search → Content extractor: parallel extraction
- get-web-search-summaries → Search pipeline: searches
- get-single-web-page-content → Content extractor: extracts
- Search pipeline → Browser pool: browser engines
- Search pipeline → Bing: primary
- Search pipeline → Brave Search: fallback
- Search pipeline → DuckDuckGo HTML: HTTP fallback
- Content extractor → Content fetcher: retrieves
- Content fetcher → Browser pool: recovery fallback
- Content fetcher → Target web pages: HTTP fetch
- Browser pool → Chromium / Firefox: provisions
- Rate limiter → Search pipeline: constrains
- Rate limiter → Content fetcher: constrains
- Operational utilities → Search pipeline: supports
- Operational utilities → Content fetcher: supports