mrkrsl/web-search-mcp

Purpose and Entry Point

This is a Node.js/TypeScript stdio MCP server for local LLM clients. It exposes web-search and page-extraction tools without search API keys, combining browser automation and direct HTTP. src/index.ts initializes the MCP server, registers tools, validates inputs, and formats tool responses. The distributable entry is built from this source via scripts/bundle.js.

MCP Tool Orchestration

Three tool paths share the same services:

• full-web-search: search, then concurrently fetch/extract result pages.

• get-web-search-summaries: search only, returning result metadata/snippets.

• get-single-web-page-content: extract a specified URL directly.

The orchestration boundary is src/index.ts; shared result/request contracts are in src/types.ts.

Multi-Engine Search Pipeline

src/search-engine.ts implements engine selection, parsing, fallback, and optional relevance validation. Search prioritizes browser-driven Bing, then browser-driven Brave, then HTTP-based DuckDuckGo. It may evaluate multiple engines when configured, while normal operation stops after a satisfactory result set. External dependencies are Bing, Brave Search, and DuckDuckGo HTML endpoints; no authenticated search provider is required.

Browser Lifecycle and Isolation

src/browser-pool.ts owns Playwright browser provisioning, reuse/limits, per-engine isolation, and cleanup. Chromium and Firefox are material runtime dependencies, installed separately through Playwright. Browser pool behavior is controlled by environment settings such as maximum browser count, allowed browser types, headless mode, and browser-fallback threshold. This separates expensive, stateful automation from search and extraction policy.

Content Retrieval and Extraction

src/content-extractor.ts is the primary page-fetch/extract path, favoring Axios-based HTTP for speed and falling back to browser navigation after repeated failures or inaccessible pages. It handles timeout and protocol recovery, including HTTP/2-to-HTTP/1.1 fallback behavior. src/enhanced-content-extractor.ts provides the higher-level extraction path used to obtain cleaned main-page content, stripping navigation, ads, and other peripheral material.

Concurrency, Throttling, and Reliability

For full searches, page extraction is parallelized with timeout protection; one failed URL should not prevent other results from being returned. src/rate-limiter.ts constrains outbound activity, while src/utils.ts centralizes cross-cutting normalization and operational helpers. Environment-configured defaults govern request timeout, maximum extracted length, engine quality threshold, and whether all engines are queried.

Data Boundary and Deployment

src/types.ts defines the boundary between raw engine/page responses and structured MCP tool output: search result metadata, extracted content, and errors are normalized before returning to the client. The server runs under Node.js 18+ and communicates through MCP stdio, typically launched as compiled dist/index.js from an MCP client configuration (mcp.json). It depends on unrestricted outbound web access plus locally installed Playwright browser binaries.