How regenrek/deepwiki-mcp works
Project type and purpose
• This is a Node.js–based MCP server and CLI tool (“Deepwiki MCP Server”) that crawls Deepwiki.com pages, sanitizes and converts them to Markdown, and returns the result via a standard MCP interface or a CLI/HTTP transport.
File structure
• Entry points:
• bin/cli.mjs – CLI adapter for MCP requests
• src/server.ts – HTTP transport entry point
• src/index.ts – core request handler and MCP tool registration
• Core modules:
• src/lib/httpCrawler.ts – page fetching logic
• src/converter/htmlToMarkdown.ts – Markdown conversion
• src/lib/sanitizeSchema.ts – HTML sanitization
• src/lib/linkRewrite.ts – URL rewriting
• src/schemas/deepwiki.ts – JSON schema for page structure
• src/tools/deepwiki.ts & deepwikiSearch.ts – MCP tool definitions
• Utilities: src/utils*, src/types.ts
• Config & deployment: Dockerfile, build.config.ts, repomix.config.json, vite.config.ts
• Tests under tests/ verify crawler, client, server behavior
Key README insights
• Dual transport modes: stdio/CLI and HTTP
• Domain safety enforcement (only deepwiki.com)
• Concurrency, timeout, retry controlled via env vars
• Two output modes: “aggregate” vs “pages”
• No persistent storage – pure request/response service
• Docker support for containerized deployment
Main components
a. Clients & Interfaces
• MCP-compatible client (e.g., Cursor)
• CLI tool (deepwiki fetch …)
• HTTP client (curl or HTTP-enabled MCP)
b. Transport Layer
• CLI Adapter (bin/cli.mjs)
• HTTP Server (src/server.ts)
c. Core Service Pipeline
1. Request Handler (src/index.ts)
2. URL Validator & Schema Enforcer (src/schemas/deepwiki.ts)
3. HTTP Crawler (src/lib/httpCrawler.ts)
4. HTML Sanitizer (src/lib/sanitizeSchema.ts)
5. Markdown Converter (src/converter/htmlToMarkdown.ts)
6. Link Rewriter (src/lib/linkRewrite.ts)
7. Aggregator / Response Builder (returns “aggregate” or “pages” JSON)
d. Configuration
• Environment variables (concurrency, timeout, retries)
• Docker container boundary
e. External System
• Deepwiki.com (target site being crawled)
Architectural patterns & principles
• Modular pipeline: each concern (fetch, sanitize, convert, rewrite) is isolated
• Schema-driven validation ensures consistent page structure
• Single-service design: no database, stateless requests
• Pluggable transports: CLI and HTTP adapters share the same core
Technologies & frameworks
• Node.js + TypeScript
• ESM modules (bin/cli.mjs, .mjs files)
• pnpm workspace & lockfile
• Optional Docker for containerization
• Vite (build pipeline), esbuild/Rollup via build.config.ts