# regenrek/deepwiki-mcp architecture

> Architecture diagram of the GitHub repository regenrek/deepwiki-mcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/regenrek/deepwiki-mcp
- Repository: https://github.com/regenrek/deepwiki-mcp

## Overview

##### Project type and purpose
- This is a Node.js–based MCP server and CLI tool (“Deepwiki MCP Server”) that crawls Deepwiki.com pages, sanitizes and converts them to Markdown, and returns the result via a standard MCP interface or a CLI/HTTP transport.

##### File structure
- Entry points:
- bin/cli.mjs – CLI adapter for MCP requests
- src/server.ts – HTTP transport entry point
- src/index.ts – core request handler and MCP tool registration
- Core modules:
- src/lib/httpCrawler.ts – page fetching logic
- src/converter/htmlToMarkdown.ts – Markdown conversion
- src/lib/sanitizeSchema.ts – HTML sanitization
- src/lib/linkRewrite.ts – URL rewriting
- src/schemas/deepwiki.ts – JSON schema for page structure
- src/tools/deepwiki.ts & deepwikiSearch.ts – MCP tool definitions
- Utilities: src/utils*, src/types.ts
- Config & deployment: Dockerfile, build.config.ts, repomix.config.json, vite.config.ts
- Tests under tests/ verify crawler, client, server behavior

##### Key README insights
- Dual transport modes: stdio/CLI and HTTP
- Domain safety enforcement (only deepwiki.com)
- Concurrency, timeout, retry controlled via env vars
- Two output modes: “aggregate” vs “pages”
- No persistent storage – pure request/response service
- Docker support for containerized deployment

##### Main components
a. Clients & Interfaces
- MCP-compatible client (e.g., Cursor)
- CLI tool (`deepwiki fetch …`)
- HTTP client (curl or HTTP-enabled MCP)
b. Transport Layer
- CLI Adapter (bin/cli.mjs)
- HTTP Server (src/server.ts)
c. Core Service Pipeline
1. Request Handler (src/index.ts)
2. URL Validator & Schema Enforcer (src/schemas/deepwiki.ts)
3. HTTP Crawler (src/lib/httpCrawler.ts)
4. HTML Sanitizer (src/lib/sanitizeSchema.ts)
5. Markdown Converter (src/converter/htmlToMarkdown.ts)
6. Link Rewriter (src/lib/linkRewrite.ts)
7. Aggregator / Response Builder (returns “aggregate” or “pages” JSON)
d. Configuration
- Environment variables (concurrency, timeout, retries)
- Docker container boundary
e. External System
- Deepwiki.com (target site being crawled)

##### Architectural patterns & principles
- Modular pipeline: each concern (fetch, sanitize, convert, rewrite) is isolated
- Schema-driven validation ensures consistent page structure
- Single-service design: no database, stateless requests
- Pluggable transports: CLI and HTTP adapters share the same core

##### Technologies & frameworks
- Node.js + TypeScript
- ESM modules (bin/cli.mjs, .mjs files)
- pnpm workspace & lockfile
- Optional Docker for containerization
- Vite (build pipeline), esbuild/Rollup via build.config.ts

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Clients
    subgraph "Clients"
        CLI_Client["CLI Tool"]:::client
        HTTP_Client["HTTP Client (curl)"]:::client
        MCP_Client["MCP Client (Cursor)"]:::client
    end

    %% Transport Layer
    subgraph "Transport Layer"
        CLI_Adapter["CLI Adapter"]:::client
        HTTP_Server["HTTP Server"]:::client
    end

    %% Docker Container
    subgraph DC["Docker Container"]
        direction TB
        %% Core Pipeline
        subgraph "Core Pipeline"
            direction TB
            Request_Handler["Request Handler\n(MCP Registration)"]:::pipeline
            URL_Validator["URL Validator & Schema Enforcer"]:::pipeline
            HTTP_Crawler["HTTP Crawler"]:::pipeline
            HTML_Sanitizer["HTML Sanitizer"]:::pipeline
            MD_Converter["Markdown Converter"]:::pipeline
            Link_Rewriter["Link Rewriter"]:::pipeline
            Aggregator["Aggregator / Response Builder"]:::pipeline
        end

        %% MCP Tools
        subgraph "MCP Tools"
            DeepwikiTool["Deepwiki Fetch Tool"]:::pipeline
            DeepwikiSearch["Deepwiki Search Tool"]:::pipeline
        end

        %% Config
        Env["Environment Variables\n(concurrency, timeout, retries)"]:::config
    end
    style DC stroke-dasharray: 5 5

    %% External System
    DeepWiki["Deepwiki.com"]:::external

    %% Connections
    CLI_Client -->|"deepwiki fetch"| CLI_Adapter
    HTTP_Client -->|"POST /fetch"| HTTP_Server
    MCP_Client -->|"MCP request"| CLI_Adapter
    MCP_Client -->|"MCP request"| HTTP_Server

    CLI_Adapter -->|"invoke handler"| Request_Handler
    HTTP_Server -->|"invoke handler"| Request_Handler

    Request_Handler -->|"register tools"| DeepwikiTool
    Request_Handler -->|"register tools"| DeepwikiSearch

    Request_Handler -->|"validate URL"| URL_Validator
    URL_Validator -->|"fetch HTML"| HTTP_Crawler
    HTTP_Crawler --> DeepWiki
    DeepWiki --> HTTP_Crawler
    HTTP_Crawler -->|"sanitize HTML"| HTML_Sanitizer
    HTML_Sanitizer -->|"convert to Markdown"| MD_Converter
    MD_Converter -->|"rewrite links"| Link_Rewriter
    Link_Rewriter -->|"build response"| Aggregator

    Aggregator -->|"return JSON"| CLI_Adapter
    Aggregator -->|"return JSON"| HTTP_Server

    Env -.-> HTTP_Crawler

    %% Click Events
    click CLI_Adapter "https://github.com/regenrek/deepwiki-mcp/blob/main/bin/cli.mjs"
    click HTTP_Server "https://github.com/regenrek/deepwiki-mcp/blob/main/src/server.ts"
    click Request_Handler "https://github.com/regenrek/deepwiki-mcp/blob/main/src/index.ts"
    click URL_Validator "https://github.com/regenrek/deepwiki-mcp/blob/main/src/schemas/deepwiki.ts"
    click HTTP_Crawler "https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/httpCrawler.ts"
    click HTML_Sanitizer "https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/sanitizeSchema.ts"
    click MD_Converter "https://github.com/regenrek/deepwiki-mcp/blob/main/src/converter/htmlToMarkdown.ts"
    click Link_Rewriter "https://github.com/regenrek/deepwiki-mcp/blob/main/src/lib/linkRewrite.ts"
    click DeepwikiTool "https://github.com/regenrek/deepwiki-mcp/blob/main/src/tools/deepwiki.ts"
    click DeepwikiSearch "https://github.com/regenrek/deepwiki-mcp/blob/main/src/tools/deepwikiSearch.ts"

    %% Styles
    classDef client fill:#e0f7e9,stroke:#34a853,stroke-width:2px
    classDef pipeline fill:#e8f0fe,stroke:#1a73e8,stroke-width:2px
    classDef external fill:#fce8e6,stroke:#ea4335,stroke-width:2px
    classDef config fill:#f5f5f5,stroke:#80868b,stroke-width:2px
```
