# arabold/docs-mcp-server architecture

> Architecture diagram of the GitHub repository arabold/docs-mcp-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arabold/docs-mcp-server
- Repository: https://github.com/arabold/docs-mcp-server

## Overview

##### Project Type & Purpose
- This is a full-stack, modular Node.js/TypeScript application that can run as a single process (standalone) or be split into microservices via Docker Compose.
- It implements a “Docs MCP Server” that scrapes, processes, indexes, and serves up-to-date documentation via the Model Context Protocol (MCP), plus a web UI and CLI for managing and querying the index.

##### Key Architectural Domains (Top-Level Folders)
- src/app: The HTTP/MCP server (AppServer) exposing REST/MCP endpoints.
- src/cli: A command-line interface offering the same features (scrape, search, list, etc.).
- src/mcp: Protocol layer implementing SSE/HTTP streaming for LLM clients.
- src/pipeline: Job orchestration (PipelineManager, workers, API clients).
- src/scraper: Fetching and parsing documentation from web, GitHub, npm, PyPI, or local files.
- src/splitter: Semantic chunking of fetched content into meaningful sections.
- src/store: Persistence layer—SQLite/Postgres migrations + abstractions for documents and embeddings.
- src/services: High-level facades tying together pipeline, scraper, store, and web logic.
- src/tools: CLI “tools” invoked by LLMs through MCP to queue jobs or retrieve status.
- src/web: React/Vite single-page application for end-user interaction.

##### Main Components
A. Clients
- Web Browser running the React Web UI
- CLI (npx @arabold/docs-mcp-server)
- MCP-compatible LLM clients (VSCode extension, Claude Desktop)
B. API Layer
- AppServer (src/app/AppServer.ts) exposing:
– REST endpoints (/api/jobs, /api/search, etc.)
– MCP endpoints (/sse, /mcp)
C. Services (src/services)
- MCPService: bridges HTTP requests to pipeline and tools
- WebService: handles Web UI data fetches
- WorkerService: dispatches background jobs
- PipelineApiService: internal HTTP client for pipeline worker
D. Pipeline & Worker
- PipelineManager & PipelineWorker (src/pipeline)
- Job Queue (in-DB or in-memory queue table)
E. Scraper & Splitter
- ScraperRegistry → ScraperService → various fetcher/middleware/strategy modules
- SplitterFactory → GreedySplitter, SemanticMarkdownSplitter, CodeContentSplitter, etc.
F. Store & Embeddings
- DocumentStore & DocumentRetrieverService (DB tables)
- EmbeddingFactory & embedding model providers (OpenAI, Vertex, AWS, Azure)
G. Database
- SQL migrations under db/migrations
- Docker volume or local directory for persistent store
H. External Services
- Embedding API providers (OpenAI, Google, AWS, Azure)
- Third-party documentation sources (websites, GitHub, npm, PyPI)

##### Relationships & Data Flow
1. User (Browser/CLI/LLM) → HTTP/MCP Server (AppServer)
- Browser ↔ WebService via REST
- CLI ↔ AppServer REST endpoints
- LLM ↔ AppServer via SSE/HTTP MCP endpoints
2. AppServer → Services layer
- Routes calls to MCPService, WebService, WorkerService, PipelineApiService
3. WorkerService / PipelineApiService → PipelineManager
- Enqueue or claim jobs from the Job Queue in DB
4. PipelineWorker → ScraperService
- Retrieves job details (URL, library, version)
- Fetches raw docs via ScraperRegistry → applicable Strategy (Web, GitHub, npm, local)
- Applies Middleware (HTML parsing, link extraction, sanitizing, metadata extraction)
5. PipelineWorker → Splitter
- Chunks processed docs into semantically coherent segments
6. PipelineWorker → EmbeddingFactory
- Calls external embedding model API to generate vectors
7. PipelineWorker → DocumentStore
- Persists text chunks + embeddings into DB
8. AppServer / CLI / LLM → DocumentRetrieverService
- Query embeddings + full-text index for similarity search
- Returns ranked doc segments

##### Architectural Patterns & Principles
- Modular, domain-driven folder structure (each major concern in its own module)
- Hexagonal/Ports & Adapters style: Services interface with core (pipeline, store) through well-defined interfaces.
- CQRS-like separation: Writes (scrape & index) vs. Reads (search & retrieval).
- Pluggable strategy & middleware pipelines for scraping and splitting.
- Configurable embedding providers behind a factory abstraction.

##### Essential Elements to Capture
- All four operational modes: Web UI, CLI, embedded MCP, Docker Compose.
- Core pipelines: Scraper → Middleware → Splitter → Embedding → Store.
- Interaction patterns: synchronous API vs. asynchronous job processing.
- External dependencies: model providers and documentation sources.
- Database migrations and configuration as part of the data layer.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Clients
    subgraph "Clients" 
        direction TB
        Browser["Web Browser\n(src/web)"]:::client
        CLI["CLI Client\n(src/cli)"]:::client
        LLMClient["MCP LLM Client"]:::client
    end

    %% API Layer
    subgraph "API Layer" 
        direction TB
        AppServer["AppServer (HTTP & MCP)\nsrc/app"]:::api
        MCPServer["MCP Server\n(src/mcp)"]:::api
    end

    %% Services Layer
    subgraph "Services" 
        direction TB
        MCPService["MCPService\nsrc/services"]:::api
        WebService["WebService\nsrc/services"]:::api
        WorkerService["WorkerService\nsrc/services"]:::api
        PipelineApiService["PipelineApiService\nsrc/services"]:::api
    end

    %% Pipeline & Worker
    subgraph "Pipeline Domain" 
        direction TB
        PipelineManager["PipelineManager\nsrc/pipeline"]:::worker
        PipelineWorker["PipelineWorker\nsrc/pipeline"]:::worker
        PipelineClient["PipelineClient & Factory\nsrc/pipeline"]:::worker
    end

    %% Scraper Domain
    subgraph "Scraper Domain" 
        direction TB
        ScraperRegistry["ScraperRegistry\nsrc/scraper"]:::worker
        ScraperService["ScraperService\nsrc/scraper"]:::worker
        Fetchers["Fetchers\nsrc/scraper/fetcher"]:::worker
        Middleware["Middleware\nsrc/scraper/middleware"]:::worker
        Strategies["Strategies\nsrc/scraper/strategies"]:::worker
    end

    %% Splitter Domain
    subgraph "Splitter Domain"
        direction TB
        SplitterFactory["SplitterFactory\nsrc/splitter"]:::worker
        GreedySplitter["GreedySplitter\nsrc/splitter"]:::worker
        SemanticSplitter["SemanticMarkdownSplitter\nsrc/splitter"]:::worker
        ContentSplitters["Content Splitters\nsrc/splitter/splitters"]:::worker
    end

    %% Store & Embeddings
    subgraph "Store & Embeddings"
        direction TB
        DocumentStore["DocumentStore\nsrc/store"]:::worker
        DocumentRetriever["DocumentRetrieverService\nsrc/store"]:::worker
        DocumentManagement["DocumentManagementService\nsrc/store"]:::worker
        EmbeddingFactory["EmbeddingFactory\nsrc/store/embeddings"]:::worker
        FixedEmbeddings["FixedDimensionEmbeddings\nsrc/store/embeddings"]:::worker
        Migrations["DB Migrations\ndb/migrations"]:::db
    end

    %% Tools
    Tools["Tools (CLI via MCP)\nsrc/tools"]:::worker

    %% Database
    DB["Database\n(SQLite/Postgres)"]:::db

    %% External Services
    subgraph "External Services"
        direction TB
        EmbeddingAPIs["Embedding API Providers"]:::external
        DocSources["Doc Sources (Web, GitHub, npm, PyPI)"]:::external
    end

    %% Connections
    Browser -->|REST| AppServer
    CLI -->|REST| AppServer
    LLMClient -->|SSE/MCP| MCPServer

    AppServer -->|routes to| MCPService
    AppServer -->|routes to| WebService
    AppServer -->|routes to| WorkerService
    AppServer -->|routes to| PipelineApiService

    WorkerService -->|enqueue/claim| PipelineManager
    PipelineApiService -->|HTTP client| PipelineManager

    PipelineManager -->|dispatch| PipelineWorker

    PipelineWorker -->|job details| ScraperService
    ScraperService -->|uses| ScraperRegistry
    ScraperRegistry -->|fetch via| Fetchers
    Fetchers -->|apply| Middleware
    Middleware -->|strategy| Strategies

    PipelineWorker -->|chunk| SplitterFactory
    SplitterFactory -->|uses| GreedySplitter
    SplitterFactory -->|uses| SemanticSplitter
    SplitterFactory -->|uses| ContentSplitters

    PipelineWorker -->|embed| EmbeddingFactory
    EmbeddingFactory -->|provider| FixedEmbeddings
    EmbeddingFactory -->|calls| EmbeddingAPIs

    PipelineWorker -->|store chunks| DocumentStore
    DocumentStore -->|persist| DB
    DocumentManagement -->|manage| DB

    WebService -->|query| DocumentRetriever
    MCPService -->|query| DocumentRetriever
    DocumentRetriever -->|vector query| DB

    DocSources -->|source docs| ScraperService

    Tools -->|invoke jobs| PipelineApiService

    %% Click Events
    click Browser "https://github.com/arabold/docs-mcp-server/tree/main/src/web"
    click CLI "https://github.com/arabold/docs-mcp-server/tree/main/src/cli"
    click MCPServer "https://github.com/arabold/docs-mcp-server/blob/main/src/mcp/mcpServer.ts"
    click AppServer "https://github.com/arabold/docs-mcp-server/blob/main/src/app/AppServer.ts"
    click MCPService "https://github.com/arabold/docs-mcp-server/blob/main/src/services/mcpService.ts"
    click WebService "https://github.com/arabold/docs-mcp-server/blob/main/src/services/webService.ts"
    click WorkerService "https://github.com/arabold/docs-mcp-server/blob/main/src/services/workerService.ts"
    click PipelineApiService "https://github.com/arabold/docs-mcp-server/blob/main/src/services/pipelineApiService.ts"
    click PipelineManager "https://github.com/arabold/docs-mcp-server/blob/main/src/pipeline/PipelineManager.ts"
    click PipelineWorker "https://github.com/arabold/docs-mcp-server/blob/main/src/pipeline/PipelineWorker.ts"
    click PipelineClient "https://github.com/arabold/docs-mcp-server/blob/main/src/pipeline/PipelineClient.ts"
    click ScraperRegistry "https://github.com/arabold/docs-mcp-server/blob/main/src/scraper/ScraperRegistry.ts"
    click ScraperService "https://github.com/arabold/docs-mcp-server/blob/main/src/scraper/ScraperService.ts"
    click Fetchers "https://github.com/arabold/docs-mcp-server/blob/main/src/scraper/fetcher/HttpFetcher.ts"
    click Middleware "https://github.com/arabold/docs-mcp-server/tree/main/src/scraper/middleware"
    click Strategies "https://github.com/arabold/docs-mcp-server/blob/main/src/scraper/strategies/GitHubScraperStrategy.ts"
    click SplitterFactory "https://github.com/arabold/docs-mcp-server/blob/main/src/splitter/index.ts"
    click GreedySplitter "https://github.com/arabold/docs-mcp-server/blob/main/src/splitter/GreedySplitter.ts"
    click SemanticSplitter "https://github.com/arabold/docs-mcp-server/blob/main/src/splitter/SemanticMarkdownSplitter.ts"
    click ContentSplitters "https://github.com/arabold/docs-mcp-server/blob/main/src/splitter/splitters/CodeContentSplitter.ts"
    click DocumentStore "https://github.com/arabold/docs-mcp-server/blob/main/src/store/DocumentStore.ts"
    click DocumentRetriever "https://github.com/arabold/docs-mcp-server/blob/main/src/store/DocumentRetrieverService.ts"
    click DocumentManagement "https://github.com/arabold/docs-mcp-server/blob/main/src/store/DocumentManagementService.ts"
    click EmbeddingFactory "https://github.com/arabold/docs-mcp-server/blob/main/src/store/embeddings/EmbeddingFactory.ts"
    click FixedEmbeddings "https://github.com/arabold/docs-mcp-server/blob/main/src/store/embeddings/FixedDimensionEmbeddings.ts"
    click Migrations "https://github.com/arabold/docs-mcp-server/tree/main/db/migrations"
    click Tools "https://github.com/arabold/docs-mcp-server/tree/main/src/tools"

    %% Styles
    classDef client fill:#bbeeff,stroke:#333,stroke-width:1px
    classDef api fill:#ccffcc,stroke:#333,stroke-width:1px
    classDef worker fill:#ffddcc,stroke:#333,stroke-width:1px
    classDef db fill:#eeeeee,stroke:#333,stroke-width:1px
    classDef external fill:#ffcccc,stroke:#333,stroke-width:1px
```
