How arabold/docs-mcp-server works
Project Type & Purpose
• This is a full-stack, modular Node.js/TypeScript application that can run as a single process (standalone) or be split into microservices via Docker Compose.
• It implements a “Docs MCP Server” that scrapes, processes, indexes, and serves up-to-date documentation via the Model Context Protocol (MCP), plus a web UI and CLI for managing and querying the index.
Key Architectural Domains (Top-Level Folders)
• src/app: The HTTP/MCP server (AppServer) exposing REST/MCP endpoints.
• src/cli: A command-line interface offering the same features (scrape, search, list, etc.).
• src/mcp: Protocol layer implementing SSE/HTTP streaming for LLM clients.
• src/pipeline: Job orchestration (PipelineManager, workers, API clients).
• src/scraper: Fetching and parsing documentation from web, GitHub, npm, PyPI, or local files.
• src/splitter: Semantic chunking of fetched content into meaningful sections.
• src/store: Persistence layer—SQLite/Postgres migrations + abstractions for documents and embeddings.
• src/services: High-level facades tying together pipeline, scraper, store, and web logic.
• src/tools: CLI “tools” invoked by LLMs through MCP to queue jobs or retrieve status.
• src/web: React/Vite single-page application for end-user interaction.
Main Components
A. Clients
• Web Browser running the React Web UI
• CLI (npx @arabold/docs-mcp-server)
• MCP-compatible LLM clients (VSCode extension, Claude Desktop)
B. API Layer
• AppServer (src/app/AppServer.ts) exposing:
– REST endpoints (/api/jobs, /api/search, etc.)
– MCP endpoints (/sse, /mcp)
C. Services (src/services)
• MCPService: bridges HTTP requests to pipeline and tools
• WebService: handles Web UI data fetches
• WorkerService: dispatches background jobs
• PipelineApiService: internal HTTP client for pipeline worker
D. Pipeline & Worker
• PipelineManager & PipelineWorker (src/pipeline)
• Job Queue (in-DB or in-memory queue table)
E. Scraper & Splitter
• ScraperRegistry → ScraperService → various fetcher/middleware/strategy modules
• SplitterFactory → GreedySplitter, SemanticMarkdownSplitter, CodeContentSplitter, etc.
F. Store & Embeddings
• DocumentStore & DocumentRetrieverService (DB tables)
• EmbeddingFactory & embedding model providers (OpenAI, Vertex, AWS, Azure)
G. Database
• SQL migrations under db/migrations
• Docker volume or local directory for persistent store
H. External Services
• Embedding API providers (OpenAI, Google, AWS, Azure)
• Third-party documentation sources (websites, GitHub, npm, PyPI)
Relationships & Data Flow
1. User (Browser/CLI/LLM) → HTTP/MCP Server (AppServer)
• Browser ↔ WebService via REST
• CLI ↔ AppServer REST endpoints
• LLM ↔ AppServer via SSE/HTTP MCP endpoints
2. AppServer → Services layer
• Routes calls to MCPService, WebService, WorkerService, PipelineApiService
3. WorkerService / PipelineApiService → PipelineManager
• Enqueue or claim jobs from the Job Queue in DB
4. PipelineWorker → ScraperService
• Retrieves job details (URL, library, version)
• Fetches raw docs via ScraperRegistry → applicable Strategy (Web, GitHub, npm, local)
• Applies Middleware (HTML parsing, link extraction, sanitizing, metadata extraction)
5. PipelineWorker → Splitter
• Chunks processed docs into semantically coherent segments
6. PipelineWorker → EmbeddingFactory
• Calls external embedding model API to generate vectors
7. PipelineWorker → DocumentStore
• Persists text chunks + embeddings into DB
8. AppServer / CLI / LLM → DocumentRetrieverService
• Query embeddings + full-text index for similarity search
• Returns ranked doc segments
Architectural Patterns & Principles
• Modular, domain-driven folder structure (each major concern in its own module)
• Hexagonal/Ports & Adapters style: Services interface with core (pipeline, store) through well-defined interfaces.
• CQRS-like separation: Writes (scrape & index) vs. Reads (search & retrieval).
• Pluggable strategy & middleware pipelines for scraping and splitting.
• Configurable embedding providers behind a factory abstraction.
Essential Elements to Capture
• All four operational modes: Web UI, CLI, embedded MCP, Docker Compose.
• Core pipelines: Scraper → Middleware → Splitter → Embedding → Store.
• Interaction patterns: synchronous API vs. asynchronous job processing.
• External dependencies: model providers and documentation sources.
• Database migrations and configuration as part of the data layer.