How dhruvtoprani/docflow-mcp works
1. What this repo is
- DocFlow MCP is a TypeScript/Node.js MCP server that turns a documentation URL plus a goal into a compact, implementation-oriented context pack.
- It is stateless: no database, no auth layer, no user sessions, no persistence.
2. Primary user-facing contract
- The main MCP tool is extract_docs_context.
- Inputs: url, goal, and optional stack.
- Output: structured documentation context, including a compact markdown pack, coverage signals, missing-info warnings, prompt-injection flags, and size-reduction stats.
3. Core processing pipeline
- The extraction flow is concentrated under src/core.
- The likely sequence is:
- normalize and fetch the target page
- parse and clean HTML
- convert content to markdown
- detect meaningful documentation sections
- extract task-relevant context
- compact implementation guidance
- estimate token savings
- sanitize suspicious prompt-injection content
- This core layer is the main architecture boundary: it contains the repo’s business logic, independent of MCP transport.
4. Document intelligence layer
- detectDocSections.ts, extractDocsContext.ts, and generateContextPack.ts are the central “understanding” modules.
- cleanHtml.ts, htmlToMarkdown.ts, and fetchPage.ts handle the document ingestion and normalization side.
- compactImplementationContext.ts and estimateTokenSavings.ts shape the final output for AI-assisted coding use.
5. Safety and trust boundary
- sanitizePromptInjection.ts is a dedicated defense layer against hostile or misleading documentation text.
- The README’s emphasis on hallucination reduction and injection detection suggests the server treats fetched docs as untrusted input and filters them before packaging.
6. Schemas, types, and validation
- src/schemas/extractDocsContext.schema.ts defines the shape of the extraction output.
- src/types/docflow.ts holds shared domain types.
- This separates the free-form document processing from the contract exposed to MCP clients.
7. Transport and server layer
- src/server/mcpServer.ts and src/server/httpServer.ts wrap the core extraction logic in MCP transports.
- src/server/tools/extractDocsContextTool.ts is the bridge between MCP protocol requests and the core extractor.
- The repo supports both stdio MCP and Streamable HTTP MCP, with /mcp as the HTTP endpoint and /healthz as the liveness check.
8. HTTP / deployment surface
- src/http.ts, src/index.ts, and the api/ folder indicate shared server entrypoints and deployment adapters.
- api/mcp.ts and api/healthz.ts are likely serverless-compatible endpoints for hosted deployment.
- vercel.json plus the README’s live URLs show the production target is Vercel-hosted HTTP MCP.
9. Local developer workflow
- src/dev/runExtract.ts provides a local extraction runner for sample inputs.
- examples/ contains sample MCP config and sample input/output artifacts to demonstrate the tool contract and expected results.
- Dockerfile supports containerized execution.
10. Evaluation subsystem
- src/eval/, eval/, and the npm run eval:* scripts form a separate quality-checking pipeline.
- This subsystem compares DocFlow’s extracted context against a rendered browser copy/paste baseline using rubric-based and fast keyword-based methods.
- It is architecturally separate from the runtime server but important because it validates the extraction strategy.
11. Tests as architecture documentation
- The tests/ directory covers core extraction, prompt-injection sanitization, context pack generation, HTTP server behavior, and MCP harness integration.
- The test suite reflects the main boundaries: core logic, transport layer, and end-to-end tool behavior.
12. Static docs and landing page
- docs/ and public/ are separate static-site assets for the project landing page and connector instructions.
- These assets are not part of the MCP runtime itself; they support discovery, documentation, and GitHub Pages deployment.
13. Key architectural constraint
- The scope is intentionally narrow: single-page documentation extraction only.
- The roadmap mentions future crawling and multi-page support, but the current repo is centered on one fetch-to-pack pipeline with minimal external dependencies.
14. How to diagram it
- A clear diagram should show:
- MCP client or HTTP caller
- transport layer (stdio or /mcp)
- tool wrapper
- core document-processing pipeline
- output schema/response
- optional eval and static-docs side systems
- The most important boundary is between untrusted remote documentation content and the trusted structured output produced by the core extractor.
Main entry points: extractDocsContextTool.ts Doc tool · extractDocsContext.ts Context extractor · generateContextPack.ts Context pack · mcpServer.ts MCP server
Components 21
Runtime
MCP server and HTTP entrypoints
- CLI entry · Main process bootstrap and mode selection
src/index.ts - HTTP entry · Shared HTTP server wiring
src/http.ts - MCP server · Runs the MCP runtime over stdio or HTTP
src/server/mcpServer.ts - HTTP server · HTTP transport with liveness and MCP endpoint support
src/server/httpServer.ts - Doc tool · Bridges MCP requests to the extractor
src/server/tools/extractDocsContextTool.ts
Core
Document ingestion, understanding, and packing pipeline
- Context extractor · Coordinates fetch, parsing, sectioning, compaction, and safety
src/core/extractDocsContext.ts - Normalize URL · Cleans and prepares the target docs URL
src/core/normalizeUrl.ts - Fetch page · Loads the remote documentation page
src/core/fetchPage.ts - Clean HTML · Removes noise and prepares markup for parsing
src/core/cleanHtml.ts - To markdown · Converts cleaned HTML into markdown text
src/core/htmlToMarkdown.ts - Doc sections · Finds meaningful documentation structure
src/core/detectDocSections.ts - Sanitize text · Flags or removes prompt-injection style content
src/core/sanitizePromptInjection.ts - Context pack · Builds compact implementation-oriented output
src/core/generateContextPack.ts - Compact context · Shrinks guidance for coding use
src/core/compactImplementationContext.ts - Token savings · Estimates compression and size reduction
src/core/estimateTokenSavings.ts - Output schema · Defines the structured extraction contract
src/schemas/extractDocsContext.schema.ts - Docflow types · Common domain types for extraction results
src/types/docflow.ts
Quality
Evaluation and test coverage
- AB eval · Compares extraction quality against baseline
src/eval/runAbEval.ts - Workflow eval · Runs workflow-oriented quality checks
src/eval/runWorkflowEval.ts
Connections 21
- CLI entry → MCP server: starts · Bootstraps the MCP runtime
- HTTP entry → HTTP server: starts · Bootstraps the HTTP server
- MCP server → Doc tool: registers · Exposes the extraction tool to MCP clients
- HTTP server → Doc tool: routes to · Forwards HTTP MCP requests to the tool
- Doc tool → Context extractor: calls · Invokes the core extraction pipeline
- Context extractor → Normalize URL: prepares url · Normalizes the requested documentation URL
- Context extractor → Fetch page: loads · Fetches remote documentation content
- Fetch page → Clean HTML: passes content · Sends fetched HTML into cleanup
- Clean HTML → To markdown: transforms · Converts cleaned markup to markdown
- To markdown → Doc sections: analyzes · Detects document structure and sections
- Doc sections → Sanitize text: filters · Removes suspicious or misleading text
- Sanitize text → Context pack: feeds · Provides trusted document content to packaging
- Context pack → Compact context: compresses · Creates a compact implementation context
- Compact context → Token savings: measures · Calculates token and size reduction
- Context pack → Output schema: validates · Shapes output to the response contract
- Context pack → Docflow types: uses · Shares domain types for the result payload
- Doc tool → Output schema: returns · Emits structured extraction output
- AB eval → Context extractor: benchmarks · Evaluates extractor quality against baseline
- Workflow eval → Context extractor: benchmarks · Evaluates workflow extraction behavior
- Examples → Doc tool: demonstrates · Shows expected MCP usage and sample outputs
- Docs site → Doc tool: documents · Explains connector and runtime usage