How ratacat/slurp-ai works
Project type and purpose
• SlurpAI is a Node.js command-line application for web and NPM package documentation scraping, transformation into Markdown, and compilation into a single output file.
• It also includes an optional MCP (Model Context Protocol) server component to serve the compiled documentation.
File structure
• bin/slurp – CLI entry point script
• index.js – primary dispatcher for CLI commands
• mcp-server.js – HTTP server for serving MCP-compatible payloads
• config.js – central configuration for scraping, filtering, compilation, and file paths
• src/
– DocumentationScraper.js – handles HTTP requests, HTML parsing, navigation removal
– MarkdownCompiler.js – merges partial Markdown files into one document
– slurpWorkflow.js – orchestrates scraper + compiler per run
– cli.js – translates CLI args into workflow invocations
– utils/ – reusable helpers (logger, path normalization, markdown utilities)
• slurps_partials/ and slurps/ (default FS output directories)
• __tests__/ – unit and integration tests for each module and CLI
• package.json / build / publish workflow – NPM packaging and GitHub Actions for release
README for additional insights
• Core features: URL scraping, NPM docs fetching, Turndown-powered HTML→Markdown conversion, cleanup of navigation elements, asynchronous concurrency controls, and final compilation.
• Configuration sections define how components interact with external sites, file system, and concurrency policies.
• MCP server is in testing and uses the same compiled docs as its data source.
– CLI Interface (bin/slurp + cli.js)
– Configuration Module (config.js)
– Workflow Orchestrator (slurpWorkflow.js)
– Documentation Scraper (DocumentationScraper.js)
– Markdown Compiler (MarkdownCompiler.js)
– File System (temp partials + final slurps directory)
– MCP Server (mcp-server.js)
– External Dependencies (Websites, NPM Registry)
1. User input → CLI Interface
2. CLI reads Config → invokes Workflow Orchestrator
3. Workflow → spawns Documentation Scraper tasks → makes HTTP requests to Websites/NPM Registry → writes partial Markdown files to FS
4. Workflow → triggers Markdown Compiler → reads partials from FS → writes final Markdown to FS
5. (Optional) MCP Server reads final Markdown and exposes an HTTP API
c. Architectural patterns / design principles:
– Modular separation of concerns (Scraper vs. Compiler vs. Orchestrator)
– Asynchronous concurrency (configured pool of scraper promises)
– Pluggable filters via config.js (basePath enforcement, query-param rules)
d. Key technologies and libraries:
– Node.js v20+ runtime
– Turndown for HTML→Markdown
– Native fs/promises and HTTP/HTTPS modules (or optionally headless browser)
– Commander or yargs for CLI parsing (if used inside cli.js)
– Standard test framework (Vitest or Jest) for __tests__