# ratacat/slurp-ai architecture

> Architecture diagram of the GitHub repository ratacat/slurp-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ratacat/slurp-ai
- Repository: https://github.com/ratacat/slurp-ai

## Overview

##### Project type and purpose
- SlurpAI is a Node.js command-line application for web and NPM package documentation scraping, transformation into Markdown, and compilation into a single output file.
- It also includes an optional MCP (Model Context Protocol) server component to serve the compiled documentation.

##### File structure
- bin/slurp – CLI entry point script
- index.js – primary dispatcher for CLI commands
- mcp-server.js – HTTP server for serving MCP-compatible payloads
- config.js – central configuration for scraping, filtering, compilation, and file paths
- src/
– DocumentationScraper.js – handles HTTP requests, HTML parsing, navigation removal
– MarkdownCompiler.js – merges partial Markdown files into one document
– slurpWorkflow.js – orchestrates scraper + compiler per run
– cli.js – translates CLI args into workflow invocations
– utils/ – reusable helpers (logger, path normalization, markdown utilities)
- slurps_partials/ and slurps/ (default FS output directories)
- __tests__/ – unit and integration tests for each module and CLI
- package.json / build / publish workflow – NPM packaging and GitHub Actions for release

##### README for additional insights
- Core features: URL scraping, NPM docs fetching, Turndown-powered HTML→Markdown conversion, cleanup of navigation elements, asynchronous concurrency controls, and final compilation.
- Configuration sections define how components interact with external sites, file system, and concurrency policies.
- MCP server is in testing and uses the same compiled docs as its data source.

– CLI Interface (bin/slurp + cli.js)
– Configuration Module (config.js)
– Workflow Orchestrator (slurpWorkflow.js)
– Documentation Scraper (DocumentationScraper.js)
– Markdown Compiler (MarkdownCompiler.js)
– File System (temp partials + final slurps directory)
– MCP Server (mcp-server.js)
– External Dependencies (Websites, NPM Registry)
1. User input → CLI Interface
2. CLI reads Config → invokes Workflow Orchestrator
3. Workflow → spawns Documentation Scraper tasks → makes HTTP requests to Websites/NPM Registry → writes partial Markdown files to FS
4. Workflow → triggers Markdown Compiler → reads partials from FS → writes final Markdown to FS
5. (Optional) MCP Server reads final Markdown and exposes an HTTP API
c. Architectural patterns / design principles:
– Modular separation of concerns (Scraper vs. Compiler vs. Orchestrator)
– Asynchronous concurrency (configured pool of scraper promises)
– Pluggable filters via config.js (basePath enforcement, query-param rules)
d. Key technologies and libraries:
– Node.js v20+ runtime
– Turndown for HTML→Markdown
– Native fs/promises and HTTP/HTTPS modules (or optionally headless browser)
– Commander or yargs for CLI parsing (if used inside cli.js)
– Standard test framework (Vitest or Jest) for __tests__

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Actors and External
    User(["User"]):::external
    subgraph "External Dependencies"
        Websites(["Websites"]):::external
        NPM(["NPM Registry"]):::external
    end

    %% CLI Layer
    User --> CLI["CLI Interface (bin/slurp)"]:::cli
    click CLI "https://github.com/ratacat/slurp-ai/tree/main/bin/slurp"
    CLI --> CLIProc["CLI Argument Processor (src/cli.js)"]:::cli
    click CLIProc "https://github.com/ratacat/slurp-ai/blob/main/src/cli.js"
    CLIProc --> Dispatcher["Primary Dispatcher (index.js)"]:::core
    click Dispatcher "https://github.com/ratacat/slurp-ai/blob/main/index.js"
    Dispatcher --> Config["Configuration Module (config.js)"]:::infra
    click Config "https://github.com/ratacat/slurp-ai/blob/main/config.js"

    %% Orchestration and Core Modules
    Dispatcher --> Orchestrator["Workflow Orchestrator (src/slurpWorkflow.js)"]:::core
    click Orchestrator "https://github.com/ratacat/slurp-ai/blob/main/src/slurpWorkflow.js"
    subgraph "Core Modules"
        Orchestrator
        Scraper["Documentation Scraper (src/DocumentationScraper.js)"]:::core
        click Scraper "https://github.com/ratacat/slurp-ai/blob/main/src/DocumentationScraper.js"
        Compiler["Markdown Compiler (src/MarkdownCompiler.js)"]:::core
        click Compiler "https://github.com/ratacat/slurp-ai/blob/main/src/MarkdownCompiler.js"
    end

    %% Utilities
    subgraph "Utilities"
        Logger["Logger (src/utils/logger.js)"]:::infra
        click Logger "https://github.com/ratacat/slurp-ai/blob/main/src/utils/logger.js"
        PathUtils["Path Utils (src/utils/pathUtils.js)"]:::infra
        click PathUtils "https://github.com/ratacat/slurp-ai/blob/main/src/utils/pathUtils.js"
        MDUtils["Markdown Utils (src/utils/markdownUtils.js)"]:::infra
        click MDUtils "https://github.com/ratacat/slurp-ai/blob/main/src/utils/markdownUtils.js"
    end

    %% File System
    Scraper -->|"writes partials"| FSPartial["File System (slurps_partials/)"]:::infra
    Orchestrator --> Scraper
    Orchestrator --> Compiler
    Compiler -->|"writes final docs"| FSFinal["Output File System (slurps/)"]:::infra
    click FSFinal "https://github.com/ratacat/slurp-ai/tree/main/slurps/"

    %% Data Flows
    Scraper --> Websites
    Scraper --> NPM
    Orchestrator --> Logger
    Orchestrator --> PathUtils
    Scraper --> PathUtils
    Compiler --> MDUtils

    %% MCP Server
    FSFinal --> MCP["MCP Server (mcp-server.js)"]:::server
    click MCP "https://github.com/ratacat/slurp-ai/blob/main/mcp-server.js"

    %% Testing
    subgraph "Testing"
        Tests["Unit & Integration Tests (__tests__)"]:::test
        click Tests "https://github.com/ratacat/slurp-ai/tree/main/__tests__/"
    end

    %% Styles
    classDef cli fill:#f9f,stroke:#333,stroke-width:1px
    classDef core fill:#bbf,stroke:#333,stroke-width:1px
    classDef infra fill:#bfb,stroke:#333,stroke-width:1px
    classDef server fill:#fbf,stroke:#333,stroke-width:1px
    classDef external fill:#ffd,stroke:#333,stroke-width:1px
    classDef test fill:#eee,stroke:#333,stroke-width:1px
```
