# ryoppippi/sitemcp architecture

> Architecture diagram of the GitHub repository ryoppippi/sitemcp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ryoppippi/sitemcp
- Repository: https://github.com/ryoppippi/sitemcp

## Overview

##### Main Components
- Command-Line Interface (CLI): The entry point is in src/cli.ts, where command-line arguments (such as the target site URL, concurrency level, match patterns, and content selectors) are parsed.
- Site Fetcher Module: Located in src/fetch-site.ts, this component is responsible for retrieving the web pages. It likely handles the logic for recursively fetching pages from the given site.
- Content Processing: The src/to-markdown.ts module suggests that, once pages are fetched, their content is converted into Markdown format (possibly after processing via libraries like mozilla/readability).
- Server Module: Found in src/server.ts, this component sets up an MCP server, allowing the fetched and processed pages to be served, integrated later with an MCP client configuration.
- Logging and Utilities: The src/logger.ts and src/utils.ts files indicate additional support for logging actions and common utility functions, while src/types.ts describes project-specific TypeScript types.
- Configuration and Build: The supporting configuration files (e.g., package.json, tsconfig.json, biome.json, tsdown.config.ts) indicate a modern JavaScript/TypeScript toolchain and project setup.

##### Relationships and Interactions
- The CLI component acts as the orchestrator: It parses the arguments and then routes the workflow into either “fetch” mode or “server” mode.
- When a URL is provided, the CLI calls the Site Fetcher Module to crawl the site, using built-in options for concurrency and filtering (via matching patterns).
- The fetching process uses utilities and logging to track progress and handle errors.
- Once the HTML content is retrieved, the Content Processing module converts it to Markdown if requested.
- The Server module, when invoked, serves the fetched site content, enabling its integration via the MCP client (as seen in the README’s sample configuration).

##### Architectural Patterns and Design Principles
- Single Responsibility: Each module has a clear purpose—parsing, fetching, converting, and serving.
- Extensibility: The usage instructions in the README suggest that the tool can be tailored (through flags like --match, --content-selector, and concurrency settings) and integrated with external MCP clients.
- Dependency Flow: The CLI triggers the fetcher, fetcher uses utilities and logging, and then passes data to the Markdown converter; similarly, the Server module then exposes this data for MCP clients.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Modules
    subgraph "Core Modules"
        cli["Command-Line Interface (CLI)"]:::core
        fetcher["Site Fetcher Module"]:::core
        markdown["Content Processing (Markdown Conversion)"]:::core
        server["Server Module (MCP Server)"]:::core
    end

    %% Supporting Modules
    subgraph "Supporting Modules"
        logger["Logging Module"]:::support
        utils["Utility Functions"]:::support
        types["Type Definitions"]:::support
    end

    %% Configuration & Build Setup
    subgraph "Configuration & Build Setup"
        pkg["package.json"]:::config
        tsconfig["tsconfig.json"]:::config
        biome["biome.json"]:::config
        tsdown["tsdown.config.ts"]:::config
        bun_lock["bun.lock"]:::config
    end

    %% Core Flow
    cli -->|"Parse_URL&Options"| fetcher
    fetcher -->|"Raw_HTML_Content"| markdown
    markdown -->|"Processed_Content"| server
    cli -->|"Serve_Mode"| server

    %% Supporting interactions
    cli -->|"uses"| logger
    cli -->|"uses"| utils
    cli -->|"defines"| types

    fetcher -->|"logs_to"| logger
    fetcher -->|"utilizes"| utils
    fetcher -->|"uses"| types

    markdown -->|"utilizes"| utils
    markdown -->|"uses"| types

    server -->|"relies_on"| types

    %% Configuration interactions
    cli ---|"reads"| pkg
    cli ---|"reads"| tsconfig
    cli ---|"reads"| biome
    cli ---|"reads"| tsdown
    cli ---|"reads"| bun_lock

    server ---|"reads"| pkg
    server ---|"reads"| tsconfig

    %% Styles
    classDef core fill:#bbf,stroke:#333,stroke-width:2px;
    classDef support fill:#cfc,stroke:#333,stroke-width:2px;
    classDef config fill:#fdd,stroke:#333,stroke-width:2px;

    %% Click Events
    click cli "https://github.com/ryoppippi/sitemcp/blob/main/src/cli.ts"
    click fetcher "https://github.com/ryoppippi/sitemcp/blob/main/src/fetch-site.ts"
    click markdown "https://github.com/ryoppippi/sitemcp/blob/main/src/to-markdown.ts"
    click server "https://github.com/ryoppippi/sitemcp/blob/main/src/server.ts"
    click logger "https://github.com/ryoppippi/sitemcp/blob/main/src/logger.ts"
    click utils "https://github.com/ryoppippi/sitemcp/blob/main/src/utils.ts"
    click types "https://github.com/ryoppippi/sitemcp/blob/main/src/types.ts"
    click pkg "https://github.com/ryoppippi/sitemcp/blob/main/package.json"
    click tsconfig "https://github.com/ryoppippi/sitemcp/blob/main/tsconfig.json"
    click biome "https://github.com/ryoppippi/sitemcp/blob/main/biome.json"
    click tsdown "https://github.com/ryoppippi/sitemcp/blob/main/tsdown.config.ts"
    click bun_lock "https://github.com/ryoppippi/sitemcp/blob/main/bun.lock"
```
