How ryoppippi/sitemcp works
Main Components
• Command-Line Interface (CLI): The entry point is in src/cli.ts, where command-line arguments (such as the target site URL, concurrency level, match patterns, and content selectors) are parsed.
• Site Fetcher Module: Located in src/fetch-site.ts, this component is responsible for retrieving the web pages. It likely handles the logic for recursively fetching pages from the given site.
• Content Processing: The src/to-markdown.ts module suggests that, once pages are fetched, their content is converted into Markdown format (possibly after processing via libraries like mozilla/readability).
• Server Module: Found in src/server.ts, this component sets up an MCP server, allowing the fetched and processed pages to be served, integrated later with an MCP client configuration.
• Logging and Utilities: The src/logger.ts and src/utils.ts files indicate additional support for logging actions and common utility functions, while src/types.ts describes project-specific TypeScript types.
• Configuration and Build: The supporting configuration files (e.g., package.json, tsconfig.json, biome.json, tsdown.config.ts) indicate a modern JavaScript/TypeScript toolchain and project setup.
Relationships and Interactions
• The CLI component acts as the orchestrator: It parses the arguments and then routes the workflow into either “fetch” mode or “server” mode.
• When a URL is provided, the CLI calls the Site Fetcher Module to crawl the site, using built-in options for concurrency and filtering (via matching patterns).
• The fetching process uses utilities and logging to track progress and handle errors.
• Once the HTML content is retrieved, the Content Processing module converts it to Markdown if requested.
• The Server module, when invoked, serves the fetched site content, enabling its integration via the MCP client (as seen in the README’s sample configuration).
Architectural Patterns and Design Principles
• Single Responsibility: Each module has a clear purpose—parsing, fetching, converting, and serving.
• Extensibility: The usage instructions in the README suggest that the tool can be tailored (through flags like --match, --content-selector, and concurrency settings) and integrated with external MCP clients.
• Dependency Flow: The CLI triggers the fetcher, fetcher uses utilities and logging, and then passes data to the Markdown converter; similarly, the Server module then exposes this data for MCP clients.