How egoist/fetch-mcp works
Project Type and Purpose
• The project is a command-line MCP server focused on fetching content. It primarily deals with two tasks:
• Fetching URLs (returning HTML or Markdown).
• Fetching YouTube video transcripts.
• The README indicates that it runs as either a stdio or SSE server, and the CLI is the entry point (via “npx -y fetch-mcp”).
File Structure
• The files are relatively flat, with key files including:
• cli.js: Likely the command-line interface module that handles execution and input parameters.
• index.ts: Probably the main entry point or application bootstrapping logic.
• markdown.ts: Likely responsible for handling or converting content to Markdown format.
• youtube.ts: Implements fetching YouTube transcripts.
• utils.ts: Contains common helper functions used by other modules.
• TypeScript configuration files (tsconfig.json, shims.d.ts) and package configuration files indicate a TypeScript project.
• There are no separate folders for frontend/backend, so the project is primarily a CLI/utility tool.
README for Additional Insights
• The README provides usage instructions for both standard input/output and SSE server modes.
• It explicitly lists “fetch_url” and “fetch_youtube_transcript” as the two core tools.
• The sponsorship by ChatWise suggests integration or compatibility with a larger ecosystem, though that aspect is handled externally.
a. Main Components:
• Command-Line Interface (CLI): Entry point for the application (cli.js).
• Application Core: Main processing unit (index.ts) that initializes and routes commands.
• URL Fetcher: Component that fetches URLs (logic likely split across index.ts, utils.ts, and markdown.ts).
• YouTube Transcript Fetcher: Dedicated module for retrieving YouTube transcripts (youtube.ts).
• Data Formatter/Parser: Optional but suggested by markdown.ts for converting or handling data in Markdown format.
• Server Modes: Represent the two operating modes (stdio and SSE) as separate system behaviors that might share common logic.
b. Relationships and Interactions:
• The CLI interfaces with the Application Core by parsing command-line arguments (including mode selection, e.g., --sse).
• The Application Core then directs the control flow:
• If the task is to fetch a URL, it calls into the URL Fetcher and Data Formatter to process and possibly convert HTML to Markdown.
• If the task involves YouTube, the core delegates to the YouTube Transcript Fetcher.
• Both fetchers may utilize shared utilities provided by the utils.ts module.
• Data flows from these components back to the CLI output or is pushed through SSE if that mode is activated.
c. Architectural Patterns and Design Principles:
• Separation of Concerns: Each module is responsible for a specific task (CLI, URL fetching, YouTube transcript fetching).
• Modular Design: Encourages extensions or modifications in individual components without affecting the entire system.
• Clear input-processing-output flow, with routing in the main index.ts.
• Optional Server Mode Pattern: Handling different modes (stdio vs. SSE) can be seen as a configuration-based branching within the architecture.
d. Relevant Technologies, Frameworks, or Libraries:
• TypeScript as evidenced by .ts files and tsconfig.json.
• Bun (bun.lock) might be used for package management or running the code.
• Node.js runtime (implied by CLI usage and JavaScript/TypeScript nature).