How pskill9/website-downloader works
Project type and purpose
• This is a single‐service “MCP Server” written in TypeScript/Node.js whose sole responsibility is to download an entire website via the external wget binary, preserve its structure, fix links for local browsing, and output the files to a local directory.
• It is neither a multi‐tier full‐stack app nor a compiler; rather it’s a small backend microservice/tool that orchestrates an external CLI (wget) and exposes a simple interface to an MCP orchestrator or CLI.
File structure
• Top level:
• package.json / package-lock.json – Node.js project manifest
• tsconfig.json – TypeScript compiler config
• .gitignore – ignored files
• README.md – project description, usage, prerequisites
• src/index.ts – single entry point where the server logic
lives; likely parses incoming JSON, spawns wget, streams output or returns status.
• No “frontend”, no “database”, no multi-module layout—this is a standalone backend service.
README for additional insights
• Describes a CLI-like interface called download_website with parameters (url, outputPath, depth).
• Requires system‐level dependency wget.
• Usage within an MCP configuration: the MCP orchestrator invokes node build/index.js with a JSON payload.
• No mention of HTTP; interaction might be via STDIN/STDOUT or MCP’s IPC.
Main components
a. MCP Orchestrator
b. Website Downloader Service (Node.js process running build/index.js)
c. External CLI Dependency (wget)
d. Local File System (output directory)
e. Client (could be CLI or internal MCP execution engine)
Relationships and interactions
1. Client or MCP Orchestrator → (invoke, pass JSON) → Node.js Service
2. Node.js Service → (spawn child process) → wget CLI
3. wget CLI → (fetch & write) → Local File System
4. Node.js Service ← (exit code / stdout/stderr) ← wget CLI
5. Node.js Service → (return status, logs) → Client / MCP Orchestrator
Architectural patterns and design principles
• Single‐responsibility microservice: one process → one job.
• Process orchestration: Node.js manages an external process.
• “Adapter” pattern: Node.js adapts MCP’s JSON API to wget’s command‐line interface.
• No persistent database—uses local file system as “storage”.
Key technologies and libraries
• Node.js & TypeScript
• child_process (Node.js) to spawn wget
• system’s wget binary (external)
• (Optional) any logging framework in index.ts