How mendableai/firecrawl-mcp-server works
Main Components
• MCP Server Core: Represent the central module (src/index.ts) that initializes the server and ties together all the functionalities.
• Command Dispatcher / Tool Handlers: Each supported command (scrape, batch scrape, search, crawl, extract, deep research, generate llms.txt, etc.) should be depicted as individual components, representing the encapsulated business logic for handling those distinct operations.
• Configuration Manager: A module that loads configuration settings (see the extensive environment variable configurations and default constants—retry behavior, API key/url, credit thresholds) essential for operation.
• Logging System: A dedicated logging component responsible for tracking operation status, errors, rate limit events, and credit usage.
• Retry & Rate Limiting Module: Highlight the mechanism that automatically handles failed requests with exponential backoff in response to rate limits.
• External API Integration: Include the external Firecrawl API (or self-hosted instance endpoint) that the server communicates with for its scraping and extraction work.
• Testing and Linting Setup: Although not a runtime component, briefly note the test harness (index.test.ts, jest configurations) and linting tools (eslint, prettier) as peripheral elements that ensure code quality.
Out Relationships and Interactions
• The MCP Server Core initializes and wires up all submodules.
• When a tool command (for example, “firecrawl_scrape” or “firecrawl_batch_scrape”) is invoked via a command-line call (npx), the flow is routed through the command dispatcher, which then interacts with the Configuration Manager for required settings.
• The tool handlers call the External API Integration module to perform web scraping, crawling, or extraction.
• Before and after all requests, the Retry & Rate Limiting Module intercepts responses to handle rate limits and perform retries as needed.
• The Logging System records every activity, including API calls, responses, status updates, and errors.
• There is a feedback loop where credit monitoring and error handling are both checked and logged.
Architectural Patterns & Design Principles
• Modular Design: Each feature (scraping, crawling, extraction) is implemented as a distinct module (or command) that interacts with shared core services (configuration, logging, retrying).
• Environment-based Configuration: Emphasize that all runtime behavior (such as API keys, retry limits, and credit usage thresholds) is determined by environment variables—a common 12-factor application practice.
• Error Handling with Retries: The automatic retry mechanism using exponential backoff is a critical design element.
Final Recommendations
• A detailed breakdown: Do not simplify too much by merely grouping the entire system into “frontend” and “backend”. Instead, delineate each command and core service.
• Both synchronous (command dispatching, local configuration) and asynchronous operations (API calls with retries, handling asynchronous web scraping).
• Dependency on environment settings that control retry parameters, rate limiting, and credit monitoring.
Happy diagramming!