# mendableai/firecrawl-mcp-server architecture

> Architecture diagram of the GitHub repository mendableai/firecrawl-mcp-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mendableai/firecrawl-mcp-server
- Repository: https://github.com/mendableai/firecrawl-mcp-server

## Overview

##### Main Components
- MCP Server Core: Represent the central module (src/index.ts) that initializes the server and ties together all the functionalities.
- Command Dispatcher / Tool Handlers: Each supported command (scrape, batch scrape, search, crawl, extract, deep research, generate llms.txt, etc.) should be depicted as individual components, representing the encapsulated business logic for handling those distinct operations.
- Configuration Manager: A module that loads configuration settings (see the extensive environment variable configurations and default constants—retry behavior, API key/url, credit thresholds) essential for operation.
- Logging System: A dedicated logging component responsible for tracking operation status, errors, rate limit events, and credit usage.
- Retry & Rate Limiting Module: Highlight the mechanism that automatically handles failed requests with exponential backoff in response to rate limits.
- External API Integration: Include the external Firecrawl API (or self-hosted instance endpoint) that the server communicates with for its scraping and extraction work.
- Testing and Linting Setup: Although not a runtime component, briefly note the test harness (index.test.ts, jest configurations) and linting tools (eslint, prettier) as peripheral elements that ensure code quality.

##### Out Relationships and Interactions
- The MCP Server Core initializes and wires up all submodules.
- When a tool command (for example, “firecrawl_scrape” or “firecrawl_batch_scrape”) is invoked via a command-line call (npx), the flow is routed through the command dispatcher, which then interacts with the Configuration Manager for required settings.
- The tool handlers call the External API Integration module to perform web scraping, crawling, or extraction.
- Before and after all requests, the Retry & Rate Limiting Module intercepts responses to handle rate limits and perform retries as needed.
- The Logging System records every activity, including API calls, responses, status updates, and errors.
- There is a feedback loop where credit monitoring and error handling are both checked and logged.

##### Architectural Patterns & Design Principles
- Modular Design: Each feature (scraping, crawling, extraction) is implemented as a distinct module (or command) that interacts with shared core services (configuration, logging, retrying).
- Environment-based Configuration: Emphasize that all runtime behavior (such as API keys, retry limits, and credit usage thresholds) is determined by environment variables—a common 12-factor application practice.
- Error Handling with Retries: The automatic retry mechanism using exponential backoff is a critical design element.

##### Final Recommendations
- A detailed breakdown: Do not simplify too much by merely grouping the entire system into “frontend” and “backend”. Instead, delineate each command and core service.
- Both synchronous (command dispatching, local configuration) and asynchronous operations (API calls with retries, handling asynchronous web scraping).
- Dependency on environment settings that control retry parameters, rate limiting, and credit monitoring.

Happy diagramming!

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["CLI/Entry Point (npx -y firecrawl-mcp)"]:::entry
    B["MCP Server Core"]:::core
    C["Configuration Manager"]:::config
    D1["firecrawl_scrape Handler"]:::handler
    D2["firecrawl_batch_scrape Handler"]:::handler
    D3["check_batch_status Handler"]:::handler
    D4["search Handler"]:::handler
    D5["crawl Handler"]:::handler
    D6["extract Handler"]:::handler
    D7["deep_research Handler"]:::handler
    D8["generate_llms.txt Handler"]:::handler
    E["Retry & Rate Limiting"]:::retry
    F["External Firecrawl API"]:::external
    G["Logging System"]:::logging

    subgraph "Quality Assurance"
        subgraph "Testing Setup"
            T1["index.test.ts"]:::qa
            T2["jest.config.js"]:::qa
            T3["jest.setup.ts"]:::qa
        end
        subgraph "Linting Tools"
            L1[".eslintrc.json"]:::qa
            L2[".prettierrc"]:::qa
        end
    end

    %% Connections
    A -->|"calls"| B
    B -->|"loads"| C
    B -->|"dispatches"| D1
    B -->|"dispatches"| D2
    B -->|"dispatches"| D3
    B -->|"dispatches"| D4
    B -->|"dispatches"| D5
    B -->|"dispatches"| D6
    B -->|"dispatches"| D7
    B -->|"dispatches"| D8

    D1 -->|"invokes"| E
    D2 -->|"invokes"| E
    D3 -->|"invokes"| E
    D4 -->|"invokes"| E
    D5 -->|"invokes"| E
    D6 -->|"invokes"| E
    D7 -->|"invokes"| E
    D8 -->|"invokes"| E

    E -->|"callsAPI"| F

    %% Logging connections
    B -->|"logs"| G
    C -->|"logs"| G
    D1 -->|"logs"| G
    D2 -->|"logs"| G
    D3 -->|"logs"| G
    D4 -->|"logs"| G
    D5 -->|"logs"| G
    D6 -->|"logs"| G
    D7 -->|"logs"| G
    D8 -->|"logs"| G
    E -->|"logs"| G
    F -->|"logs"| G

    %% Click Events
    click B "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click C "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D1 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D2 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D3 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D4 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D5 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D6 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D7 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click D8 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.ts"
    click T1 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/src/index.test.ts"
    click T2 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/jest.config.js"
    click T3 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/jest.setup.ts"
    click L1 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/.eslintrc.json"
    click L2 "https://github.com/mendableai/firecrawl-mcp-server/blob/main/.prettierrc"

    %% Styles
    classDef entry fill:#f4f142,stroke:#000,stroke-width:2px;
    classDef core fill:#42f456,stroke:#000,stroke-width:2px;
    classDef config fill:#42d1f4,stroke:#000,stroke-width:2px;
    classDef handler fill:#f44174,stroke:#000,stroke-width:2px;
    classDef retry fill:#f49741,stroke:#000,stroke-width:2px;
    classDef external fill:#41a2f4,stroke:#000,stroke-width:2px;
    classDef logging fill:#a241f4,stroke:#000,stroke-width:2px;
    classDef qa fill:#8a8a8a,stroke:#000,stroke-width:2px;
```
