# weidwonder/crawl4ai-mcp-server architecture

> Architecture diagram of the GitHub repository weidwonder/crawl4ai-mcp-server, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/weidwonder/crawl4ai-mcp-server
- Repository: https://github.com/weidwonder/crawl4ai-mcp-server

## Overview

##### Project Overview and Purpose
- This is a Python-based server project designed to provide enhanced search capabilities and intelligent web content extraction to support LLM (Large Language Model) applications. It supports multi-engine search (DuckDuckGo as default and Google when API credentials are provided) and includes functionality to optimize webpage content for LLM processing. Notably, the README mentions the use of FastMCP for high-performance asynchronous operations.

##### Key Components
a. Core Server Component:
- Main server process (src/index.py) that orchestrates incoming requests, coordinates the search and content extraction functionalities, and returns optimized results.

b. Search Module:
- “Search Engine Module” implemented in src/search.py.
- That this module encapsulates multi-engine support: using DuckDuckGo (default) and Google (conditional on API keys in the configuration).

c. Content Extraction / LLM Optimization:
- Although implemented along with the search functionality, the README explains a separate use-case endpoint (read_url) that provides several content customization formats (e.g., markdown_with_citations, fit_markdown).
- A “Content Extraction & Optimization” component that is part of the server’s logic.
- This component can be shown as processing the raw data fetched from web pages and reformatting/filtering it appropriately.

d. Configuration Management:
- Presence of configuration files (config_demo.json, and the instructions to copy this to config.json). This file contains external service credentials (e.g., API keys for Google).

e. External Services and Dependencies:
- Mention the asynchronous design using FastMCP, which impacts the overall communication and performance model. You might include a note or a sub-label indicating “Async Message Handling” or “Asynchronous I/O” to capture this design decision.
- Optionally, show dependencies like the “playwright” tooling (as required for browser interactions if applicable) and containerization (Dockerfile shows the container deployment).

f. Testing and Build/Deployment:
- A “Testing” section representing the test files (test_search.py and test_search_manager.py) used for ensuring functionality. It’s helpful to illustrate that the system has a dedicated testing suite.

##### Relationships and Data Flow
- Client Requests → MCP Server Core.
- MCP Server Core → Search Module & Content Extraction Module.
- Content Extraction taps into fetched webpage data (from external web requests) to produce multiple output formats.
- Configuration Management flows into these components to set parameters like API keys.

##### Architectural Patterns & Design Principles
- Asynchronous design based on FastMCP for handling concurrent operations.
- Clearly separate modules (controller, search, content extraction) which is indicative of a modular design.
- Containerization with Docker suggests the deployment is environment-independent.
- Dependency injection or configuration management aspects that allow external API keys and other parameters to be managed externally.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Global Entities
    Client["Client Requests"]:::external

    %% Core Layer
    mcp["MCP Server Core (Controller)"]:::internal
    config["Configuration Service"]:::config
    search["Search Engine Module"]:::internal
    content["Content Extraction & Optimization"]:::internal

    %% External Services
    ddg["DuckDuckGo"]:::external
    google["Google"]:::external

    %% Testing Suite
    subgraph "Testing Suite"
        tst1["test_search.py"]:::test
        tst2["test_search_manager.py"]:::test
    end

    %% Deployment & Build Config
    subgraph "Deployment & Build Config"
        deploy1["Dockerfile"]:::deploy
        deploy2["pyproject.toml"]:::deploy
        deploy3["requirements.txt"]:::deploy
        deploy4["smithery.yaml"]:::deploy
    end

    %% Connections - Client to Core
    Client -->|"request"| mcp

    %% Core Routing
    mcp -->|"searchRequest Async"| search
    mcp -->|"contentRequest"| content

    %% Configuration Injection
    config -->|"injectConfig"| search

    %% Search Module to External Services
    search -->|"query Async"| ddg
    search -->|"query Async"| google
    ddg -->|"results"| search
    google -->|"results"| search

    %% Returning Results to Core
    search -->|"searchResults"| mcp
    content -->|"optimizedContent"| mcp

    %% Asynchronous Note
    %% (Async operations handled via FastMCP)

    %% Click Events
    click mcp "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/src/index.py"
    click search "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/src/search.py"
    click config "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/config_demo.json"
    click tst1 "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/test_search.py"
    click tst2 "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/test_search_manager.py"
    click deploy1 "https://github.com/weidwonder/crawl4ai-mcp-server/tree/main/Dockerfile"
    click deploy2 "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/pyproject.toml"
    click deploy3 "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/requirements.txt"
    click deploy4 "https://github.com/weidwonder/crawl4ai-mcp-server/blob/main/smithery.yaml"

    %% Styles
    classDef internal fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef external fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px;
    classDef config fill:#FFF9C4,stroke:#FBC02D,stroke-width:2px;
    classDef test fill:#E1BEE7,stroke:#8E24AA,stroke-width:2px;
    classDef deploy fill:#FFCDD2,stroke:#C62828,stroke-width:2px;
```
