# coleam00/mcp-crawl4ai-rag architecture

> Architecture diagram of the GitHub repository coleam00/mcp-crawl4ai-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/coleam00/mcp-crawl4ai-rag
- Repository: https://github.com/coleam00/mcp-crawl4ai-rag

## Overview

##### Project type and purpose
- This is a specialized microservice (“MCP Server”) that offers web‐crawling and Retrieval‐Augmented‐Generation (RAG) capabilities to AI agents and assistants.
- It implements the Model Context Protocol (MCP) and integrates:
– Crawl4AI crawling logic
– Supabase (Postgres + pgvector) for vector storage and search
– OpenAI for embedding generation
– Optional Neo4j knowledge‐graph tools for hallucination detection

##### File structure
- Top‐level files and dirs
– Dockerfile / .dockerignore / .env.example – containerization and configuration
– pyproject.toml – Python packaging
– src/ – core MCP server entrypoint and helper utils
– knowledge_graphs/ – standalone scripts for repo parsing, AST analysis, validation and Cypher queries
– crawled_pages.sql – Supabase schema (pgvector extension)
- Patterns & architectural hints
– “src/crawl4ai_mcp.py” is the MCP application (registers @mcp.tool functions)
– “utils.py” holds shared crawling, chunking, embedding, and database‐client code
– “knowledge_graphs/” implements a plugin‐style submodule that can be toggled with env flag

##### README for additional insights
- Describes five RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) each enabled by an env var
- Details external dependencies: OpenAI API, Supabase, Neo4j
- Shows how clients connect (SSE, stdio) and register this server in their MCP config

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Container & Configuration"
        Dockerfile["Dockerfile"]:::core
        EnvIgnore[".dockerignore"]:::core
        EnvExample[".env.example"]:::core
        pyproject["pyproject.toml"]:::core
    end

    subgraph "Crawl4AI RAG MCP Server"
        Transport["Transport & Tool Registry"]:::core
        CrawlIndex["Crawl & Index Engine"]:::core
        RAGQuery["RAG Query Engine"]:::core
    end

    subgraph "Embedding & Vector Store"
        OpenAI["OpenAI Embedding API"]:::externalAI
        Supabase["Supabase Vector Store"]:::persistence
    end

    subgraph "Knowledge Graph Service (optional)"
        Parse["parse_repo_into_neo4j"]:::optional
        Analyzer["ai_script_analyzer"]:::optional
        Validator["knowledge_graph_validator"]:::optional
        Reporter["hallucination_reporter"]:::optional
        QueryKG["query_knowledge_graph"]:::optional
        Detector["ai_hallucination_detector"]:::optional
        Neo4j["Neo4j Bolt Endpoint"]:::persistence
    end

    Client["MCP Client (AI Agent)"]:::external

    Client -->|Calls| Transport
    Transport -->|Crawl Tool| CrawlIndex
    Transport -->|RAG Tool| RAGQuery
    Transport -->|KG Tool| Parse
    CrawlIndex -->|Embed| OpenAI
    OpenAI -->|Embeddings| Supabase
    RAGQuery -->|Search| Supabase
    Parse -->|Ingest| Neo4j
    Analyzer -->|Analyze| Neo4j
    Validator -->|Validate| Neo4j
    Reporter -->|Report| Neo4j
    QueryKG -->|Query| Neo4j

    classDef core fill:#D6EAF8,stroke:#1B4F72
    classDef persistence fill:#D5F5E3,stroke:#145A32
    classDef externalAI fill:#FCF3CF,stroke:#B7950B
    classDef external fill:#F0F0F0,stroke:#888888
    classDef optional fill:#E5E7E9,stroke:#A6ACAF

    click Transport "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/src/crawl4ai_mcp.py"
    click CrawlIndex "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/src/utils.py"
    click RAGQuery "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/src/utils.py"
    click Supabase "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/crawled_pages.sql"
    click Parse "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/knowledge_graphs/parse_repo_into_neo4j.py"
    click Analyzer "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/knowledge_graphs/ai_script_analyzer.py"
    click Validator "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/knowledge_graphs/knowledge_graph_validator.py"
    click Reporter "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/knowledge_graphs/hallucination_reporter.py"
    click QueryKG "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/knowledge_graphs/query_knowledge_graph.py"
    click Detector "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/knowledge_graphs/ai_hallucination_detector.py"
    click Dockerfile "https://github.com/coleam00/mcp-crawl4ai-rag/tree/main/Dockerfile"
    click EnvIgnore "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/.dockerignore"
    click EnvExample "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/.env.example"
    click pyproject "https://github.com/coleam00/mcp-crawl4ai-rag/blob/main/pyproject.toml"
```
