How coleam00/mcp-crawl4ai-rag works
Project type and purpose
• This is a specialized microservice (“MCP Server”) that offers web‐crawling and Retrieval‐Augmented‐Generation (RAG) capabilities to AI agents and assistants.
• It implements the Model Context Protocol (MCP) and integrates:
– Crawl4AI crawling logic
– Supabase (Postgres + pgvector) for vector storage and search
– OpenAI for embedding generation
– Optional Neo4j knowledge‐graph tools for hallucination detection
File structure
• Top‐level files and dirs
– Dockerfile / .dockerignore / .env.example – containerization and configuration
– pyproject.toml – Python packaging
– src/ – core MCP server entrypoint and helper utils
– knowledge_graphs/ – standalone scripts for repo parsing, AST analysis, validation and Cypher queries
– crawled_pages.sql – Supabase schema (pgvector extension)
• Patterns & architectural hints
– “src/crawl4ai_mcp.py” is the MCP application (registers @mcp.tool functions)
– “utils.py” holds shared crawling, chunking, embedding, and database‐client code
– “knowledge_graphs/” implements a plugin‐style submodule that can be toggled with env flag
README for additional insights
• Describes five RAG strategies (contextual embeddings, hybrid search, agentic RAG, reranking, knowledge graph) each enabled by an env var
• Details external dependencies: OpenAI API, Supabase, Neo4j
• Shows how clients connect (SSE, stdio) and register this server in their MCP config