How appaidsc/conflux works
1. What this repository is
This is a Python-based MCP server for Confluence knowledge access. It has two main operating paths: an offline ingestion/indexing pipeline that builds the search corpus, and an online query layer that serves AI agents and IDEs over MCP using either SSE or stdio transport.
2. Primary system boundary
The repo sits between Confluence as the source system and agent clients as the consumers. Confluence content is pulled in, normalized, embedded, and stored; later, agent queries come back through MCP tools and are answered from retrieved context rather than directly from Confluence.
3. Ingestion pipeline
The mcp_server/pipeline/ package is the content-processing backbone. The flow is: discovery → fetch → parse → analyze → importance scoring → summarization → embedding → indexing. The run_pipeline.py entrypoint suggests batch or full reindex runs, while background_worker.py supports asynchronous/progressive processing.
4. Content normalization and enrichment
The pipeline is more than raw import. parser.py converts Confluence HTML into a more usable text form, analyzer.py and importance_scorer.py add metadata and ranking signals, and summarizer.py produces compact page summaries. This means indexing stores both the original page representation and derived retrieval aids.
5. Retrieval storage layer
The architecture centers on Qdrant as the search index. The README and model cache indicate a hybrid dense + sparse retrieval design: dense embeddings for semantic matching and sparse embeddings for lexical precision. The local models_cache/ directory shows the embedding/ranking models are shipped or cached locally, supporting offline inference.
6. Search and ranking services
mcp_server/services/search_service.py is the main retrieval boundary. It likely orchestrates query embedding, hybrid search, score fusion, and reranking. The repository emphasizes Reciprocal Rank Fusion and a cross-encoder rerank step, so search is not a single vector lookup but a multi-stage ranking pipeline.
7. RAG answer generation
mcp_server/services/rag_service.py sits above retrieval and turns search results into grounded answers. It supports the repo’s three modes: extractive, ai, and hybrid. grounding_validator.py is the hallucination-control boundary, checking that generated claims are supported by retrieved source content.
8. LLM provider abstraction
mcp_server/llm/ isolates model backends behind a common interface. There are concrete providers for OpenAI and LM Studio, which means generation can run against cloud or local models without changing the rest of the system. This provider boundary is important for deployment flexibility and offline-capable operation.
9. MCP tool surface
mcp_server/tools/ defines the externally visible capability set for agents. The tools cover page retrieval, page tree traversal, section extraction, related-page lookup, search, question answering, code search, image search, and image analysis. Architecturally, these are thin action endpoints over the services layer rather than standalone business logic.
10. Core data contracts
mcp_server/schemas/ defines the internal API shapes for pages, search results, and RAG responses. These schemas form the contract between ingestion, retrieval, and tool exposure, so they are the main place where payload boundaries are stabilized.
11. Configuration and deployment
mcp_server/config.py and .env.example show the system is environment-driven. Key runtime dependencies are Confluence credentials, Qdrant host/port, and the selected LLM provider. docker-compose.yml is used for local infrastructure, especially Qdrant, while the MCP server itself can be launched in SSE or stdio mode.
12. Testing and documentation
tests/test_pipeline.py and tests/test_services.py indicate coverage is centered on the two critical paths: indexing and query/answer services. The docs/ folder contains architecture and deep-dive material, which likely documents the intended data flow, schema, and operational choices in more detail than the code comments.
13. How to read the architecture diagram
A correct diagram should show a source ingestion lane from Confluence into pipeline stages, then into Qdrant, then a query lane from MCP tools into search/RAG services, then into an LLM provider, with grounding validation before the response returns to the agent. The clearest subsystem boundaries are: source integration, pipeline/indexing, retrieval/ranking, generation/validation, and MCP serving.
Main entry points: search_service.py Search · rag_service.py RAG · tools Tools · base_provider.py LLM Provider
Components 23
Source & Runtime
External source system plus repo-level runtime/configuration
- Confluence · Upstream knowledge source ingested into the repository's corpus
- Config · Environment-driven settings for credentials, transport, and backend selection
mcp_server/config.py
Pipeline & Indexing
Offline ingestion, normalization, embedding, and indexing flow
- Pipeline Run · Entry point for full or scheduled indexing runs
mcp_server/pipeline/run_pipeline.py - Worker · Background processing for progressive or asynchronous ingestion
mcp_server/pipeline/background_worker.py - Discovery · Finds pages and content units to ingest
mcp_server/pipeline/discovery.py - Fetcher · Pulls Confluence page content for processing
mcp_server/pipeline/fetcher.py - Parser · Converts Confluence HTML into normalized text and structure
mcp_server/pipeline/parser.py - Enrichment · Adds metadata, importance signals, and summaries
- Embedder · Produces dense and sparse representations for retrieval
mcp_server/pipeline/embedder.py - Indexer · Writes processed content and vectors into the search store
mcp_server/pipeline/indexer.py
Retrieval & RAG
Online search, ranking, generation, and validation services
- Search · Orchestrates query embedding, hybrid search, fusion, and reranking
mcp_server/services/search_service.py - Cache · Caches expensive retrieval or lookup results
mcp_server/services/cache_service.py - RAG · Turns retrieved context into grounded answers
mcp_server/services/rag_service.py - Validator · Checks generated claims against retrieved source evidence
mcp_server/services/grounding_validator.py - LLM Provider · Shared abstraction for generation backends
mcp_server/llm/base_provider.py - OpenAI · Remote generation backend implementation
mcp_server/llm/openai_provider.py - LM Studio · Local generation backend implementation
mcp_server/llm/lm_studio_provider.py
MCP Serving
MCP transport and tool surface for agent clients
- MCP Server · Serves the repository over SSE or stdio transports
mcp_server/server.py - Tools · Externally visible agent actions over pages, search, images, and code
mcp_server/tools - Schemas · Shared page, search, and RAG payload definitions
mcp_server/schemas - Agent Clients · AI agents and IDEs consuming MCP tools
Storage & Models
Persistent search store and local model artifacts
- Qdrant · Hybrid search index used for retrieval
- Model Cache · Cached ONNX embedding and reranking models for offline-capable inference
models_cache
Connections 25
- Config → Pipeline Run: drives · Supplies runtime settings for ingestion runs
- Config → MCP Server: configures · Selects transport and backend behavior
- Pipeline Run → Discovery: starts · Launches the ingestion pipeline
- Worker → Fetcher: processes · Handles progressive fetch and enrichment work
- Discovery → Fetcher: feeds · Provides discovered page targets
- Fetcher → Parser: passes · Sends raw Confluence content for normalization
- Parser → Enrichment: normalizes · Produces structured text for downstream analysis
- Enrichment → Embedder: enriches · Adds metadata, scores, and summaries before embedding
- Embedder → Indexer: outputs · Hands off vectorized documents for indexing
- Indexer → Qdrant: writes · Stores hybrid retrieval records and vectors
- Model Cache → Embedder: backs · Provides local embedding models
- Model Cache → Search: backs · Provides local reranking or query models
- Tools → Search: queries · Search tools call into retrieval services
- Tools → RAG: asks · Question-answering tools call the RAG layer
- Tools → MCP Server: registered by · Tool definitions are exposed through the MCP host
- MCP Server → Agent Clients: serves · Streams tool access to MCP consumers
- Search → Qdrant: retrieves from · Executes hybrid retrieval against the index
- Search → Cache: uses · Reuses cached search and lookup results
- Search → Schemas: returns · Shapes search results with shared contracts
- RAG → Search: depends on · Uses retrieved context as grounding input
- RAG → LLM Provider: generates via · Delegates generation to a backend provider
- RAG → Validator: checks with · Validates answers against evidence
- RAG → Schemas: returns · Shapes RAG responses with shared contracts
- LLM Provider → OpenAI: implemented by · Cloud generation backend
- LLM Provider → LM Studio: implemented by · Local generation backend