# appaidsc/conflux architecture

> Architecture diagram of the GitHub repository appaidsc/conflux, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/appaidsc/conflux
- Repository: https://github.com/appaidsc/conflux

## Overview

1. **What this repository is**
   This is a Python-based **MCP server** for Confluence knowledge access. It has two main operating paths: an **offline ingestion/indexing pipeline** that builds the search corpus, and an **online query layer** that serves AI agents and IDEs over MCP using either **SSE** or **stdio** transport.

2. **Primary system boundary**
   The repo sits between **Confluence as the source system** and **agent clients** as the consumers. Confluence content is pulled in, normalized, embedded, and stored; later, agent queries come back through MCP tools and are answered from retrieved context rather than directly from Confluence.

3. **Ingestion pipeline**
   The `mcp_server/pipeline/` package is the content-processing backbone. The flow is: **discovery → fetch → parse → analyze → importance scoring → summarization → embedding → indexing**. The `run_pipeline.py` entrypoint suggests batch or full reindex runs, while `background_worker.py` supports asynchronous/progressive processing.

4. **Content normalization and enrichment**
   The pipeline is more than raw import. `parser.py` converts Confluence HTML into a more usable text form, `analyzer.py` and `importance_scorer.py` add metadata and ranking signals, and `summarizer.py` produces compact page summaries. This means indexing stores both the original page representation and derived retrieval aids.

5. **Retrieval storage layer**
   The architecture centers on **Qdrant** as the search index. The README and model cache indicate a **hybrid dense + sparse retrieval** design: dense embeddings for semantic matching and sparse embeddings for lexical precision. The local `models_cache/` directory shows the embedding/ranking models are shipped or cached locally, supporting offline inference.

6. **Search and ranking services**
   `mcp_server/services/search_service.py` is the main retrieval boundary. It likely orchestrates query embedding, hybrid search, score fusion, and reranking. The repository emphasizes **Reciprocal Rank Fusion** and a **cross-encoder rerank** step, so search is not a single vector lookup but a multi-stage ranking pipeline.

7. **RAG answer generation**
   `mcp_server/services/rag_service.py` sits above retrieval and turns search results into grounded answers. It supports the repo’s three modes: **extractive**, **ai**, and **hybrid**. `grounding_validator.py` is the hallucination-control boundary, checking that generated claims are supported by retrieved source content.

8. **LLM provider abstraction**
   `mcp_server/llm/` isolates model backends behind a common interface. There are concrete providers for **OpenAI** and **LM Studio**, which means generation can run against cloud or local models without changing the rest of the system. This provider boundary is important for deployment flexibility and offline-capable operation.

9. **MCP tool surface**
   `mcp_server/tools/` defines the externally visible capability set for agents. The tools cover page retrieval, page tree traversal, section extraction, related-page lookup, search, question answering, code search, image search, and image analysis. Architecturally, these are thin action endpoints over the services layer rather than standalone business logic.

10. **Core data contracts**
   `mcp_server/schemas/` defines the internal API shapes for pages, search results, and RAG responses. These schemas form the contract between ingestion, retrieval, and tool exposure, so they are the main place where payload boundaries are stabilized.

11. **Configuration and deployment**
   `mcp_server/config.py` and `.env.example` show the system is **environment-driven**. Key runtime dependencies are Confluence credentials, Qdrant host/port, and the selected LLM provider. `docker-compose.yml` is used for local infrastructure, especially Qdrant, while the MCP server itself can be launched in SSE or stdio mode.

12. **Testing and documentation**
   `tests/test_pipeline.py` and `tests/test_services.py` indicate coverage is centered on the two critical paths: indexing and query/answer services. The `docs/` folder contains architecture and deep-dive material, which likely documents the intended data flow, schema, and operational choices in more detail than the code comments.

13. **How to read the architecture diagram**
   A correct diagram should show a **source ingestion lane** from Confluence into pipeline stages, then into **Qdrant**, then a **query lane** from MCP tools into search/RAG services, then into an **LLM provider**, with **grounding validation** before the response returns to the agent. The clearest subsystem boundaries are: **source integration**, **pipeline/indexing**, **retrieval/ranking**, **generation/validation**, and **MCP serving**.

Main entry points: [`mcp_server/services/search_service.py`](https://github.com/appaidsc/conflux/blob/main/mcp_server/services/search_service.py) (Search), [`mcp_server/services/rag_service.py`](https://github.com/appaidsc/conflux/blob/main/mcp_server/services/rag_service.py) (RAG), [`mcp_server/tools`](https://github.com/appaidsc/conflux/tree/main/mcp_server/tools) (Tools), [`mcp_server/llm/base_provider.py`](https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/base_provider.py) (LLM Provider)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_source["Source & Runtime"]
  node_confluence[("Confluence<br/>source system")]
  node_config["Config<br/>[config.py]"]
end

subgraph group_pipeline["Pipeline & Indexing"]
  node_run_pipeline["Pipeline Run<br/>batch entrypoint<br/>[run_pipeline.py]"]
  node_worker["Worker<br/>async processor"]
  node_discovery["Discovery<br/>[discovery.py]"]
  node_fetcher["Fetcher<br/>content fetch<br/>[fetcher.py]"]
  node_parser["Parser<br/>html normalizer<br/>[parser.py]"]
  node_enrich{{"Enrichment<br/>analysis & scoring"}}
  node_embedder["Embedder<br/>embedding stage<br/>[embedder.py]"]
  node_indexer[("Indexer<br/>index write<br/>[indexer.py]")]
end

subgraph group_retrieval["Retrieval & RAG"]
  node_search_service["Search<br/>hybrid retrieval<br/>[search_service.py]"]
  node_cache_service["Cache<br/>[cache_service.py]"]
  node_rag_service["RAG<br/>answer synthesis<br/>[rag_service.py]"]
  node_validator{{"Validator<br/>grounding check"}}
  node_llm["LLM Provider<br/>provider interface<br/>[base_provider.py]"]
  node_openai["OpenAI<br/>cloud provider<br/>[openai_provider.py]"]
  node_lmstudio["LM Studio<br/>local provider"]
end

subgraph group_serving["MCP Serving"]
  node_server["MCP Server<br/>transport host<br/>[server.py]"]
  node_tools["Tools"]
  node_schemas["Schemas<br/>data contracts"]
  node_agent_clients(("Agent Clients<br/>consumers"))
end

subgraph group_infra["Storage & Models"]
  node_qdrant[("Qdrant<br/>vector store")]
  node_models_cache[("Model Cache<br/>local models")]
end

node_config -->|"drives"| node_run_pipeline
node_config -->|"configures"| node_server
node_run_pipeline -->|"starts"| node_discovery
node_worker -->|"processes"| node_fetcher
node_discovery -->|"feeds"| node_fetcher
node_fetcher -->|"passes"| node_parser
node_parser -->|"normalizes"| node_enrich
node_enrich -->|"enriches"| node_embedder
node_embedder -->|"outputs"| node_indexer
node_indexer -->|"writes"| node_qdrant
node_models_cache -->|"backs"| node_embedder
node_models_cache -->|"backs"| node_search_service
node_tools -->|"queries"| node_search_service
node_tools -->|"asks"| node_rag_service
node_tools -->|"registered by"| node_server
node_server -->|"serves"| node_agent_clients
node_search_service -->|"retrieves from"| node_qdrant
node_search_service -->|"uses"| node_cache_service
node_search_service -->|"returns"| node_schemas
node_rag_service -->|"depends on"| node_search_service
node_rag_service -->|"generates via"| node_llm
node_rag_service -->|"checks with"| node_validator
node_rag_service -->|"returns"| node_schemas
node_llm -->|"implemented by"| node_openai
node_llm -->|"implemented by"| node_lmstudio

click node_config "https://github.com/appaidsc/conflux/blob/main/mcp_server/config.py"
click node_run_pipeline "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/run_pipeline.py"
click node_worker "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/background_worker.py"
click node_discovery "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/discovery.py"
click node_fetcher "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/fetcher.py"
click node_parser "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/parser.py"
click node_embedder "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/embedder.py"
click node_indexer "https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/indexer.py"
click node_models_cache "https://github.com/appaidsc/conflux/tree/main/models_cache"
click node_search_service "https://github.com/appaidsc/conflux/blob/main/mcp_server/services/search_service.py"
click node_cache_service "https://github.com/appaidsc/conflux/blob/main/mcp_server/services/cache_service.py"
click node_rag_service "https://github.com/appaidsc/conflux/blob/main/mcp_server/services/rag_service.py"
click node_validator "https://github.com/appaidsc/conflux/blob/main/mcp_server/services/grounding_validator.py"
click node_llm "https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/base_provider.py"
click node_openai "https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/openai_provider.py"
click node_lmstudio "https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/lm_studio_provider.py"
click node_server "https://github.com/appaidsc/conflux/blob/main/mcp_server/server.py"
click node_tools "https://github.com/appaidsc/conflux/tree/main/mcp_server/tools"
click node_schemas "https://github.com/appaidsc/conflux/tree/main/mcp_server/schemas"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_confluence,node_config toneBlue
class node_run_pipeline,node_worker,node_discovery,node_fetcher,node_parser,node_enrich,node_embedder,node_indexer toneAmber
class node_search_service,node_cache_service,node_rag_service,node_validator,node_llm,node_openai,node_lmstudio toneMint
class node_server,node_tools,node_schemas,node_agent_clients toneRose
class node_qdrant,node_models_cache toneIndigo
```

## Components

### Source & Runtime

External source system plus repo-level runtime/configuration

- **Confluence**: source system · Upstream knowledge source ingested into the repository's corpus
- [Config](https://github.com/appaidsc/conflux/blob/main/mcp_server/config.py): env config · Environment-driven settings for credentials, transport, and backend selection · `mcp_server/config.py`

### Pipeline & Indexing

Offline ingestion, normalization, embedding, and indexing flow

- [Pipeline Run](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/run_pipeline.py): batch entrypoint · Entry point for full or scheduled indexing runs · `mcp_server/pipeline/run_pipeline.py`
- [Worker](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/background_worker.py): async processor · Background processing for progressive or asynchronous ingestion · `mcp_server/pipeline/background_worker.py`
- [Discovery](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/discovery.py): content discovery · Finds pages and content units to ingest · `mcp_server/pipeline/discovery.py`
- [Fetcher](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/fetcher.py): content fetch · Pulls Confluence page content for processing · `mcp_server/pipeline/fetcher.py`
- [Parser](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/parser.py): html normalizer · Converts Confluence HTML into normalized text and structure · `mcp_server/pipeline/parser.py`
- **Enrichment**: analysis & scoring · Adds metadata, importance signals, and summaries
- [Embedder](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/embedder.py): embedding stage · Produces dense and sparse representations for retrieval · `mcp_server/pipeline/embedder.py`
- [Indexer](https://github.com/appaidsc/conflux/blob/main/mcp_server/pipeline/indexer.py): index write · Writes processed content and vectors into the search store · `mcp_server/pipeline/indexer.py`

### Retrieval & RAG

Online search, ranking, generation, and validation services

- [Search](https://github.com/appaidsc/conflux/blob/main/mcp_server/services/search_service.py): hybrid retrieval · Orchestrates query embedding, hybrid search, fusion, and reranking · `mcp_server/services/search_service.py`
- [Cache](https://github.com/appaidsc/conflux/blob/main/mcp_server/services/cache_service.py): response cache · Caches expensive retrieval or lookup results · `mcp_server/services/cache_service.py`
- [RAG](https://github.com/appaidsc/conflux/blob/main/mcp_server/services/rag_service.py): answer synthesis · Turns retrieved context into grounded answers · `mcp_server/services/rag_service.py`
- [Validator](https://github.com/appaidsc/conflux/blob/main/mcp_server/services/grounding_validator.py): grounding check · Checks generated claims against retrieved source evidence · `mcp_server/services/grounding_validator.py`
- [LLM Provider](https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/base_provider.py): provider interface · Shared abstraction for generation backends · `mcp_server/llm/base_provider.py`
- [OpenAI](https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/openai_provider.py): cloud provider · Remote generation backend implementation · `mcp_server/llm/openai_provider.py`
- [LM Studio](https://github.com/appaidsc/conflux/blob/main/mcp_server/llm/lm_studio_provider.py): local provider · Local generation backend implementation · `mcp_server/llm/lm_studio_provider.py`

### MCP Serving

MCP transport and tool surface for agent clients

- [MCP Server](https://github.com/appaidsc/conflux/blob/main/mcp_server/server.py): transport host · Serves the repository over SSE or stdio transports · `mcp_server/server.py`
- [Tools](https://github.com/appaidsc/conflux/tree/main/mcp_server/tools): mcp tools · Externally visible agent actions over pages, search, images, and code · `mcp_server/tools`
- [Schemas](https://github.com/appaidsc/conflux/tree/main/mcp_server/schemas): data contracts · Shared page, search, and RAG payload definitions · `mcp_server/schemas`
- **Agent Clients**: consumers · AI agents and IDEs consuming MCP tools

### Storage & Models

Persistent search store and local model artifacts

- **Qdrant**: vector store · Hybrid search index used for retrieval
- [Model Cache](https://github.com/appaidsc/conflux/tree/main/models_cache): local models · Cached ONNX embedding and reranking models for offline-capable inference · `models_cache`

## Connections

- Config → Pipeline Run: drives · Supplies runtime settings for ingestion runs
- Config → MCP Server: configures · Selects transport and backend behavior
- Pipeline Run → Discovery: starts · Launches the ingestion pipeline
- Worker → Fetcher: processes · Handles progressive fetch and enrichment work
- Discovery → Fetcher: feeds · Provides discovered page targets
- Fetcher → Parser: passes · Sends raw Confluence content for normalization
- Parser → Enrichment: normalizes · Produces structured text for downstream analysis
- Enrichment → Embedder: enriches · Adds metadata, scores, and summaries before embedding
- Embedder → Indexer: outputs · Hands off vectorized documents for indexing
- Indexer → Qdrant: writes · Stores hybrid retrieval records and vectors
- Model Cache → Embedder: backs · Provides local embedding models
- Model Cache → Search: backs · Provides local reranking or query models
- Tools → Search: queries · Search tools call into retrieval services
- Tools → RAG: asks · Question-answering tools call the RAG layer
- Tools → MCP Server: registered by · Tool definitions are exposed through the MCP host
- MCP Server → Agent Clients: serves · Streams tool access to MCP consumers
- Search → Qdrant: retrieves from · Executes hybrid retrieval against the index
- Search → Cache: uses · Reuses cached search and lookup results
- Search → Schemas: returns · Shapes search results with shared contracts
- RAG → Search: depends on · Uses retrieved context as grounding input
- RAG → LLM Provider: generates via · Delegates generation to a backend provider
- RAG → Validator: checks with · Validates answers against evidence
- RAG → Schemas: returns · Shapes RAG responses with shared contracts
- LLM Provider → OpenAI: implemented by · Cloud generation backend
- LLM Provider → LM Studio: implemented by · Local generation backend
