# trehansalil/pageindex architecture

> Architecture diagram of the GitHub repository trehansalil/pageindex, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/trehansalil/pageindex
- Repository: https://github.com/trehansalil/pageindex

## Overview

PageIndex MCP is a self-hosted, vectorless RAG service: clients upload documents, an asynchronous worker extracts and quality-checks content into hierarchical or flat indexes, and MCP tools retrieve relevant passages through LLM-guided tree search. Its main runtime layers are the server/API, worker and converter pipeline, search and table handling, and durable storage. The README documents the ingestion routes and shared infrastructure; sampled source confirms parts of search, conversion, and server composition, so some internal links are omitted where ownership is not evidenced.

Main entry points: [`src/pageindex_mcp/helpers/rag.py`](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/helpers/rag.py) (Tree RAG search), [`src/pageindex_mcp/worker/job.py`](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/worker/job.py) (Async worker), [`src/pageindex_mcp/server.py`](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/server.py) (MCP server), [`src/pageindex_mcp/converters/pipeline.py`](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/converters/pipeline.py) (Conversion pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["Server and API"]
  node_server["MCP server<br/>[server.py]"]
  node_query_tools["Document tools<br/>[documents.py]"]
  node_upload_api["Upload API<br/>[upload_app.py]"]
end

subgraph group_ingest["Ingestion pipeline"]
  node_worker["Async worker<br/>[job.py]"]
  node_subprocess["Job subprocess<br/>[subprocess_mgr.py]"]
  node_indexer["Page index client<br/>[indexer.py]"]
  node_conversion["Conversion pipeline<br/>[pipeline.py]"]
  node_docling_service["Docling service<br/>[app.py]"]
  node_quality["Tree quality gate<br/>[tree_validation.py]"]
  node_ocr["OCR and recovery<br/>[pictures.py]"]
end

subgraph group_search["Query and tables"]
  node_table_anchor["Table anchoring<br/>[anchor.py]"]
  node_search_view["Table search view<br/>[search_view.py]"]
  node_rag["Tree RAG search<br/>[rag.py]"]
end

subgraph group_state["Persistence and registry"]
  node_minio[("MinIO objects<br/>[minio_ops.py]")]
  node_redis[("Redis cache and queue<br/>[cache.py]")]
  node_registry[("Document registry<br/>[queries.py]")]
end

subgraph group_integrations["External services"]
  node_llm["LLM provider"]
  node_postgres[("PostgreSQL")]
  node_prometheus["Prometheus"]
end

node_mcp_client(("MCP client"))
node_upload_client(("Upload client"))

node_mcp_client -->|"calls MCP"| node_server
node_upload_client -->|"submits files"| node_upload_api
node_server -->|"registers tools"| node_query_tools
node_query_tools -->|"queries documents"| node_rag
node_server -->|"mounts API"| node_upload_api
node_upload_api -->|"stages files"| node_minio
node_upload_api -->|"enqueues jobs"| node_redis
node_worker -->|"consumes jobs"| node_redis
node_worker -->|"runs isolated job"| node_subprocess
node_subprocess -->|"starts indexing"| node_indexer
node_indexer -->|"converts documents"| node_conversion
node_conversion -.->|"offloads conversion"| node_docling_service
node_conversion -->|"validates tree"| node_quality
node_conversion -->|"recovers content"| node_ocr
node_quality -->|"anchors tables"| node_table_anchor
node_worker -->|"persists results"| node_minio
node_worker -->|"writes catalog"| node_registry
node_rag -->|"reads cache"| node_redis
node_rag -->|"loads trees"| node_minio
node_rag -->|"narrows documents"| node_registry
node_rag -->|"renders search view"| node_search_view
node_rag -->|"requests reasoning"| node_llm
node_indexer -->|"builds index tree"| node_llm
node_registry -->|"stores catalog"| node_postgres
node_server -->|"exposes metrics"| node_prometheus

click node_server "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/server.py"
click node_query_tools "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/tools/documents.py"
click node_upload_api "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/upload_app.py"
click node_worker "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/worker/job.py"
click node_subprocess "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/worker/subprocess_mgr.py"
click node_indexer "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/client/indexer.py"
click node_conversion "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/converters/pipeline.py"
click node_docling_service "https://github.com/trehansalil/pageindex/blob/master/services/docling-service/app.py"
click node_quality "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/helpers/tree_validation.py"
click node_ocr "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/converters/pictures.py"
click node_table_anchor "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/tables/anchor.py"
click node_search_view "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/tables/search_view.py"
click node_rag "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/helpers/rag.py"
click node_minio "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/storage/minio_ops.py"
click node_redis "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/cache.py"
click node_registry "https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/registry/queries.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_query_tools,node_upload_api,node_mcp_client,node_upload_client toneBlue
class node_worker,node_subprocess,node_indexer,node_conversion,node_docling_service,node_quality,node_ocr toneAmber
class node_table_anchor,node_search_view,node_rag toneMint
class node_minio,node_redis,node_registry toneRose
class node_llm,node_postgres,node_prometheus toneIndigo
```

## Components

### Server and API

- [MCP server](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/server.py): `src/pageindex_mcp/server.py`
- [Document tools](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/tools/documents.py): `src/pageindex_mcp/tools/documents.py`
- [Upload API](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/upload_app.py): `src/pageindex_mcp/upload_app.py`

### Ingestion pipeline

- [Async worker](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/worker/job.py): `src/pageindex_mcp/worker/job.py`
- [Job subprocess](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/worker/subprocess_mgr.py): `src/pageindex_mcp/worker/subprocess_mgr.py`
- [Page index client](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/client/indexer.py): `src/pageindex_mcp/client/indexer.py`
- [Conversion pipeline](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/converters/pipeline.py): `src/pageindex_mcp/converters/pipeline.py`
- [Docling service](https://github.com/trehansalil/pageindex/blob/master/services/docling-service/app.py): `services/docling-service/app.py`
- [Tree quality gate](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/helpers/tree_validation.py): `src/pageindex_mcp/helpers/tree_validation.py`
- [OCR and recovery](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/converters/pictures.py): `src/pageindex_mcp/converters/pictures.py`

### Query and tables

- [Table anchoring](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/tables/anchor.py): `src/pageindex_mcp/tables/anchor.py`
- [Table search view](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/tables/search_view.py): `src/pageindex_mcp/tables/search_view.py`
- [Tree RAG search](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/helpers/rag.py): `src/pageindex_mcp/helpers/rag.py`

### Persistence and registry

- [MinIO objects](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/storage/minio_ops.py): `src/pageindex_mcp/storage/minio_ops.py`
- [Redis cache and queue](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/cache.py): `src/pageindex_mcp/cache.py`
- [Document registry](https://github.com/trehansalil/pageindex/blob/master/src/pageindex_mcp/registry/queries.py): `src/pageindex_mcp/registry/queries.py`

### External services

- **LLM provider**
- **PostgreSQL**
- **Prometheus**

### Other components

- **MCP client**
- **Upload client**

## Connections

- MCP client → MCP server: calls MCP
- Upload client → Upload API: submits files
- MCP server → Document tools: registers tools
- Document tools → Tree RAG search: queries documents
- MCP server → Upload API: mounts API
- Upload API → MinIO objects: stages files
- Upload API → Redis cache and queue: enqueues jobs
- Async worker → Redis cache and queue: consumes jobs
- Async worker → Job subprocess: runs isolated job
- Job subprocess → Page index client: starts indexing
- Page index client → Conversion pipeline: converts documents
- Conversion pipeline → Docling service: offloads conversion
- Conversion pipeline → Tree quality gate: validates tree
- Conversion pipeline → OCR and recovery: recovers content
- Tree quality gate → Table anchoring: anchors tables
- Async worker → MinIO objects: persists results
- Async worker → Document registry: writes catalog
- Tree RAG search → Redis cache and queue: reads cache
- Tree RAG search → MinIO objects: loads trees
- Tree RAG search → Document registry: narrows documents
- Tree RAG search → Table search view: renders search view
- Tree RAG search → LLM provider: requests reasoning
- Page index client → LLM provider: builds index tree
- Document registry → PostgreSQL: stores catalog
- MCP server → Prometheus: exposes metrics