# ontos-ai/knowhere architecture

> Architecture diagram of the GitHub repository ontos-ai/knowhere, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ontos-ai/knowhere
- Repository: https://github.com/ontos-ai/knowhere

## Overview

Knowhere turns uploaded documents into persistent, hierarchy-aware memory and exposes that memory for agent retrieval. The documented flow runs from API submission through worker ingestion, format-specific parsing, unified memory construction, and retrieval with evidence references. Major documented branches include vision and text parsing, built-in retrieval and MCP tools, plus account and billing APIs. The README describes product behavior broadly; sampled source confirms the parser pipeline, worker task entry points, storage responsibilities, and retrieval routing, but not every API-to-service call.

Main entry points: [`apps/worker/app/services/document_ingestion/service.py`](https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/document_ingestion/service.py) (Worker ingestion), [`apps/api/app/api/v2/routes/jobs.py`](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/jobs.py) (Jobs API), [`packages/shared-python/shared/services/retrieval/execution/routes.py`](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/retrieval/execution/routes.py) (Retrieval routing), [`packages/shared-python/shared/core/database.py`](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/core/database.py) (Application database)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_api["API and access"]
  node_api["Document API<br/>[main.py]"]
  node_jobs_api["Jobs API<br/>[jobs.py]"]
  node_documents_api["Documents API<br/>[documents.py]"]
  node_retrieval_api["Retrieval API<br/>[retrieval.py]"]
  node_admission["Job admission<br/>[job_admission.py]"]
end

subgraph group_ingest["Ingestion and parsing"]
  node_ingestion_api["Ingestion workflow<br/>[service.py]"]
  node_task["Ingestion tasks"]
  node_worker_ingestion["Worker ingestion<br/>[service.py]"]
  node_parser["Format parser<br/>[parse_pipeline.py]"]
  node_text_track["Text track"]
  node_vision_track["Vision track"]
  node_summary["Summary engine<br/>[engine.py]"]
end

subgraph group_memory["Memory and storage"]
  node_page_memory["Page memory<br/>[memory_service.py]"]
  node_result_storage["Result storage<br/>[result_storage.py]"]
  node_job_storage["Source storage"]
  node_database[("Application database<br/>[database.py]")]
end

subgraph group_retrieve["Retrieval and tools"]
  node_mcp["MCP tools"]
  node_retrieval_routes["Retrieval routing<br/>[routes.py]"]
  node_graph_query["Graph queries<br/>[query_service.py]"]
end

subgraph group_platform["Account and billing"]
  node_billing["Billing workflows"]
  node_billing_api["Billing API<br/>[billing.py]"]
  node_auth["Authentication"]
end

node_user(("Uploader"))
node_agent(("Agent client"))
node_object_store[("Object store")]
node_dashboard(("Dashboard"))
node_llm{{"AI models"}}

node_user -->|"submits documents"| node_api
node_api -->|"routes requests"| node_jobs_api
node_jobs_api -->|"checks capacity"| node_admission
node_jobs_api -->|"starts ingestion"| node_ingestion_api
node_ingestion_api -->|"dispatches work"| node_task
node_task -->|"runs job"| node_worker_ingestion
node_worker_ingestion -->|"parses source"| node_parser
node_parser -->|"routes text formats"| node_text_track
node_parser -->|"routes vision formats"| node_vision_track
node_text_track -->|"requests summaries"| node_summary
node_vision_track -->|"requests summaries"| node_summary
node_summary -.->|"calls models"| node_llm
node_worker_ingestion -->|"builds memory"| node_page_memory
node_worker_ingestion -->|"reads source files"| node_job_storage
node_worker_ingestion -->|"stores results"| node_result_storage
node_job_storage -->|"reads and writes"| node_object_store
node_result_storage -->|"uploads artifacts"| node_object_store
node_jobs_api -->|"reads and writes jobs"| node_database
node_page_memory -->|"persists memory"| node_database
node_user -->|"manages documents"| node_documents_api
node_agent -->|"requests retrieval"| node_retrieval_api
node_retrieval_api -->|"routes query"| node_retrieval_routes
node_retrieval_routes -->|"queries sections"| node_graph_query
node_retrieval_routes -->|"loads memory"| node_database
node_agent -->|"uses corpus tools"| node_mcp
node_mcp -->|"dispatches retrieval"| node_retrieval_routes
node_dashboard -->|"provides JWT"| node_auth
node_billing_api -->|"runs billing"| node_billing
node_billing -->|"reads and writes records"| node_database

click node_api "https://github.com/ontos-ai/knowhere/blob/main/apps/api/main.py"
click node_jobs_api "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/jobs.py"
click node_documents_api "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/documents.py"
click node_retrieval_api "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/retrieval.py"
click node_mcp "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/mcp/retrieval_server.py"
click node_admission "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/dependencies/job_admission.py"
click node_ingestion_api "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/services/document_ingestion/service.py"
click node_task "https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/core/tasks/document_ingestion_tasks.py"
click node_worker_ingestion "https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/document_ingestion/service.py"
click node_parser "https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/document_parser/orchestration/parse_pipeline.py"
click node_text_track "https://github.com/ontos-ai/knowhere/tree/main/apps/worker/app/services/document_parser/formats"
click node_vision_track "https://github.com/ontos-ai/knowhere/tree/main/apps/worker/app/services/document_agent"
click node_summary "https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/ai/summary/engine.py"
click node_page_memory "https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/page_memory/memory_service.py"
click node_result_storage "https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/storage/result_storage.py"
click node_job_storage "https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/storage/job_file_storage.py"
click node_database "https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/core/database.py"
click node_retrieval_routes "https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/retrieval/execution/routes.py"
click node_graph_query "https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/retrieval/graph/query_service.py"
click node_billing "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/services/billing/billing_workflow_service.py"
click node_billing_api "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v1/routes/billing.py"
click node_auth "https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/services/auth/dashboard_jwt_authentication_service.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_jobs_api,node_documents_api,node_retrieval_api,node_admission,node_agent toneBlue
class node_ingestion_api,node_task,node_worker_ingestion,node_parser,node_text_track,node_vision_track,node_summary,node_object_store toneAmber
class node_page_memory,node_result_storage,node_job_storage,node_database toneMint
class node_mcp,node_retrieval_routes,node_graph_query toneRose
class node_billing,node_billing_api,node_auth,node_user,node_dashboard,node_llm toneIndigo
```

## Components

### API and access

- [Document API](https://github.com/ontos-ai/knowhere/blob/main/apps/api/main.py): `apps/api/main.py`
- [Jobs API](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/jobs.py): `apps/api/app/api/v2/routes/jobs.py`
- [Documents API](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/documents.py): `apps/api/app/api/v2/routes/documents.py`
- [Retrieval API](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v2/routes/retrieval.py): `apps/api/app/api/v2/routes/retrieval.py`
- [Job admission](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/dependencies/job_admission.py): `apps/api/app/api/dependencies/job_admission.py`

### Ingestion and parsing

- [Ingestion workflow](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/services/document_ingestion/service.py): `apps/api/app/services/document_ingestion/service.py`
- [Ingestion tasks](https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/core/tasks/document_ingestion_tasks.py): `apps/worker/app/core/tasks/document_ingestion_tasks.py`
- [Worker ingestion](https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/document_ingestion/service.py): `apps/worker/app/services/document_ingestion/service.py`
- [Format parser](https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/document_parser/orchestration/parse_pipeline.py): `apps/worker/app/services/document_parser/orchestration/parse_pipeline.py`
- [Text track](https://github.com/ontos-ai/knowhere/tree/main/apps/worker/app/services/document_parser/formats): `apps/worker/app/services/document_parser/formats`
- [Vision track](https://github.com/ontos-ai/knowhere/tree/main/apps/worker/app/services/document_agent): `apps/worker/app/services/document_agent`
- [Summary engine](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/ai/summary/engine.py): `packages/shared-python/shared/services/ai/summary/engine.py`

### Memory and storage

- [Page memory](https://github.com/ontos-ai/knowhere/blob/main/apps/worker/app/services/page_memory/memory_service.py): `apps/worker/app/services/page_memory/memory_service.py`
- [Result storage](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/storage/result_storage.py): `packages/shared-python/shared/services/storage/result_storage.py`
- [Source storage](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/storage/job_file_storage.py): `packages/shared-python/shared/services/storage/job_file_storage.py`
- [Application database](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/core/database.py): `packages/shared-python/shared/core/database.py`

### Retrieval and tools

- [MCP tools](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/mcp/retrieval_server.py): `apps/api/app/mcp/retrieval_server.py`
- [Retrieval routing](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/retrieval/execution/routes.py): `packages/shared-python/shared/services/retrieval/execution/routes.py`
- [Graph queries](https://github.com/ontos-ai/knowhere/blob/main/packages/shared-python/shared/services/retrieval/graph/query_service.py): `packages/shared-python/shared/services/retrieval/graph/query_service.py`

### Account and billing

- [Billing workflows](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/services/billing/billing_workflow_service.py): `apps/api/app/services/billing/billing_workflow_service.py`
- [Billing API](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/api/v1/routes/billing.py): `apps/api/app/api/v1/routes/billing.py`
- [Authentication](https://github.com/ontos-ai/knowhere/blob/main/apps/api/app/services/auth/dashboard_jwt_authentication_service.py): `apps/api/app/services/auth/dashboard_jwt_authentication_service.py`

### Other components

- **Uploader**
- **Agent client**
- **Object store**
- **Dashboard**
- **AI models**

## Connections

- Uploader → Document API: submits documents
- Document API → Jobs API: routes requests
- Jobs API → Job admission: checks capacity
- Jobs API → Ingestion workflow: starts ingestion
- Ingestion workflow → Ingestion tasks: dispatches work
- Ingestion tasks → Worker ingestion: runs job
- Worker ingestion → Format parser: parses source
- Format parser → Text track: routes text formats
- Format parser → Vision track: routes vision formats
- Text track → Summary engine: requests summaries
- Vision track → Summary engine: requests summaries
- Summary engine → AI models: calls models
- Worker ingestion → Page memory: builds memory
- Worker ingestion → Source storage: reads source files
- Worker ingestion → Result storage: stores results
- Source storage → Object store: reads and writes
- Result storage → Object store: uploads artifacts
- Jobs API → Application database: reads and writes jobs
- Page memory → Application database: persists memory
- Uploader → Documents API: manages documents
- Agent client → Retrieval API: requests retrieval
- Retrieval API → Retrieval routing: routes query
- Retrieval routing → Graph queries: queries sections
- Retrieval routing → Application database: loads memory
- Agent client → MCP tools: uses corpus tools
- MCP tools → Retrieval routing: dispatches retrieval
- Dashboard → Authentication: provides JWT
- Billing API → Billing workflows: runs billing
- Billing workflows → Application database: reads and writes records