# haris525/local-multimodal-rag architecture

> Architecture diagram of the GitHub repository haris525/local-multimodal-rag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/haris525/local-multimodal-rag
- Repository: https://github.com/haris525/local-multimodal-rag

## Overview

Local Multimodal RAG lets signed-in users ingest private documents and ask cited questions over selected files. The main workflow runs from upload through parsing, structured-table extraction, chunking and indexing, then question planning, hybrid retrieval and answer generation; chats, accounts, audit, and optional analysis services support it. The README documents several service-level connections not visible in sampled implementation excerpts, so those edges cite the README. Unsampled modules are represented only where the README or source index supports their role.

Main entry points: [`services/rag/src/rag/api.py`](https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/api.py) (RAG API), [`services/gateway/static/app.js`](https://github.com/haris525/local-multimodal-rag/blob/main/services/gateway/static/app.js) (Web UI), [`services/ingest/src/ingest/pipeline.py`](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/pipeline.py) (Document pipeline), [`services/ingest/src/ingest/api.py`](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/api.py) (Upload and registry API)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_access["User access"]
  node_ui["Web UI<br/>[app.js]"]
  node_auth["Accounts and sessions<br/>[api.py]"]
end

subgraph group_ingestion["Document ingestion"]
  node_ingest_api["Upload and registry API<br/>[api.py]"]
  node_worker["Ingestion worker<br/>[worker.py]"]
  node_pipeline["Document pipeline<br/>[pipeline.py]"]
  node_parsers["Format parsers"]
  node_chunking["Chunking<br/>[chunking.py]"]
  node_indexer["Index writer<br/>[indexer.py]"]
end

subgraph group_answers["Question answering"]
  node_rag_api["RAG API<br/>[api.py]"]
  node_planner["Query planning<br/>[planner.py]"]
  node_retrieval["Hybrid retrieval<br/>[retrieval.py]"]
  node_agent["Deep search agent<br/>[agent.py]"]
  node_exhaustive["Read-all mode<br/>[exhaustive.py]"]
  node_models["Embedding and reranking<br/>[encoders.py]"]
  node_llm["Local language model"]
end

subgraph group_state["State and services"]
  node_contracts["Service contracts<br/>[contracts.py]"]
  node_qdrant[("Vector and BM25 index")]
  node_sql["Read-only SQL service<br/>[api.py]"]
  node_sandbox["Python sandbox<br/>[api.py]"]
  node_threads["Chats and memory<br/>[api.py]"]
  node_audit["Audit log<br/>[api.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval["RAG evaluation<br/>[cli.py]"]
end

node_user(("User"))

node_user -->|"asks through"| node_ui
node_ui -->|"signs in via"| node_auth
node_ui -->|"uploads documents"| node_ingest_api
node_ui -->|"asks questions"| node_rag_api
node_ingest_api -->|"queues documents"| node_worker
node_worker -->|"runs jobs"| node_pipeline
node_pipeline -->|"parses files"| node_parsers
node_pipeline -->|"chunks content"| node_chunking
node_pipeline -->|"writes index"| node_indexer
node_indexer -->|"writes vectors"| node_qdrant
node_rag_api -->|"routes questions"| node_planner
node_planner -->|"selects search"| node_retrieval
node_retrieval -->|"searches index"| node_qdrant
node_retrieval -->|"embeds and reranks"| node_models
node_rag_api -->|"runs deep search"| node_agent
node_rag_api -->|"runs read-all"| node_exhaustive
node_rag_api -->|"generates answers"| node_llm
node_rag_api -->|"loads and saves chats"| node_threads
node_planner -->|"queries tables"| node_sql
node_agent -.->|"may run Python"| node_sandbox
node_worker -->|"records events"| node_audit
node_ingest_api -.->|"uses shared models"| node_contracts
node_eval -->|"evaluates answers"| node_rag_api

click node_ui "https://github.com/haris525/local-multimodal-rag/blob/main/services/gateway/static/app.js"
click node_auth "https://github.com/haris525/local-multimodal-rag/blob/main/services/auth/src/auth/api.py"
click node_contracts "https://github.com/haris525/local-multimodal-rag/blob/main/libs/common/src/rag_common/contracts.py"
click node_ingest_api "https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/api.py"
click node_worker "https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/worker.py"
click node_pipeline "https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/pipeline.py"
click node_parsers "https://github.com/haris525/local-multimodal-rag/tree/main/services/ingest/src/ingest/parsers"
click node_chunking "https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/chunking.py"
click node_indexer "https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/indexer.py"
click node_rag_api "https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/api.py"
click node_planner "https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/planner.py"
click node_retrieval "https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/retrieval.py"
click node_agent "https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/agent.py"
click node_exhaustive "https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/exhaustive.py"
click node_models "https://github.com/haris525/local-multimodal-rag/blob/main/services/models/src/models_svc/encoders.py"
click node_sql "https://github.com/haris525/local-multimodal-rag/blob/main/services/sql/src/sql_sandbox/api.py"
click node_sandbox "https://github.com/haris525/local-multimodal-rag/blob/main/services/sandbox/src/sandbox/api.py"
click node_threads "https://github.com/haris525/local-multimodal-rag/blob/main/services/threads/src/threads/api.py"
click node_audit "https://github.com/haris525/local-multimodal-rag/blob/main/services/audit/src/audit/api.py"
click node_eval "https://github.com/haris525/local-multimodal-rag/blob/main/eval/src/rag_eval/cli.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_auth,node_user toneBlue
class node_ingest_api,node_worker,node_pipeline,node_parsers,node_chunking,node_indexer toneAmber
class node_rag_api,node_planner,node_retrieval,node_agent,node_exhaustive,node_models,node_llm toneMint
class node_contracts,node_qdrant,node_sql,node_sandbox,node_threads,node_audit toneRose
class node_eval toneIndigo
```

## Components

### User access

- [Web UI](https://github.com/haris525/local-multimodal-rag/blob/main/services/gateway/static/app.js): `services/gateway/static/app.js`
- [Accounts and sessions](https://github.com/haris525/local-multimodal-rag/blob/main/services/auth/src/auth/api.py): `services/auth/src/auth/api.py`

### Document ingestion

- [Upload and registry API](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/api.py): `services/ingest/src/ingest/api.py`
- [Ingestion worker](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/worker.py): `services/ingest/src/ingest/worker.py`
- [Document pipeline](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/pipeline.py): `services/ingest/src/ingest/pipeline.py`
- [Format parsers](https://github.com/haris525/local-multimodal-rag/tree/main/services/ingest/src/ingest/parsers): `services/ingest/src/ingest/parsers`
- [Chunking](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/chunking.py): `services/ingest/src/ingest/chunking.py`
- [Index writer](https://github.com/haris525/local-multimodal-rag/blob/main/services/ingest/src/ingest/indexer.py): `services/ingest/src/ingest/indexer.py`

### Question answering

- [RAG API](https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/api.py): `services/rag/src/rag/api.py`
- [Query planning](https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/planner.py): `services/rag/src/rag/planner.py`
- [Hybrid retrieval](https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/retrieval.py): `services/rag/src/rag/retrieval.py`
- [Deep search agent](https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/agent.py): `services/rag/src/rag/agent.py`
- [Read-all mode](https://github.com/haris525/local-multimodal-rag/blob/main/services/rag/src/rag/exhaustive.py): `services/rag/src/rag/exhaustive.py`
- [Embedding and reranking](https://github.com/haris525/local-multimodal-rag/blob/main/services/models/src/models_svc/encoders.py): `services/models/src/models_svc/encoders.py`
- **Local language model**

### State and services

- [Service contracts](https://github.com/haris525/local-multimodal-rag/blob/main/libs/common/src/rag_common/contracts.py): `libs/common/src/rag_common/contracts.py`
- **Vector and BM25 index**
- [Read-only SQL service](https://github.com/haris525/local-multimodal-rag/blob/main/services/sql/src/sql_sandbox/api.py): `services/sql/src/sql_sandbox/api.py`
- [Python sandbox](https://github.com/haris525/local-multimodal-rag/blob/main/services/sandbox/src/sandbox/api.py): `services/sandbox/src/sandbox/api.py`
- [Chats and memory](https://github.com/haris525/local-multimodal-rag/blob/main/services/threads/src/threads/api.py): `services/threads/src/threads/api.py`
- [Audit log](https://github.com/haris525/local-multimodal-rag/blob/main/services/audit/src/audit/api.py): `services/audit/src/audit/api.py`

### Evaluation

- [RAG evaluation](https://github.com/haris525/local-multimodal-rag/blob/main/eval/src/rag_eval/cli.py): `eval/src/rag_eval/cli.py`

### Other components

- **User**

## Connections

- User → Web UI: asks through (evidence: `README.md`)
- Web UI → Accounts and sessions: signs in via (evidence: `README.md`)
- Web UI → Upload and registry API: uploads documents (evidence: `README.md`)
- Web UI → RAG API: asks questions (evidence: `README.md`)
- Upload and registry API → Ingestion worker: queues documents (evidence: `README.md`)
- Ingestion worker → Document pipeline: runs jobs (evidence: `services/ingest/src/ingest/worker.py`)
- Document pipeline → Format parsers: parses files (evidence: `services/ingest/src/ingest/pipeline.py`)
- Document pipeline → Chunking: chunks content (evidence: `services/ingest/src/ingest/pipeline.py`)
- Document pipeline → Index writer: writes index (evidence: `services/ingest/src/ingest/pipeline.py`)
- Index writer → Vector and BM25 index: writes vectors (evidence: `README.md`)
- RAG API → Query planning: routes questions (evidence: `README.md`)
- Query planning → Hybrid retrieval: selects search (evidence: `README.md`)
- Hybrid retrieval → Vector and BM25 index: searches index (evidence: `README.md`)
- Hybrid retrieval → Embedding and reranking: embeds and reranks (evidence: `README.md`)
- RAG API → Deep search agent: runs deep search (evidence: `README.md`)
- RAG API → Read-all mode: runs read-all (evidence: `README.md`)
- RAG API → Local language model: generates answers (evidence: `README.md`)
- RAG API → Chats and memory: loads and saves chats (evidence: `README.md`)
- Query planning → Read-only SQL service: queries tables (evidence: `README.md`)
- Deep search agent → Python sandbox: may run Python (evidence: `README.md`)
- Ingestion worker → Audit log: records events (evidence: `README.md`)
- Upload and registry API → Service contracts: uses shared models
- RAG evaluation → RAG API: evaluates answers (evidence: `README.md`)
