# 21mohit-dotcom/vector_database_design_for_rag_system architecture

> Architecture diagram of the GitHub repository 21mohit-dotcom/vector_database_design_for_rag_system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/21mohit-dotcom/vector_database_design_for_rag_system
- Repository: https://github.com/21mohit-dotcom/vector_database_design_for_rag_system

## Overview

PrecisionRAG is an MS MARCO passage-retrieval system backed by Qdrant. Its main workflow indexes cleaned passages with dense and BM25 vectors, then serves filtered dense, keyword, or hybrid queries through an API; hybrid results can be fused and reranked before returning passages and timings. The README also documents metadata/topic filtering, live document updates, AI answers, dashboards, and evaluation. Edges follow the shown caller relationships or explicit README workflows; some documented subsystems have only directory-level paths because their implementation was not sampled.

Main entry points: [`precisionrag/retriever.py`](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/retriever.py) (Search pipelines), [`api/main.py`](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/api/main.py) (FastAPI service), [`precisionrag/updates.py`](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/updates.py) (Live upsert and delete), [`scripts/build_index.py`](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/scripts/build_index.py) (Index builder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_indexing["Corpus Indexing"]
  node_build["Index builder<br/>[build_index.py]"]
  node_data["Clean and deduplicate<br/>[data.py]"]
end

subgraph group_retrieval["Query Retrieval"]
  node_encoders["Dense and BM25 encoders<br/>[encoders.py]"]
  node_store[("Qdrant store<br/>[store.py]")]
  node_retriever["Search pipelines<br/>[retriever.py]"]
  node_fusion["Rank fusion<br/>[fusion.py]"]
  node_rerank["Cross-encoder rerank<br/>[rerank.py]"]
end

subgraph group_serving["API and UI"]
  node_api["FastAPI service<br/>[main.py]"]
  node_web["Search and dashboards UI<br/>[app.js]"]
  node_answer["AI cited answers<br/>[answer.py]"]
  node_cache["Result cache<br/>[cache.py]"]
end

subgraph group_operations["Updates and Metadata"]
  node_tags["Metadata tagging<br/>[tags.py]"]
  node_updater["Live upsert and delete<br/>[updates.py]"]
  node_changelog["Version history<br/>[changelog.py]"]
end

subgraph group_insights["Evaluation and Insights"]
  node_dashboard["Results dashboard data<br/>[dashboard.py]"]
  node_eval["Retrieval evaluation<br/>[eval_ir.py]"]
  node_metrics["Ranking metrics<br/>[metrics.py]"]
  node_results["Benchmark results<br/>[results.py]"]
end

node_user(("Search user"))
node_corpus["MS MARCO passages"]
node_qdrant[("Qdrant service")]
node_groq["Groq judge service"]

node_corpus -->|"provides passages"| node_build
node_build -->|"builds corpus"| node_data
node_build -->|"encodes passages"| node_encoders
node_build -->|"writes index"| node_store
node_store -->|"reads and writes"| node_qdrant
node_user -->|"submits query"| node_web
node_web -->|"calls API"| node_api
node_api -->|"dispatches search"| node_retriever
node_retriever -->|"encodes query"| node_encoders
node_retriever -->|"fetches candidates"| node_store
node_retriever -->|"fuses rankings"| node_fusion
node_retriever -->|"reranks candidates"| node_rerank
node_retriever -->|"routes by topic"| node_tags
node_retriever -->|"returns passages"| node_user
node_api -->|"checks and fills"| node_cache
node_api -->|"routes updates"| node_updater
node_updater -->|"prepares documents"| node_data
node_updater -->|"embeds documents"| node_encoders
node_updater -->|"upserts or deletes"| node_store
node_updater -->|"assigns metadata"| node_tags
node_updater -->|"records versions"| node_changelog
node_api -->|"generates answer"| node_answer
node_answer -.->|"requests generation"| node_groq
node_api -->|"serves dashboard data"| node_dashboard
node_web -->|"displays measurements"| node_dashboard
node_eval -->|"evaluates modes"| node_retriever
node_eval -->|"scores rankings"| node_metrics
node_eval -->|"saves measurements"| node_results
node_dashboard -->|"reads measurements"| node_results

click node_build "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/scripts/build_index.py"
click node_data "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/data.py"
click node_encoders "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/encoders.py"
click node_store "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/store.py"
click node_retriever "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/retriever.py"
click node_fusion "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/fusion.py"
click node_rerank "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/rerank.py"
click node_tags "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/tags.py"
click node_api "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/api/main.py"
click node_web "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/web/app.js"
click node_answer "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/answer.py"
click node_updater "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/updates.py"
click node_changelog "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/changelog.py"
click node_cache "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/cache.py"
click node_dashboard "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/dashboard.py"
click node_eval "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/scripts/eval_ir.py"
click node_metrics "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/metrics.py"
click node_results "https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/results.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_build,node_data,node_user toneBlue
class node_encoders,node_store,node_retriever,node_fusion,node_rerank,node_qdrant toneAmber
class node_api,node_web,node_answer,node_cache toneMint
class node_tags,node_updater,node_changelog toneRose
class node_dashboard,node_eval,node_metrics,node_results,node_corpus,node_groq toneIndigo
```

## Components

### Corpus Indexing

- [Index builder](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/scripts/build_index.py): `scripts/build_index.py`
- [Clean and deduplicate](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/data.py): `precisionrag/data.py`

### Query Retrieval

- [Dense and BM25 encoders](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/encoders.py): `precisionrag/encoders.py`
- [Qdrant store](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/store.py): `precisionrag/store.py`
- [Search pipelines](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/retriever.py): `precisionrag/retriever.py`
- [Rank fusion](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/fusion.py): `precisionrag/fusion.py`
- [Cross-encoder rerank](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/rerank.py): `precisionrag/rerank.py`

### API and UI

- [FastAPI service](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/api/main.py): `api/main.py`
- [Search and dashboards UI](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/web/app.js): `web/app.js`
- [AI cited answers](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/answer.py): `precisionrag/answer.py`
- [Result cache](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/cache.py): `precisionrag/cache.py`

### Updates and Metadata

- [Metadata tagging](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/tags.py): `precisionrag/tags.py`
- [Live upsert and delete](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/updates.py): `precisionrag/updates.py`
- [Version history](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/changelog.py): `precisionrag/changelog.py`

### Evaluation and Insights

- [Results dashboard data](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/dashboard.py): `precisionrag/dashboard.py`
- [Retrieval evaluation](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/scripts/eval_ir.py): `scripts/eval_ir.py`
- [Ranking metrics](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/metrics.py): `precisionrag/metrics.py`
- [Benchmark results](https://github.com/21mohit-dotcom/vector_database_design_for_rag_system/blob/main/precisionrag/results.py): `precisionrag/results.py`

### Other components

- **Search user**
- **MS MARCO passages**
- **Qdrant service**
- **Groq judge service**

## Connections

- MS MARCO passages → Index builder: provides passages (evidence: `README.md`)
- Index builder → Clean and deduplicate: builds corpus (evidence: `scripts/build_index.py`)
- Index builder → Dense and BM25 encoders: encodes passages (evidence: `scripts/build_index.py`)
- Index builder → Qdrant store: writes index (evidence: `scripts/build_index.py`)
- Qdrant store → Qdrant service: reads and writes (evidence: `precisionrag/store.py`)
- Search user → Search and dashboards UI: submits query (evidence: `README.md`)
- Search and dashboards UI → FastAPI service: calls API (evidence: `README.md`)
- FastAPI service → Search pipelines: dispatches search (evidence: `api/main.py`)
- Search pipelines → Dense and BM25 encoders: encodes query (evidence: `precisionrag/retriever.py`)
- Search pipelines → Qdrant store: fetches candidates (evidence: `precisionrag/retriever.py`)
- Search pipelines → Rank fusion: fuses rankings (evidence: `precisionrag/retriever.py`)
- Search pipelines → Cross-encoder rerank: reranks candidates (evidence: `precisionrag/retriever.py`)
- Search pipelines → Metadata tagging: routes by topic (evidence: `precisionrag/retriever.py`)
- Search pipelines → Search user: returns passages (evidence: `README.md`)
- FastAPI service → Result cache: checks and fills (evidence: `api/main.py`)
- FastAPI service → Live upsert and delete: routes updates (evidence: `api/main.py`)
- Live upsert and delete → Clean and deduplicate: prepares documents (evidence: `precisionrag/updates.py`)
- Live upsert and delete → Dense and BM25 encoders: embeds documents (evidence: `precisionrag/updates.py`)
- Live upsert and delete → Qdrant store: upserts or deletes (evidence: `precisionrag/updates.py`)
- Live upsert and delete → Metadata tagging: assigns metadata (evidence: `precisionrag/updates.py`)
- Live upsert and delete → Version history: records versions (evidence: `precisionrag/updates.py`)
- FastAPI service → AI cited answers: generates answer (evidence: `api/main.py`)
- AI cited answers → Groq judge service: requests generation (evidence: `README.md`)
- FastAPI service → Results dashboard data: serves dashboard data (evidence: `api/main.py`)
- Search and dashboards UI → Results dashboard data: displays measurements (evidence: `README.md`)
- Retrieval evaluation → Search pipelines: evaluates modes (evidence: `scripts/eval_ir.py`)
- Retrieval evaluation → Ranking metrics: scores rankings (evidence: `scripts/eval_ir.py`)
- Retrieval evaluation → Benchmark results: saves measurements (evidence: `scripts/eval_ir.py`)
- Results dashboard data → Benchmark results: reads measurements (evidence: `precisionrag/dashboard.py`)
