# rhythmgarg13/enterprise-knowledge-base-rag-system architecture

> Architecture diagram of the GitHub repository rhythmgarg13/enterprise-knowledge-base-rag-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rhythmgarg13/enterprise-knowledge-base-rag-system
- Repository: https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system

## Overview

This repository is an internal knowledge-base RAG application. Users upload documents or load a local manifest, which the pipeline parses, chunks, embeds, and indexes; questions are retrieved with optional role filtering, gated by a confidence threshold, and answered with source citations or a safe fallback. The UI also exposes audit history. Separate CLI paths support manifest reindexing and retrieval recall evaluation. The excerpts establish these major capabilities, but omit some pipeline internals, so uncertain implementation links are not shown.

Main entry points: [`rag_pipeline.py`](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py) (RAG pipeline), [`app.py`](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/app.py) (Streamlit app), [`manifest_loader.py`](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/manifest_loader.py) (Manifest loader), [`eval_retrieval.py`](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/eval_retrieval.py) (Recall@k evaluation)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User interface"]
  node_app["Streamlit app<br/>[app.py]"]
end

subgraph group_ingestion["Knowledge ingestion"]
  node_manifest["Manifest loader<br/>[manifest_loader.py]"]
  node_markdown["Markdown sections<br/>[markdown_loader.py]"]
  node_pipeline["RAG pipeline<br/>[rag_pipeline.py]"]
  node_chunking["Chunking and embeddings<br/>[rag_pipeline.py]"]
  node_vectorstore[("Vector index<br/>[rag_pipeline.py]")]
end

subgraph group_qa["Question answering"]
  node_retrieval["Role-filtered retrieval<br/>[rag_pipeline.py]"]
  node_safety["Confidence gate<br/>[rag_pipeline.py]"]
  node_generation["Grounded answer and citations<br/>[rag_pipeline.py]"]
end

subgraph group_operations["Audit and sync"]
  node_jsonl[("Q&amp;A JSONL log<br/>[rag_pipeline.py]")]
  node_audit_db[("SQLite audit and sync state<br/>[audit_sqlite.py]")]
  node_sync["Manifest sync CLI<br/>[sync_cli.py]"]
end

subgraph group_evaluation["Retrieval evaluation"]
  node_evaluation["Recall@k evaluation<br/>[eval_retrieval.py]"]
  node_report["Evaluation report<br/>[eval_retrieval.py]"]
end

node_user(("Knowledge user"))
node_admin(("Knowledge admin"))
node_source_docs["Local documents"]
node_uploads["Uploaded files"]
node_question["Question"]
node_llm{{"OpenAI or Gemini"}}
node_embedding_service{{"Embedding provider"}}
node_golden["Golden questions"]

node_user -->|"asks or uploads"| node_app
node_admin -->|"reviews audit"| node_app
node_app -->|"submits uploads"| node_pipeline
node_app -->|"loads manifest"| node_manifest
node_manifest -->|"resolves paths"| node_source_docs
node_manifest -->|"passes document metadata"| node_pipeline
node_uploads -->|"passes uploaded files"| node_pipeline
node_pipeline -->|"loads Markdown sections"| node_markdown
node_pipeline -->|"splits documents"| node_chunking
node_chunking -.->|"requests embeddings"| node_embedding_service
node_chunking -->|"indexes chunks"| node_vectorstore
node_app -->|"submits question"| node_pipeline
node_pipeline -->|"retrieves for question"| node_retrieval
node_retrieval -->|"searches index"| node_vectorstore
node_pipeline -->|"checks retrieval score"| node_safety
node_safety -->|"allows grounded response"| node_generation
node_safety -->|"returns fallback"| node_pipeline
node_generation -.->|"requests answer"| node_llm
node_pipeline -->|"appends Q&amp;A record"| node_jsonl
node_pipeline -->|"stores Q&amp;A record"| node_audit_db
node_pipeline -->|"returns answer and sources"| node_app
node_app -->|"reads recent logs"| node_audit_db
node_sync -->|"loads manifest"| node_manifest
node_sync -->|"rebuilds index"| node_pipeline
node_sync -->|"tracks hashes and runs"| node_audit_db
node_evaluation -->|"loads manifest"| node_manifest
node_evaluation -->|"loads documents and retrieves"| node_pipeline
node_golden -->|"supplies questions"| node_evaluation
node_evaluation -->|"writes recall results"| node_report

click node_app "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/app.py"
click node_manifest "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/manifest_loader.py"
click node_markdown "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/markdown_loader.py"
click node_pipeline "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_chunking "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_vectorstore "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_retrieval "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_safety "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_generation "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_jsonl "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py"
click node_audit_db "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/audit_sqlite.py"
click node_sync "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/sync_cli.py"
click node_golden "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/data/golden.sample.jsonl"
click node_evaluation "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/eval_retrieval.py"
click node_report "https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/eval_retrieval.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_user toneBlue
class node_manifest,node_markdown,node_pipeline,node_chunking,node_vectorstore toneAmber
class node_retrieval,node_safety,node_generation toneMint
class node_jsonl,node_audit_db,node_sync toneRose
class node_evaluation,node_report,node_admin,node_source_docs,node_uploads,node_question,node_llm,node_embedding_service toneIndigo
class node_golden toneTeal
```

## Components

### User interface

- [Streamlit app](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/app.py): `app.py`

### Knowledge ingestion

- [Manifest loader](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/manifest_loader.py): `manifest_loader.py`
- [Markdown sections](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/markdown_loader.py): `markdown_loader.py`
- [RAG pipeline](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`
- [Chunking and embeddings](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`
- [Vector index](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`

### Question answering

- [Role-filtered retrieval](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`
- [Confidence gate](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`
- [Grounded answer and citations](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`

### Audit and sync

- [Q&A JSONL log](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/rag_pipeline.py): `rag_pipeline.py`
- [SQLite audit and sync state](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/audit_sqlite.py): `audit_sqlite.py`
- [Manifest sync CLI](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/sync_cli.py): `sync_cli.py`

### Retrieval evaluation

- [Recall@k evaluation](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/eval_retrieval.py): `eval_retrieval.py`
- [Evaluation report](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/eval_retrieval.py): `eval_retrieval.py`

### Other components

- **Knowledge user**
- **Knowledge admin**
- **Local documents**
- **Uploaded files**
- **Question**
- **OpenAI or Gemini**
- **Embedding provider**
- [Golden questions](https://github.com/rhythmgarg13/enterprise-knowledge-base-rag-system/blob/main/data/golden.sample.jsonl): `data/golden.sample.jsonl`

## Connections

- Knowledge user → Streamlit app: asks or uploads
- Knowledge admin → Streamlit app: reviews audit
- Streamlit app → RAG pipeline: submits uploads
- Streamlit app → Manifest loader: loads manifest
- Manifest loader → Local documents: resolves paths
- Manifest loader → RAG pipeline: passes document metadata
- Uploaded files → RAG pipeline: passes uploaded files
- RAG pipeline → Markdown sections: loads Markdown sections
- RAG pipeline → Chunking and embeddings: splits documents
- Chunking and embeddings → Embedding provider: requests embeddings
- Chunking and embeddings → Vector index: indexes chunks
- Streamlit app → RAG pipeline: submits question
- RAG pipeline → Role-filtered retrieval: retrieves for question
- Role-filtered retrieval → Vector index: searches index
- RAG pipeline → Confidence gate: checks retrieval score
- Confidence gate → Grounded answer and citations: allows grounded response
- Confidence gate → RAG pipeline: returns fallback
- Grounded answer and citations → OpenAI or Gemini: requests answer
- RAG pipeline → Q&A JSONL log: appends Q&A record
- RAG pipeline → SQLite audit and sync state: stores Q&A record
- RAG pipeline → Streamlit app: returns answer and sources
- Streamlit app → SQLite audit and sync state: reads recent logs
- Manifest sync CLI → Manifest loader: loads manifest
- Manifest sync CLI → RAG pipeline: rebuilds index
- Manifest sync CLI → SQLite audit and sync state: tracks hashes and runs
- Recall@k evaluation → Manifest loader: loads manifest
- Recall@k evaluation → RAG pipeline: loads documents and retrieves
- Golden questions → Recall@k evaluation: supplies questions
- Recall@k evaluation → Evaluation report: writes recall results