How rhythmgarg13/enterprise-knowledge-base-rag-system works
This repository is an internal knowledge-base RAG application. Users upload documents or load a local manifest, which the pipeline parses, chunks, embeds, and indexes; questions are retrieved with optional role filtering, gated by a confidence threshold, and answered with source citations or a safe fallback. The UI also exposes audit history. Separate CLI paths support manifest reindexing and retrieval recall evaluation. The excerpts establish these major capabilities, but omit some pipeline internals, so uncertain implementation links are not shown.
Main entry points: rag_pipeline.py RAG pipeline · app.py Streamlit app · manifest_loader.py Manifest loader · eval_retrieval.py Recall@k evaluation
Components 22
User interface
- Streamlit app
app.py
Knowledge ingestion
- Manifest loader
manifest_loader.py - Markdown sections
markdown_loader.py - RAG pipeline
rag_pipeline.py - Chunking and embeddings
rag_pipeline.py - Vector index
rag_pipeline.py
Question answering
- Role-filtered retrieval
rag_pipeline.py - Confidence gate
rag_pipeline.py - Grounded answer and citations
rag_pipeline.py
Audit and sync
- Q&A JSONL log
rag_pipeline.py - SQLite audit and sync state
audit_sqlite.py - Manifest sync CLI
sync_cli.py
Retrieval evaluation
- Recall@k evaluation
eval_retrieval.py - Evaluation report
eval_retrieval.py
Other components
- Knowledge user
- Knowledge admin
- Local documents
- Uploaded files
- Question
- OpenAI or Gemini
- Embedding provider
- Golden questions
data/golden.sample.jsonl
Connections 29
- Knowledge user → Streamlit app: asks or uploads
- Knowledge admin → Streamlit app: reviews audit
- Streamlit app → RAG pipeline: submits uploads
- Streamlit app → Manifest loader: loads manifest
- Manifest loader → Local documents: resolves paths
- Manifest loader → RAG pipeline: passes document metadata
- Uploaded files → RAG pipeline: passes uploaded files
- RAG pipeline → Markdown sections: loads Markdown sections
- RAG pipeline → Chunking and embeddings: splits documents
- Chunking and embeddings → Embedding provider: requests embeddings
- Chunking and embeddings → Vector index: indexes chunks
- Streamlit app → RAG pipeline: submits question
- RAG pipeline → Role-filtered retrieval: retrieves for question
- Role-filtered retrieval → Vector index: searches index
- RAG pipeline → Confidence gate: checks retrieval score
- Confidence gate → Grounded answer and citations: allows grounded response
- Confidence gate → RAG pipeline: returns fallback
- Grounded answer and citations → OpenAI or Gemini: requests answer
- RAG pipeline → Q&A JSONL log: appends Q&A record
- RAG pipeline → SQLite audit and sync state: stores Q&A record
- RAG pipeline → Streamlit app: returns answer and sources
- Streamlit app → SQLite audit and sync state: reads recent logs
- Manifest sync CLI → Manifest loader: loads manifest
- Manifest sync CLI → RAG pipeline: rebuilds index
- Manifest sync CLI → SQLite audit and sync state: tracks hashes and runs
- Recall@k evaluation → Manifest loader: loads manifest
- Recall@k evaluation → RAG pipeline: loads documents and retrieves
- Golden questions → Recall@k evaluation: supplies questions
- Recall@k evaluation → Evaluation report: writes recall results