How ubaidur404786/secure-rag-benchmark works
Secure RAG is a help-desk chatbot that retrieves company documents and generates answers, with a baseline mode and a guarded mode. The user asks through the Streamlit demo, which builds both RAG graphs over a shared vector store and invokes the selected graph. In guarded mode, input checks can refuse a request; role-filtered retrieval and context cleaning precede generation, then output checks protect the answer. A separate LiteLLM gateway adds shared guardrails, caching, fallback and call logging for applications routed through it. Edges reflect shown calls and README claims; unsampled benchmark and NeMo notebook internals are not expanded.
Main entry points: secure_rag.py Secure RAG graph · rag_core.py In-memory vector store · app.py Streamlit demo · gateway.py LLM gateway
Components 18
User interface
- Streamlit demo
app.py
Knowledge base
- Public documents
fake_docs.py - Role-restricted documents
confidential_docs.py - Poisoned wiki document
poisoned_docs.py - Users and role access
confidential_docs.py - In-memory vector store
rag_core.py
RAG and security
- Baseline RAG graph
rag_core.py - Secure RAG graph
secure_rag.py - Input checks
security.py - Role-filtered retrieval
security.py - Context cleaning
security.py - Output checks
security.py - Model provider
llm_provider.py - Attack and normal cases
attacks.py
Model gateway
- LLM gateway
gateway.py - LiteLLM model router
gateway.py
Other components
- Help-desk user
- Ollama or OpenAI
Connections 22
- Help-desk user → Streamlit demo: asks
app.py - Streamlit demo → Users and role access: offers role choice
app.py - Streamlit demo → In-memory vector store: builds shared store
app.py - Streamlit demo → Baseline RAG graph: builds and invokes
app.py - Streamlit demo → Secure RAG graph: builds and invokes
app.py - Public documents → In-memory vector store: indexed with
rag_core.py - Role-restricted documents → In-memory vector store: indexed with
rag_core.py - Poisoned wiki document → In-memory vector store: indexed with
rag_core.py - Users and role access → Role-filtered retrieval: defines permissions
security.py - Baseline RAG graph → In-memory vector store: searches
rag_core.py - Baseline RAG graph → Model provider: gets chat model
rag_core.py - Secure RAG graph → Input checks: checks input
secure_rag.py - Secure RAG graph → Role-filtered retrieval: filters retrieval
secure_rag.py - Secure RAG graph → In-memory vector store: searches
secure_rag.py - Secure RAG graph → Context cleaning: cleans context
secure_rag.py - Secure RAG graph → Model provider: gets chat model
secure_rag.py - Secure RAG graph → Output checks: checks answer
secure_rag.py - Model provider → Ollama or OpenAI: creates model clients
llm_provider.py - LLM gateway → LiteLLM model router: routes model calls
gateway.py - LLM gateway → Input checks: checks inputs
gateway.py - LLM gateway → Output checks: checks outputs
gateway.py - LiteLLM model router → Ollama or OpenAI: calls model