How manyruumpus/privacy-aware-retrieval-for-secure-rag-systems- works
SecureRAG-Bench is a research benchmark for comparing baseline vector RAG, access-controlled retrieval, and heuristic query sanitization on synthetic data. A caller identity and question enter the guarded pipeline; retrieval selects documents, context is sent to local Ollama generation, and results are scored for privacy and utility. Separate dataset builders and development/held-out experiment runners support the benchmark. The runtime and selected scripts are evidenced by excerpts and the source index; other documented capabilities are represented at their existing script paths without assuming their internal wiring.
Main entry points: guarded_pipeline.py Guarded pipeline · pipeline.py RAG pipeline · run_heldout_v2.py Held-out runner · retrieval.py Vector retrieval
Components 16
Benchmark data
- Corpus builder
scripts/build_corpus.py - Query builder
scripts/build_evaluation_queries.py - Holdout builder
scripts/build_heldout.py
RAG runtime
- Query guard
src/secure_rag/query_guard.py - Guarded pipeline
src/secure_rag/guarded_pipeline.py - RAG pipeline
src/secure_rag/pipeline.py - Vector retrieval
src/secure_rag/retrieval.py
Experiments and evaluation
- Interactive CLI
scripts/chat.py - Development runner
scripts/run_experiment.py - V2 runner
scripts/run_experiment_v2.py - Held-out runner
scripts/run_heldout_v2.py - Answer audit
scripts/audit_answer_scoring.py - Metrics
scripts/compute_metrics.py
External models
- Sentence transformer
- Ollama
Other components
- Caller
Connections 14
- Caller → Guarded pipeline: submits query
README.md - Guarded pipeline → Query guard: sanitizes query
src/secure_rag/guarded_pipeline.py - Guarded pipeline → RAG pipeline: delegates run
src/secure_rag/guarded_pipeline.py - RAG pipeline → Vector retrieval: searches documents
src/secure_rag/pipeline.py - Vector retrieval → Sentence transformer: encodes text
src/secure_rag/retrieval.py - RAG pipeline → Ollama: requests generation
src/secure_rag/pipeline.py - Corpus builder → Query builder: supplies corpus
- Holdout builder → Query builder: reuses builder
scripts/build_heldout.py - Interactive CLI → Guarded pipeline: runs queries
- Development runner → Guarded pipeline: runs experiments
- V2 runner → Guarded pipeline: runs variants
- Held-out runner → Guarded pipeline: runs holdout
scripts/run_heldout_v2.py - Held-out runner → V2 runner: uses prompt pilot
scripts/run_heldout_v2.py - Answer audit → Metrics: audits scores