How koduriabhinay/enterprise-rag-pipeline works
This repository is an offline-first enterprise RAG application: users ingest policy documents, ask questions, and receive answers with citations. The main runtime is orchestrated by generation/answer.py, which loads and chunks documents for an in-memory hybrid index, screens questions, retrieves and reranks relevant passages, redacts PII, and generates cited answers. Users can enter through the CLI or HTTP API; an optional configured chat endpoint supplies abstractive answers. Evaluation is a separate documented capability. The README documents the stages; the sampled source confirms their wiring and core implementations, but not every subsystem implementation.
Main entry points: answer.py RAG orchestrator · pipeline.py Hybrid retriever · run_eval.py Evaluation runner · cli.py CLI
Components 24
Document ingestion
- Document loader
ingest/loader.py - Chunking
ingest/chunker.py - Document and chunks
ingest/models.py - In-memory chunks
generation/answer.py
Hybrid retrieval
- Hybrid retriever
retrieval/pipeline.py - Metadata filters
retrieval/filters.py - BM25 index
retrieval/bm25_index.py - Hashed TF-IDF
retrieval/tfidf.py - Text processing
retrieval/text.py - Rank fusion
retrieval/fusion.py - Heuristic reranker
retrieval/reranker.py
Answer and safety
- RAG orchestrator
generation/answer.py - Injection screening
guardrails/injection.py - PII redaction
guardrails/pii.py - Extractive answers
generation/extractive.py - Optional LLM
generation/llm.py
Evaluation
- Evaluation runner
eval/run_eval.py - Evaluation metrics
eval/metrics.py - Evaluation questions
data_samples/eval_qa.json
Other components
- User
- Policy documents
data_samples/policies - Chat completion service
Connections 29
- User → CLI: runs commands
- User → HTTP API: sends requests
- CLI → RAG orchestrator: calls
- HTTP API → RAG orchestrator: calls
- Policy documents → Document loader: provides files
- RAG orchestrator → Document loader: loads documents
- RAG orchestrator → Chunking: chunks documents
- Document loader → Document and chunks: creates documents
- Chunking → Document and chunks: creates chunks
- RAG orchestrator → In-memory chunks: updates chunks
- RAG orchestrator → Hybrid retriever: builds and queries
- Hybrid retriever → In-memory chunks: indexes chunks
- Hybrid retriever → Metadata filters: filters candidates
- Hybrid retriever → BM25 index: searches terms
- Hybrid retriever → Hashed TF-IDF: vectorizes and searches
- BM25 index → Text processing: tokenizes text
- Hashed TF-IDF → Text processing: tokenizes text
- Hybrid retriever → Rank fusion: fuses rankings
- Hybrid retriever → Heuristic reranker: reranks shortlist
- RAG orchestrator → Injection screening: screens question
- RAG orchestrator → PII redaction: redacts context and answer
- RAG orchestrator → Extractive answers: generates cited answer
- RAG orchestrator → Optional LLM: optionally generates
- Optional LLM → Chat completion service: requests completion
- CLI → Evaluation runner: runs evaluation
- Evaluation runner → Evaluation questions: reads questions
- Evaluation runner → RAG orchestrator: evaluates answers
- Evaluation runner → Evaluation metrics: scores results
- Evaluation metrics → Hybrid retriever: retrieves evaluation context