# koduriabhinay/enterprise-rag-pipeline architecture

> Architecture diagram of the GitHub repository koduriabhinay/enterprise-rag-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/koduriabhinay/enterprise-rag-pipeline
- Repository: https://github.com/koduriabhinay/enterprise-rag-pipeline

## Overview

This repository is an offline-first enterprise RAG application: users ingest policy documents, ask questions, and receive answers with citations. The main runtime is orchestrated by `generation/answer.py`, which loads and chunks documents for an in-memory hybrid index, screens questions, retrieves and reranks relevant passages, redacts PII, and generates cited answers. Users can enter through the CLI or HTTP API; an optional configured chat endpoint supplies abstractive answers. Evaluation is a separate documented capability. The README documents the stages; the sampled source confirms their wiring and core implementations, but not every subsystem implementation.

Main entry points: [`generation/answer.py`](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/answer.py) (RAG orchestrator), [`retrieval/pipeline.py`](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/pipeline.py) (Hybrid retriever), [`eval/run_eval.py`](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/eval/run_eval.py) (Evaluation runner), [`cli.py`](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/cli.py) (CLI)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["User interfaces"]
  node_cli["CLI<br/>[cli.py]"]
  node_api["HTTP API<br/>[main.py]"]
end

subgraph group_ingestion["Document ingestion"]
  node_loader["Document loader<br/>[loader.py]"]
  node_chunker["Chunking<br/>[chunker.py]"]
  node_models["Document and chunks<br/>[models.py]"]
  node_corpus[("In-memory chunks<br/>[answer.py]")]
end

subgraph group_retrieval["Hybrid retrieval"]
  node_retriever["Hybrid retriever<br/>[pipeline.py]"]
  node_filters["Metadata filters<br/>[filters.py]"]
  node_bm25["BM25 index<br/>[bm25_index.py]"]
  node_tfidf["Hashed TF-IDF<br/>[tfidf.py]"]
  node_text["Text processing<br/>[text.py]"]
  node_fusion["Rank fusion<br/>[fusion.py]"]
  node_reranker["Heuristic reranker<br/>[reranker.py]"]
end

subgraph group_answering["Answer and safety"]
  node_rag["RAG orchestrator<br/>[answer.py]"]
  node_injection["Injection screening<br/>[injection.py]"]
  node_pii["PII redaction<br/>[pii.py]"]
  node_extractive["Extractive answers<br/>[extractive.py]"]
  node_llm["Optional LLM<br/>[llm.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_runner["Evaluation runner<br/>[run_eval.py]"]
  node_eval_metrics["Evaluation metrics<br/>[metrics.py]"]
  node_eval_data["Evaluation questions<br/>[eval_qa.json]"]
end

node_user(("User"))
node_documents["Policy documents"]
node_llm_service(("Chat completion service"))

node_user -->|"runs commands"| node_cli
node_user -->|"sends requests"| node_api
node_cli -->|"calls"| node_rag
node_api -->|"calls"| node_rag
node_documents -->|"provides files"| node_loader
node_rag -->|"loads documents"| node_loader
node_rag -->|"chunks documents"| node_chunker
node_loader -->|"creates documents"| node_models
node_chunker -->|"creates chunks"| node_models
node_rag -->|"updates chunks"| node_corpus
node_rag -->|"builds and queries"| node_retriever
node_retriever -->|"indexes chunks"| node_corpus
node_retriever -->|"filters candidates"| node_filters
node_retriever -->|"searches terms"| node_bm25
node_retriever -->|"vectorizes and searches"| node_tfidf
node_bm25 -->|"tokenizes text"| node_text
node_tfidf -->|"tokenizes text"| node_text
node_retriever -->|"fuses rankings"| node_fusion
node_retriever -->|"reranks shortlist"| node_reranker
node_rag -->|"screens question"| node_injection
node_rag -->|"redacts context and answer"| node_pii
node_rag -->|"generates cited answer"| node_extractive
node_rag -.->|"optionally generates"| node_llm
node_llm -.->|"requests completion"| node_llm_service
node_cli -->|"runs evaluation"| node_eval_runner
node_eval_runner -->|"reads questions"| node_eval_data
node_eval_runner -->|"evaluates answers"| node_rag
node_eval_runner -->|"scores results"| node_eval_metrics
node_eval_metrics -->|"retrieves evaluation context"| node_retriever

click node_documents "https://github.com/koduriabhinay/enterprise-rag-pipeline/tree/main/data_samples/policies"
click node_cli "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/cli.py"
click node_api "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/api/main.py"
click node_rag "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/answer.py"
click node_loader "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/ingest/loader.py"
click node_chunker "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/ingest/chunker.py"
click node_models "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/ingest/models.py"
click node_corpus "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/answer.py"
click node_retriever "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/pipeline.py"
click node_filters "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/filters.py"
click node_bm25 "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/bm25_index.py"
click node_tfidf "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/tfidf.py"
click node_text "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/text.py"
click node_fusion "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/fusion.py"
click node_reranker "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/reranker.py"
click node_injection "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/guardrails/injection.py"
click node_pii "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/guardrails/pii.py"
click node_extractive "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/extractive.py"
click node_llm "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/llm.py"
click node_eval_runner "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/eval/run_eval.py"
click node_eval_metrics "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/eval/metrics.py"
click node_eval_data "https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/data_samples/eval_qa.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_api,node_user toneBlue
class node_loader,node_chunker,node_models,node_corpus toneAmber
class node_retriever,node_filters,node_bm25,node_tfidf,node_text,node_fusion,node_reranker toneMint
class node_rag,node_injection,node_pii,node_extractive,node_llm toneRose
class node_eval_runner,node_eval_metrics,node_eval_data,node_llm_service toneIndigo
class node_documents toneTeal
```

## Components

### User interfaces

- [CLI](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/cli.py): `cli.py`
- [HTTP API](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/api/main.py): `api/main.py`

### Document ingestion

- [Document loader](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/ingest/loader.py): `ingest/loader.py`
- [Chunking](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/ingest/chunker.py): `ingest/chunker.py`
- [Document and chunks](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/ingest/models.py): `ingest/models.py`
- [In-memory chunks](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/answer.py): `generation/answer.py`

### Hybrid retrieval

- [Hybrid retriever](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/pipeline.py): `retrieval/pipeline.py`
- [Metadata filters](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/filters.py): `retrieval/filters.py`
- [BM25 index](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/bm25_index.py): `retrieval/bm25_index.py`
- [Hashed TF-IDF](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/tfidf.py): `retrieval/tfidf.py`
- [Text processing](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/text.py): `retrieval/text.py`
- [Rank fusion](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/fusion.py): `retrieval/fusion.py`
- [Heuristic reranker](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/retrieval/reranker.py): `retrieval/reranker.py`

### Answer and safety

- [RAG orchestrator](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/answer.py): `generation/answer.py`
- [Injection screening](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/guardrails/injection.py): `guardrails/injection.py`
- [PII redaction](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/guardrails/pii.py): `guardrails/pii.py`
- [Extractive answers](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/extractive.py): `generation/extractive.py`
- [Optional LLM](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/generation/llm.py): `generation/llm.py`

### Evaluation

- [Evaluation runner](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/eval/run_eval.py): `eval/run_eval.py`
- [Evaluation metrics](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/eval/metrics.py): `eval/metrics.py`
- [Evaluation questions](https://github.com/koduriabhinay/enterprise-rag-pipeline/blob/main/data_samples/eval_qa.json): `data_samples/eval_qa.json`

### Other components

- **User**
- [Policy documents](https://github.com/koduriabhinay/enterprise-rag-pipeline/tree/main/data_samples/policies): `data_samples/policies`
- **Chat completion service**

## Connections

- User → CLI: runs commands
- User → HTTP API: sends requests
- CLI → RAG orchestrator: calls
- HTTP API → RAG orchestrator: calls
- Policy documents → Document loader: provides files
- RAG orchestrator → Document loader: loads documents
- RAG orchestrator → Chunking: chunks documents
- Document loader → Document and chunks: creates documents
- Chunking → Document and chunks: creates chunks
- RAG orchestrator → In-memory chunks: updates chunks
- RAG orchestrator → Hybrid retriever: builds and queries
- Hybrid retriever → In-memory chunks: indexes chunks
- Hybrid retriever → Metadata filters: filters candidates
- Hybrid retriever → BM25 index: searches terms
- Hybrid retriever → Hashed TF-IDF: vectorizes and searches
- BM25 index → Text processing: tokenizes text
- Hashed TF-IDF → Text processing: tokenizes text
- Hybrid retriever → Rank fusion: fuses rankings
- Hybrid retriever → Heuristic reranker: reranks shortlist
- RAG orchestrator → Injection screening: screens question
- RAG orchestrator → PII redaction: redacts context and answer
- RAG orchestrator → Extractive answers: generates cited answer
- RAG orchestrator → Optional LLM: optionally generates
- Optional LLM → Chat completion service: requests completion
- CLI → Evaluation runner: runs evaluation
- Evaluation runner → Evaluation questions: reads questions
- Evaluation runner → RAG orchestrator: evaluates answers
- Evaluation runner → Evaluation metrics: scores results
- Evaluation metrics → Hybrid retriever: retrieves evaluation context