# gunjannnn30/rag-pipeline-hybrid-search architecture

> Architecture diagram of the GitHub repository gunjannnn30/rag-pipeline-hybrid-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gunjannnn30/rag-pipeline-hybrid-search
- Repository: https://github.com/gunjannnn30/rag-pipeline-hybrid-search

## Overview

This is a local-first RAG platform for ingesting internal documents, indexing them for hybrid search, and returning grounded answers with citation checks and confidence-based abstention. The main workflow runs from dashboard or API upload through parsing, chunking, embedding, and indexing; questions then pass through hybrid retrieval, generation, verification, and scoring. The README also describes evaluation and a comparison interface. Some API and dashboard excerpts are incomplete, so only documented or directly evidenced connections are shown.

Main entry points: [`src/api/main.py`](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/api/main.py) (FastAPI), [`src/indexing/pipeline.py`](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/pipeline.py) (Indexing Pipeline), [`src/retrieval/hybrid.py`](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/hybrid.py) (Hybrid Retriever), [`src/generation/pipeline.py`](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/pipeline.py) (Generation Pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Interfaces & API"]
  node_dashboard["Streamlit Studio<br/>[app.py]"]
  node_api["FastAPI<br/>[main.py]"]
end

subgraph group_ingest_index["Ingestion &amp; Indexing"]
  node_enterprise["Enterprise Loader"]
  node_loader["File Loader<br/>[loader.py]"]
  node_documents["Documents &amp; Chunks<br/>[models.py]"]
  node_indexing["Indexing Pipeline<br/>[pipeline.py]"]
  node_chunking["Chunking Strategies<br/>[factory.py]"]
  node_embeddings["Embedding Client<br/>[embeddings.py]"]
  node_dedup["Chunk Deduplication<br/>[deduplication.py]"]
  node_vector[("Vector Store<br/>[vector_store.py]")]
  node_bm25[("BM25 Index<br/>[bm25_index.py]")]
end

subgraph group_retrieval["Hybrid Retrieval"]
  node_hybrid["Hybrid Retriever<br/>[hybrid.py]"]
  node_dense["Dense Search<br/>[dense.py]"]
  node_sparse["Sparse Search<br/>[sparse.py]"]
  node_fusion["Rank Fusion<br/>[fusion.py]"]
  node_reranker["Cross-Encoder Reranker<br/>[reranker.py]"]
end

subgraph group_answer["Grounded Answers"]
  node_generation["Generation Pipeline<br/>[pipeline.py]"]
  node_generator["Grounded Generator<br/>[generator.py]"]
  node_citations["Citation Verifier<br/>[citations.py]"]
  node_confidence["Confidence Scorer<br/>[confidence.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_eval_runner["Evaluation Runner<br/>[run_eval.py]"]
  node_eval_dataset["Evaluation Dataset<br/>[dataset.py]"]
  node_eval_metrics["Evaluation Metrics<br/>[metrics.py]"]
end

node_user(("User"))
node_ollama["Ollama"]

node_user -->|"uses"| node_dashboard
node_dashboard -->|"calls endpoints"| node_api
node_api -->|"asks question"| node_generation
node_api -->|"loads corpus"| node_enterprise
node_api -->|"parses uploads"| node_loader
node_enterprise -->|"produces documents"| node_documents
node_loader -->|"produces documents"| node_documents
node_api -->|"indexes documents"| node_indexing
node_indexing -->|"chunks documents"| node_chunking
node_indexing -->|"embeds chunks"| node_embeddings
node_indexing -->|"deduplicates chunks"| node_dedup
node_indexing -->|"writes vectors"| node_vector
node_indexing -->|"builds index"| node_bm25
node_chunking -.->|"embeds sentences"| node_embeddings
node_embeddings -->|"requests embeddings"| node_ollama
node_api -->|"configures retrieval"| node_hybrid
node_generation -->|"retrieves context"| node_hybrid
node_hybrid -->|"dispatches search"| node_dense
node_hybrid -->|"dispatches search"| node_sparse
node_dense -->|"embeds query"| node_embeddings
node_dense -->|"searches vectors"| node_vector
node_sparse -->|"searches terms"| node_bm25
node_hybrid -->|"fuses rankings"| node_fusion
node_hybrid -->|"reranks candidates"| node_reranker
node_generation -->|"generates answer"| node_generator
node_generation -->|"verifies citations"| node_citations
node_generation -->|"scores answer"| node_confidence
node_generator -->|"generates text"| node_ollama
node_citations -->|"checks support"| node_ollama
node_api -->|"reads status and documents"| node_vector
node_api -->|"clears index"| node_bm25
node_eval_runner -->|"loads cases"| node_eval_dataset
node_eval_runner -->|"runs questions"| node_generation
node_eval_runner -->|"evaluates results"| node_eval_metrics
node_eval_metrics -->|"scores answers"| node_ollama

click node_dashboard "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/dashboard/app.py"
click node_api "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/api/main.py"
click node_enterprise "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/ingestion/enterprise_loader.py"
click node_loader "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/ingestion/loader.py"
click node_documents "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/models.py"
click node_indexing "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/pipeline.py"
click node_chunking "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/chunking/factory.py"
click node_embeddings "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/embeddings.py"
click node_dedup "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/deduplication.py"
click node_vector "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/vector_store.py"
click node_bm25 "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/bm25_index.py"
click node_hybrid "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/hybrid.py"
click node_dense "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/dense.py"
click node_sparse "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/sparse.py"
click node_fusion "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/fusion.py"
click node_reranker "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/reranker.py"
click node_generation "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/pipeline.py"
click node_generator "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/generator.py"
click node_citations "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/citations.py"
click node_confidence "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/confidence.py"
click node_eval_runner "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/evaluation/run_eval.py"
click node_eval_dataset "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/evaluation/dataset.py"
click node_eval_metrics "https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/evaluation/metrics.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dashboard,node_api,node_user toneBlue
class node_enterprise,node_loader,node_documents,node_indexing,node_chunking,node_embeddings,node_dedup,node_vector,node_bm25 toneAmber
class node_hybrid,node_dense,node_sparse,node_fusion,node_reranker toneMint
class node_generation,node_generator,node_citations,node_confidence toneRose
class node_eval_runner,node_eval_dataset,node_eval_metrics,node_ollama toneIndigo
```

## Components

### Interfaces & API

- [Streamlit Studio](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/dashboard/app.py): `src/dashboard/app.py`
- [FastAPI](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/api/main.py): `src/api/main.py`

### Ingestion & Indexing

- [Enterprise Loader](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/ingestion/enterprise_loader.py): `src/ingestion/enterprise_loader.py`
- [File Loader](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/ingestion/loader.py): `src/ingestion/loader.py`
- [Documents & Chunks](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/models.py): `src/models.py`
- [Indexing Pipeline](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/pipeline.py): `src/indexing/pipeline.py`
- [Chunking Strategies](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/chunking/factory.py): `src/chunking/factory.py`
- [Embedding Client](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/embeddings.py): `src/indexing/embeddings.py`
- [Chunk Deduplication](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/deduplication.py): `src/indexing/deduplication.py`
- [Vector Store](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/vector_store.py): `src/indexing/vector_store.py`
- [BM25 Index](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/indexing/bm25_index.py): `src/indexing/bm25_index.py`

### Hybrid Retrieval

- [Hybrid Retriever](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/hybrid.py): `src/retrieval/hybrid.py`
- [Dense Search](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/dense.py): `src/retrieval/dense.py`
- [Sparse Search](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/sparse.py): `src/retrieval/sparse.py`
- [Rank Fusion](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/fusion.py): `src/retrieval/fusion.py`
- [Cross-Encoder Reranker](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/retrieval/reranker.py): `src/retrieval/reranker.py`

### Grounded Answers

- [Generation Pipeline](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/pipeline.py): `src/generation/pipeline.py`
- [Grounded Generator](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/generator.py): `src/generation/generator.py`
- [Citation Verifier](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/citations.py): `src/generation/citations.py`
- [Confidence Scorer](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/generation/confidence.py): `src/generation/confidence.py`

### Evaluation

- [Evaluation Runner](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/evaluation/run_eval.py): `src/evaluation/run_eval.py`
- [Evaluation Dataset](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/evaluation/dataset.py): `src/evaluation/dataset.py`
- [Evaluation Metrics](https://github.com/gunjannnn30/rag-pipeline-hybrid-search/blob/main/src/evaluation/metrics.py): `src/evaluation/metrics.py`

### Other components

- **User**
- **Ollama**

## Connections

- User → Streamlit Studio: uses
- Streamlit Studio → FastAPI: calls endpoints
- FastAPI → Generation Pipeline: asks question
- FastAPI → Enterprise Loader: loads corpus
- FastAPI → File Loader: parses uploads
- Enterprise Loader → Documents & Chunks: produces documents
- File Loader → Documents & Chunks: produces documents
- FastAPI → Indexing Pipeline: indexes documents
- Indexing Pipeline → Chunking Strategies: chunks documents
- Indexing Pipeline → Embedding Client: embeds chunks
- Indexing Pipeline → Chunk Deduplication: deduplicates chunks
- Indexing Pipeline → Vector Store: writes vectors
- Indexing Pipeline → BM25 Index: builds index
- Chunking Strategies → Embedding Client: embeds sentences
- Embedding Client → Ollama: requests embeddings
- FastAPI → Hybrid Retriever: configures retrieval
- Generation Pipeline → Hybrid Retriever: retrieves context
- Hybrid Retriever → Dense Search: dispatches search
- Hybrid Retriever → Sparse Search: dispatches search
- Dense Search → Embedding Client: embeds query
- Dense Search → Vector Store: searches vectors
- Sparse Search → BM25 Index: searches terms
- Hybrid Retriever → Rank Fusion: fuses rankings
- Hybrid Retriever → Cross-Encoder Reranker: reranks candidates
- Generation Pipeline → Grounded Generator: generates answer
- Generation Pipeline → Citation Verifier: verifies citations
- Generation Pipeline → Confidence Scorer: scores answer
- Grounded Generator → Ollama: generates text
- Citation Verifier → Ollama: checks support
- FastAPI → Vector Store: reads status and documents
- FastAPI → BM25 Index: clears index
- Evaluation Runner → Evaluation Dataset: loads cases
- Evaluation Runner → Generation Pipeline: runs questions
- Evaluation Runner → Evaluation Metrics: evaluates results
- Evaluation Metrics → Ollama: scores answers