How gunjannnn30/rag-pipeline-hybrid-search works
This is a local-first RAG platform for ingesting internal documents, indexing them for hybrid search, and returning grounded answers with citation checks and confidence-based abstention. The main workflow runs from dashboard or API upload through parsing, chunking, embedding, and indexing; questions then pass through hybrid retrieval, generation, verification, and scoring. The README also describes evaluation and a comparison interface. Some API and dashboard excerpts are incomplete, so only documented or directly evidenced connections are shown.
Main entry points: main.py FastAPI · indexing/pipeline.py Indexing Pipeline · hybrid.py Hybrid Retriever · generation/pipeline.py Generation Pipeline
Components 25
Interfaces & API
- Streamlit Studio
src/dashboard/app.py - FastAPI
src/api/main.py
Ingestion & Indexing
- Enterprise Loader
src/ingestion/enterprise_loader.py - File Loader
src/ingestion/loader.py - Documents & Chunks
src/models.py - Indexing Pipeline
src/indexing/pipeline.py - Chunking Strategies
src/chunking/factory.py - Embedding Client
src/indexing/embeddings.py - Chunk Deduplication
src/indexing/deduplication.py - Vector Store
src/indexing/vector_store.py - BM25 Index
src/indexing/bm25_index.py
Hybrid Retrieval
- Hybrid Retriever
src/retrieval/hybrid.py - Dense Search
src/retrieval/dense.py - Sparse Search
src/retrieval/sparse.py - Rank Fusion
src/retrieval/fusion.py - Cross-Encoder Reranker
src/retrieval/reranker.py
Grounded Answers
- Generation Pipeline
src/generation/pipeline.py - Grounded Generator
src/generation/generator.py - Citation Verifier
src/generation/citations.py - Confidence Scorer
src/generation/confidence.py
Evaluation
- Evaluation Runner
src/evaluation/run_eval.py - Evaluation Dataset
src/evaluation/dataset.py - Evaluation Metrics
src/evaluation/metrics.py
Other components
- User
- Ollama
Connections 35
- User → Streamlit Studio: uses
- Streamlit Studio → FastAPI: calls endpoints
- FastAPI → Generation Pipeline: asks question
- FastAPI → Enterprise Loader: loads corpus
- FastAPI → File Loader: parses uploads
- Enterprise Loader → Documents & Chunks: produces documents
- File Loader → Documents & Chunks: produces documents
- FastAPI → Indexing Pipeline: indexes documents
- Indexing Pipeline → Chunking Strategies: chunks documents
- Indexing Pipeline → Embedding Client: embeds chunks
- Indexing Pipeline → Chunk Deduplication: deduplicates chunks
- Indexing Pipeline → Vector Store: writes vectors
- Indexing Pipeline → BM25 Index: builds index
- Chunking Strategies → Embedding Client: embeds sentences
- Embedding Client → Ollama: requests embeddings
- FastAPI → Hybrid Retriever: configures retrieval
- Generation Pipeline → Hybrid Retriever: retrieves context
- Hybrid Retriever → Dense Search: dispatches search
- Hybrid Retriever → Sparse Search: dispatches search
- Dense Search → Embedding Client: embeds query
- Dense Search → Vector Store: searches vectors
- Sparse Search → BM25 Index: searches terms
- Hybrid Retriever → Rank Fusion: fuses rankings
- Hybrid Retriever → Cross-Encoder Reranker: reranks candidates
- Generation Pipeline → Grounded Generator: generates answer
- Generation Pipeline → Citation Verifier: verifies citations
- Generation Pipeline → Confidence Scorer: scores answer
- Grounded Generator → Ollama: generates text
- Citation Verifier → Ollama: checks support
- FastAPI → Vector Store: reads status and documents
- FastAPI → BM25 Index: clears index
- Evaluation Runner → Evaluation Dataset: loads cases
- Evaluation Runner → Generation Pipeline: runs questions
- Evaluation Runner → Evaluation Metrics: evaluates results
- Evaluation Metrics → Ollama: scores answers