How divyyadav007/advanced-nlp-search-retrieval-engine works
This is a document-grounded hybrid RAG search and question-answering product, exposed through a Streamlit interface and a FastAPI service. Users upload documents for parsing, chunking, deduplication, and dual indexing; queries retrieve and fuse sparse and semantic matches, rerank them, generate an answer, and audit its citations. The README documents PDF, Markdown, HTML, and TXT support and an evaluation capability. The main API and UI excerpts show orchestration and an optional HTTP mode; unsampled details are represented only where documented.
Main entry points: main.py FastAPI Service · hybrid_retriever.py RRF Retriever · generator.py Grounded Generator · dense.py Chroma Vector Index
Components 17
User Interfaces
- Streamlit UI
frontend/src/app.py - FastAPI Service
backend/app/main.py
Document Ingestion
- Document Parser
backend/app/ingestion/parsers.py - Chunking Engine
backend/app/ingestion/chunkers.py - Chunk Deduplicator
backend/app/ingestion/deduplicator.py - Document and Chunk Models
backend/app/ingestion/schemas.py
Retrieval and Ranking
- BM25 Index
backend/app/indexing/sparse.py - Chroma Vector Index
backend/app/indexing/dense.py - RRF Retriever
backend/app/indexing/hybrid_retriever.py - Cross-Encoder Reranker
backend/app/reranking/cross_encoder.py
Answer Quality
- Grounded Generator
backend/app/generation/generator.py - Citation Verifier
backend/app/generation/verifier.py - RAG Evaluation
backend/app/evaluation/metrics_runner.py - Engine Configuration
backend/app/config.py
Other components
- User
- Source Documents
- Groq LLM
Connections 24
- User → Streamlit UI: asks or uploads
frontend/src/app.py - Source Documents → Streamlit UI: uploads
frontend/src/app.py - Streamlit UI → FastAPI Service: HTTP requests
frontend/src/app.py - FastAPI Service → Document Parser: parses files
backend/app/main.py - FastAPI Service → Chunking Engine: chunks documents
backend/app/main.py - FastAPI Service → Chunk Deduplicator: deduplicates chunks
README.md - FastAPI Service → BM25 Index: indexes chunks
backend/app/main.py - FastAPI Service → Chroma Vector Index: indexes chunks
backend/app/main.py - Document Parser → Document and Chunk Models: creates documents
backend/app/ingestion/parsers.py - Chunking Engine → Document and Chunk Models: creates chunks
backend/app/ingestion/chunkers.py - Chunk Deduplicator → Chroma Vector Index: embeds chunks
- FastAPI Service → RRF Retriever: retrieves candidates
backend/app/main.py - RRF Retriever → BM25 Index: searches keywords
backend/app/indexing/hybrid_retriever.py - RRF Retriever → Chroma Vector Index: searches vectors
backend/app/indexing/hybrid_retriever.py - FastAPI Service → Cross-Encoder Reranker: reranks candidates
backend/app/main.py - FastAPI Service → Grounded Generator: generates answer
backend/app/main.py - Grounded Generator → Groq LLM: requests completion
backend/app/generation/generator.py - FastAPI Service → Citation Verifier: audits citations
backend/app/main.py - RAG Evaluation → RRF Retriever: evaluates retrieval
backend/app/evaluation/metrics_runner.py - RAG Evaluation → Cross-Encoder Reranker: evaluates ranking
backend/app/evaluation/metrics_runner.py - RAG Evaluation → Grounded Generator: evaluates answers
backend/app/evaluation/metrics_runner.py - Engine Configuration → BM25 Index: configures index
backend/app/indexing/sparse.py - Engine Configuration → Chroma Vector Index: configures index
backend/app/indexing/dense.py - Engine Configuration → Grounded Generator: configures model
backend/app/generation/generator.py