# divyyadav007/advanced-nlp-search-retrieval-engine architecture

> Architecture diagram of the GitHub repository divyyadav007/advanced-nlp-search-retrieval-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/divyyadav007/advanced-nlp-search-retrieval-engine
- Repository: https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine

## Overview

This is a document-grounded hybrid RAG search and question-answering product, exposed through a Streamlit interface and a FastAPI service. Users upload documents for parsing, chunking, deduplication, and dual indexing; queries retrieve and fuse sparse and semantic matches, rerank them, generate an answer, and audit its citations. The README documents PDF, Markdown, HTML, and TXT support and an evaluation capability. The main API and UI excerpts show orchestration and an optional HTTP mode; unsampled details are represented only where documented.

Main entry points: [`backend/app/main.py`](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/main.py) (FastAPI Service), [`backend/app/indexing/hybrid_retriever.py`](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/hybrid_retriever.py) (RRF Retriever), [`backend/app/generation/generator.py`](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/generation/generator.py) (Grounded Generator), [`backend/app/indexing/dense.py`](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/dense.py) (Chroma Vector Index)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["User Interfaces"]
  node_ui["Streamlit UI<br/>[app.py]"]
  node_api["FastAPI Service<br/>[main.py]"]
end

subgraph group_ingestion["Document Ingestion"]
  node_parser["Document Parser<br/>[parsers.py]"]
  node_chunker["Chunking Engine<br/>[chunkers.py]"]
  node_dedup["Chunk Deduplicator<br/>[deduplicator.py]"]
  node_schemas["Document and Chunk Models<br/>[schemas.py]"]
end

subgraph group_retrieval["Retrieval and Ranking"]
  node_sparse[("BM25 Index<br/>[sparse.py]")]
  node_dense[("Chroma Vector Index<br/>[dense.py]")]
  node_hybrid["RRF Retriever"]
  node_reranker["Cross-Encoder Reranker<br/>[cross_encoder.py]"]
end

subgraph group_answers["Answer Quality"]
  node_generator["Grounded Generator<br/>[generator.py]"]
  node_verifier["Citation Verifier<br/>[verifier.py]"]
  node_evaluation["RAG Evaluation<br/>[metrics_runner.py]"]
  node_config["Engine Configuration<br/>[config.py]"]
end

node_user(("User"))
node_source["Source Documents"]
node_groq{{"Groq LLM"}}

node_user -->|"asks or uploads"| node_ui
node_source -->|"uploads"| node_ui
node_ui -->|"HTTP requests"| node_api
node_api -->|"parses files"| node_parser
node_api -->|"chunks documents"| node_chunker
node_api -->|"deduplicates chunks"| node_dedup
node_api -->|"indexes chunks"| node_sparse
node_api -->|"indexes chunks"| node_dense
node_parser -->|"creates documents"| node_schemas
node_chunker -->|"creates chunks"| node_schemas
node_dedup -.->|"embeds chunks"| node_dense
node_api -->|"retrieves candidates"| node_hybrid
node_hybrid -->|"searches keywords"| node_sparse
node_hybrid -->|"searches vectors"| node_dense
node_api -->|"reranks candidates"| node_reranker
node_api -->|"generates answer"| node_generator
node_generator -->|"requests completion"| node_groq
node_api -->|"audits citations"| node_verifier
node_evaluation -->|"evaluates retrieval"| node_hybrid
node_evaluation -->|"evaluates ranking"| node_reranker
node_evaluation -->|"evaluates answers"| node_generator
node_config -->|"configures index"| node_sparse
node_config -->|"configures index"| node_dense
node_config -->|"configures model"| node_generator

click node_ui "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/frontend/src/app.py"
click node_api "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/main.py"
click node_parser "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/parsers.py"
click node_chunker "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/chunkers.py"
click node_dedup "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/deduplicator.py"
click node_schemas "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/schemas.py"
click node_sparse "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/sparse.py"
click node_dense "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/dense.py"
click node_hybrid "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/hybrid_retriever.py"
click node_reranker "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/reranking/cross_encoder.py"
click node_generator "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/generation/generator.py"
click node_verifier "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/generation/verifier.py"
click node_evaluation "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/evaluation/metrics_runner.py"
click node_config "https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/config.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_ui,node_api,node_user toneBlue
class node_parser,node_chunker,node_dedup,node_schemas toneAmber
class node_sparse,node_dense,node_hybrid,node_reranker toneMint
class node_generator,node_verifier,node_evaluation,node_config toneRose
class node_source,node_groq toneIndigo
```

## Components

### User Interfaces

- [Streamlit UI](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/frontend/src/app.py): `frontend/src/app.py`
- [FastAPI Service](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/main.py): `backend/app/main.py`

### Document Ingestion

- [Document Parser](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/parsers.py): `backend/app/ingestion/parsers.py`
- [Chunking Engine](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/chunkers.py): `backend/app/ingestion/chunkers.py`
- [Chunk Deduplicator](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/deduplicator.py): `backend/app/ingestion/deduplicator.py`
- [Document and Chunk Models](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/ingestion/schemas.py): `backend/app/ingestion/schemas.py`

### Retrieval and Ranking

- [BM25 Index](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/sparse.py): `backend/app/indexing/sparse.py`
- [Chroma Vector Index](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/dense.py): `backend/app/indexing/dense.py`
- [RRF Retriever](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/indexing/hybrid_retriever.py): `backend/app/indexing/hybrid_retriever.py`
- [Cross-Encoder Reranker](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/reranking/cross_encoder.py): `backend/app/reranking/cross_encoder.py`

### Answer Quality

- [Grounded Generator](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/generation/generator.py): `backend/app/generation/generator.py`
- [Citation Verifier](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/generation/verifier.py): `backend/app/generation/verifier.py`
- [RAG Evaluation](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/evaluation/metrics_runner.py): `backend/app/evaluation/metrics_runner.py`
- [Engine Configuration](https://github.com/divyyadav007/advanced-nlp-search-retrieval-engine/blob/main/backend/app/config.py): `backend/app/config.py`

### Other components

- **User**
- **Source Documents**
- **Groq LLM**

## Connections

- User → Streamlit UI: asks or uploads (evidence: `frontend/src/app.py`)
- Source Documents → Streamlit UI: uploads (evidence: `frontend/src/app.py`)
- Streamlit UI → FastAPI Service: HTTP requests (evidence: `frontend/src/app.py`)
- FastAPI Service → Document Parser: parses files (evidence: `backend/app/main.py`)
- FastAPI Service → Chunking Engine: chunks documents (evidence: `backend/app/main.py`)
- FastAPI Service → Chunk Deduplicator: deduplicates chunks (evidence: `README.md`)
- FastAPI Service → BM25 Index: indexes chunks (evidence: `backend/app/main.py`)
- FastAPI Service → Chroma Vector Index: indexes chunks (evidence: `backend/app/main.py`)
- Document Parser → Document and Chunk Models: creates documents (evidence: `backend/app/ingestion/parsers.py`)
- Chunking Engine → Document and Chunk Models: creates chunks (evidence: `backend/app/ingestion/chunkers.py`)
- Chunk Deduplicator → Chroma Vector Index: embeds chunks
- FastAPI Service → RRF Retriever: retrieves candidates (evidence: `backend/app/main.py`)
- RRF Retriever → BM25 Index: searches keywords (evidence: `backend/app/indexing/hybrid_retriever.py`)
- RRF Retriever → Chroma Vector Index: searches vectors (evidence: `backend/app/indexing/hybrid_retriever.py`)
- FastAPI Service → Cross-Encoder Reranker: reranks candidates (evidence: `backend/app/main.py`)
- FastAPI Service → Grounded Generator: generates answer (evidence: `backend/app/main.py`)
- Grounded Generator → Groq LLM: requests completion (evidence: `backend/app/generation/generator.py`)
- FastAPI Service → Citation Verifier: audits citations (evidence: `backend/app/main.py`)
- RAG Evaluation → RRF Retriever: evaluates retrieval (evidence: `backend/app/evaluation/metrics_runner.py`)
- RAG Evaluation → Cross-Encoder Reranker: evaluates ranking (evidence: `backend/app/evaluation/metrics_runner.py`)
- RAG Evaluation → Grounded Generator: evaluates answers (evidence: `backend/app/evaluation/metrics_runner.py`)
- Engine Configuration → BM25 Index: configures index (evidence: `backend/app/indexing/sparse.py`)
- Engine Configuration → Chroma Vector Index: configures index (evidence: `backend/app/indexing/dense.py`)
- Engine Configuration → Grounded Generator: configures model (evidence: `backend/app/generation/generator.py`)
