# shubham96681/lexical-search-bot architecture

> Architecture diagram of the GitHub repository shubham96681/lexical-search-bot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shubham96681/lexical-search-bot
- Repository: https://github.com/shubham96681/lexical-search-bot

## Overview

This is a Ranger knowledge-base chatbot. A user asks a question in the web client; the FastAPI server retrieves passages through the knowledge-base engine, which combines BM25 lexical search with optional OpenAI embeddings and reciprocal-rank fusion. The server streams sources and either a plain passage-based response or an OpenAI-generated answer. Separate flows configure embeddings and manage uploaded documents, which are parsed into chunks and reindexed alongside the bundled Ranger corpus. The README documents the broader Ranger policy and guidance corpus; unsampled source details are not assumed.

Main entry points: [`kb/engine.py`](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py) (Knowledge engine), [`kb/corpus.py`](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/corpus.py) (Corpus loader and chunker), [`server.py`](https://github.com/shubham96681/lexical-search-bot/blob/main/server.py) (HTTP API), [`kb/uploads.py`](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/uploads.py) (Upload processing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interface["Web interface"]
  node_webclient["Chat and controls<br/>[app.js]"]
end

subgraph group_service["API service"]
  node_api["HTTP API<br/>[server.py]"]
end

subgraph group_retrieval["Knowledge retrieval"]
  node_engine["Knowledge engine<br/>[engine.py]"]
  node_bm25["BM25 search<br/>[engine.py]"]
  node_fusion["Rank fusion<br/>[engine.py]"]
  node_vector[("Vector index<br/>[vector_store.py]")]
end

subgraph group_documents["Document pipeline"]
  node_corpus["Corpus loader and chunker<br/>[corpus.py]"]
  node_sourcecatalog["Ranger source catalog<br/>[sources.py]"]
  node_rawdocs[("Cached Ranger corpus")]
  node_uploadmanager["Upload processing<br/>[uploads.py]"]
  node_uploaddisk[("Uploaded documents")]
end

node_user(("Chat user"))
node_openai{{"OpenAI API"}}
node_rangerdocs["Apache Ranger sources"]

node_user -->|"asks or configures"| node_webclient
node_webclient -->|"sends requests"| node_api
node_api -->|"searches and manages"| node_engine
node_engine -->|"loads chunks"| node_corpus
node_corpus -->|"uses catalog"| node_sourcecatalog
node_corpus -.->|"reads and caches"| node_rawdocs
node_corpus -->|"fetches missing files"| node_rangerdocs
node_engine -->|"ranks lexically"| node_bm25
node_engine -->|"requests embeddings"| node_openai
node_engine -->|"replaces and searches"| node_vector
node_engine -->|"fuses ranked hits"| node_fusion
node_engine -->|"ingests and removes"| node_uploadmanager
node_uploadmanager -->|"writes and reads"| node_uploaddisk
node_uploadmanager -->|"chunks documents"| node_corpus
node_api -->|"streams generated answer"| node_openai
node_api -->|"streams chat results"| node_webclient

click node_webclient "https://github.com/shubham96681/lexical-search-bot/blob/main/web/app.js"
click node_api "https://github.com/shubham96681/lexical-search-bot/blob/main/server.py"
click node_engine "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py"
click node_bm25 "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py"
click node_fusion "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py"
click node_vector "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/vector_store.py"
click node_corpus "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/corpus.py"
click node_sourcecatalog "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/sources.py"
click node_rawdocs "https://github.com/shubham96681/lexical-search-bot/tree/main/data/raw"
click node_uploadmanager "https://github.com/shubham96681/lexical-search-bot/blob/main/kb/uploads.py"
click node_uploaddisk "https://github.com/shubham96681/lexical-search-bot/tree/main/data/uploads"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_webclient,node_user toneBlue
class node_api toneAmber
class node_engine,node_bm25,node_fusion,node_vector,node_openai toneMint
class node_corpus,node_sourcecatalog,node_rawdocs,node_uploadmanager,node_uploaddisk toneRose
class node_rangerdocs toneIndigo
```

## Components

### Web interface

- [Chat and controls](https://github.com/shubham96681/lexical-search-bot/blob/main/web/app.js): `web/app.js`

### API service

- [HTTP API](https://github.com/shubham96681/lexical-search-bot/blob/main/server.py): `server.py`

### Knowledge retrieval

- [Knowledge engine](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py): `kb/engine.py`
- [BM25 search](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py): `kb/engine.py`
- [Rank fusion](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/engine.py): `kb/engine.py`
- [Vector index](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/vector_store.py): `kb/vector_store.py`

### Document pipeline

- [Corpus loader and chunker](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/corpus.py): `kb/corpus.py`
- [Ranger source catalog](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/sources.py): `kb/sources.py`
- [Cached Ranger corpus](https://github.com/shubham96681/lexical-search-bot/tree/main/data/raw): `data/raw`
- [Upload processing](https://github.com/shubham96681/lexical-search-bot/blob/main/kb/uploads.py): `kb/uploads.py`
- [Uploaded documents](https://github.com/shubham96681/lexical-search-bot/tree/main/data/uploads): `data/uploads`

### Other components

- **Chat user**
- **OpenAI API**
- **Apache Ranger sources**

## Connections

- Chat user → Chat and controls: asks or configures (evidence: `web/app.js`)
- Chat and controls → HTTP API: sends requests (evidence: `web/app.js`)
- HTTP API → Knowledge engine: searches and manages (evidence: `server.py`)
- Knowledge engine → Corpus loader and chunker: loads chunks (evidence: `kb/engine.py`)
- Corpus loader and chunker → Ranger source catalog: uses catalog (evidence: `kb/corpus.py`)
- Corpus loader and chunker → Cached Ranger corpus: reads and caches (evidence: `README.md`)
- Corpus loader and chunker → Apache Ranger sources: fetches missing files (evidence: `README.md`)
- Knowledge engine → BM25 search: ranks lexically (evidence: `kb/engine.py`)
- Knowledge engine → OpenAI API: requests embeddings (evidence: `kb/engine.py`)
- Knowledge engine → Vector index: replaces and searches (evidence: `kb/engine.py`)
- Knowledge engine → Rank fusion: fuses ranked hits (evidence: `kb/engine.py`)
- Knowledge engine → Upload processing: ingests and removes (evidence: `kb/engine.py`)
- Upload processing → Uploaded documents: writes and reads (evidence: `kb/uploads.py`)
- Upload processing → Corpus loader and chunker: chunks documents (evidence: `kb/uploads.py`)
- HTTP API → OpenAI API: streams generated answer
- HTTP API → Chat and controls: streams chat results (evidence: `server.py`)
