How shubham96681/lexical-search-bot works
This is a Ranger knowledge-base chatbot. A user asks a question in the web client; the FastAPI server retrieves passages through the knowledge-base engine, which combines BM25 lexical search with optional OpenAI embeddings and reciprocal-rank fusion. The server streams sources and either a plain passage-based response or an OpenAI-generated answer. Separate flows configure embeddings and manage uploaded documents, which are parsed into chunks and reindexed alongside the bundled Ranger corpus. The README documents the broader Ranger policy and guidance corpus; unsampled source details are not assumed.
Main entry points: engine.py Knowledge engine · corpus.py Corpus loader and chunker · server.py HTTP API · uploads.py Upload processing
Components 14
Web interface
- Chat and controls
web/app.js
API service
- HTTP API
server.py
Knowledge retrieval
- Knowledge engine
kb/engine.py - BM25 search
kb/engine.py - Rank fusion
kb/engine.py - Vector index
kb/vector_store.py
Document pipeline
- Corpus loader and chunker
kb/corpus.py - Ranger source catalog
kb/sources.py - Cached Ranger corpus
data/raw - Upload processing
kb/uploads.py - Uploaded documents
data/uploads
Other components
- Chat user
- OpenAI API
- Apache Ranger sources
Connections 16
- Chat user → Chat and controls: asks or configures
web/app.js - Chat and controls → HTTP API: sends requests
web/app.js - HTTP API → Knowledge engine: searches and manages
server.py - Knowledge engine → Corpus loader and chunker: loads chunks
kb/engine.py - Corpus loader and chunker → Ranger source catalog: uses catalog
kb/corpus.py - Corpus loader and chunker → Cached Ranger corpus: reads and caches
README.md - Corpus loader and chunker → Apache Ranger sources: fetches missing files
README.md - Knowledge engine → BM25 search: ranks lexically
kb/engine.py - Knowledge engine → OpenAI API: requests embeddings
kb/engine.py - Knowledge engine → Vector index: replaces and searches
kb/engine.py - Knowledge engine → Rank fusion: fuses ranked hits
kb/engine.py - Knowledge engine → Upload processing: ingests and removes
kb/engine.py - Upload processing → Uploaded documents: writes and reads
kb/uploads.py - Upload processing → Corpus loader and chunker: chunks documents
kb/uploads.py - HTTP API → OpenAI API: streams generated answer
- HTTP API → Chat and controls: streams chat results
server.py