How vkrypto/semantics-search works
Project type and purpose
• Project type: Full-stack web application
– Frontend: React widget (built with Vite)
– Backend: FastAPI service (Python)
• Purpose: “semantics-search-chatbot” provides a semantic search interface and conversational chatbot backed by Elasticsearch and LLMs. It ingests diverse data streams, indexes them for semantic retrieval, and uses large-language models for chat interactions.
File structure
• Top-level artifacts
– .github/workflows: CI/tests
– docker-stack.yml: Elasticsearch single-node Docker deploy
– backend/: FastAPI service, modules, tests, static assets, templates
– frontend/: React SPA, Vite config, components, build scripts
– assets/: README images
• Backend subpackages
– app.py & routers/: API entrypoints (/search, /chat)
– data_streams/: data-source plugins (faq, kafka, myntra, ornaz)
– db_connectors/elasticdb/: ES client & models
– search/: vector-search implementations (cosine, euclidean, hybrid)
– llms/: LLM adapters (OpenAI GPT, Ollama, LongChain) + session stores (memory, Redis)
– utils/: common routines (encoder, indexer, timing decorators)
– logger/: logging config
– static/ & templates/: served UI and widget assets
– tests/: pytest suites
• Frontend
– src/components/ChatbotWidget.jsx, widget.jsx: embeddable chat UI
– src/App.jsx: standalone demo
– public/index.html & dist/: built widget & CSS/JS
Key insights from README
• Core features
– Semantic search via transformer + Elasticsearch
– Dockerized ES deployment
– Automated tests (pytest)
– Data stream aggregation & indexing
– Performance logging utilities
• Usage
– “docker stack deploy -c docker-stack.yml elasticsearch”
– “uvicorn app:app” for backend
– “yarn build” for frontend
a. Main components
1. User Interface (Browser or Embedded Chatbot Widget)
2. Frontend SPA (React/Vite)
3. HTTP API Layer (FastAPI)
4. Data Stream Manager (ETL plugins)
5. Indexer & Search Engine (Elasticsearch cluster)
6. LLM Adapters (OpenAI, Ollama, LongChain)
7. Session Store (Redis or In-memory)
8. Logger & Monitoring (timing decorators, config JSON)
9. CI/CD Pipeline & Docker Orchestration
b. Relationships & data flow
– User ↔ Frontend: HTTP/JSX interfaces & WebSocket (if any)
– Frontend → FastAPI: REST calls (/search, /chat)
– FastAPI → Elasticsearch: index/query via db_connectors/elasticdb
– FastAPI → LLM Adapters: generate chat completions, prompt formatting
– FastAPI → Session Store: store/retrieve conversation context
– Data Stream Manager → Indexer → Elasticsearch: ingest & transform source files
– CI/CD (.github) & Docker Stack → deploy Elasticsearch & app
c. Architectural patterns & principles
– Layered architecture: Presentation (UI), Application (API), Data (Elasticsearch, Redis)
– Plugin pattern: data_streams and llms adapters can be extended independently
– Strategy/Factory for selecting search algorithm (cosine, euclidean, hybrid)
– Single-page-application decoupled from the API layer
d. Key technologies
– Frontend: React, Vite, JSX, CSS
– Backend: Python, FastAPI, Uvicorn
– Search: Elasticsearch, Python elastic-client
– LLMs: OpenAI API, Ollama, LangChain
– Session store: Redis or in-memory
– Testing: pytest, tox
– Deployment: Docker stack, GitHub Actions
Capturing essential architectural elements
• Do not over-abstract—expose major subcomponents (search strategies, data streams, llm adapters)
• Both runtime data flow (user → search/chat → responses) and background workflows (data ingestion → indexing)
• CI/CD and deployment artifacts to illustrate end-to-end flow