# vkrypto/semantics-search architecture

> Architecture diagram of the GitHub repository vkrypto/semantics-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vkrypto/semantics-search
- Repository: https://github.com/vkrypto/semantics-search

## Overview

##### Project type and purpose
- Project type: Full-stack web application
– Frontend: React widget (built with Vite)
– Backend: FastAPI service (Python)
- Purpose: “semantics-search-chatbot” provides a semantic search interface and conversational chatbot backed by Elasticsearch and LLMs. It ingests diverse data streams, indexes them for semantic retrieval, and uses large-language models for chat interactions.

##### File structure
- Top-level artifacts
– .github/workflows: CI/tests
– docker-stack.yml: Elasticsearch single-node Docker deploy
– backend/: FastAPI service, modules, tests, static assets, templates
– frontend/: React SPA, Vite config, components, build scripts
– assets/: README images
- Backend subpackages
– app.py & routers/: API entrypoints (/search, /chat)
– data_streams/: data-source plugins (faq, kafka, myntra, ornaz)
– db_connectors/elasticdb/: ES client & models
– search/: vector-search implementations (cosine, euclidean, hybrid)
– llms/: LLM adapters (OpenAI GPT, Ollama, LongChain) + session stores (memory, Redis)
– utils/: common routines (encoder, indexer, timing decorators)
– logger/: logging config
– static/ & templates/: served UI and widget assets
– tests/: pytest suites
- Frontend
– src/components/ChatbotWidget.jsx, widget.jsx: embeddable chat UI
– src/App.jsx: standalone demo
– public/index.html & dist/: built widget & CSS/JS

##### Key insights from README
- Core features
– Semantic search via transformer + Elasticsearch
– Dockerized ES deployment
– Automated tests (pytest)
– Data stream aggregation & indexing
– Performance logging utilities
- Usage
– “docker stack deploy -c docker-stack.yml elasticsearch”
– “uvicorn app:app” for backend
– “yarn build” for frontend

a. Main components
1. User Interface (Browser or Embedded Chatbot Widget)
2. Frontend SPA (React/Vite)
3. HTTP API Layer (FastAPI)
4. Data Stream Manager (ETL plugins)
5. Indexer & Search Engine (Elasticsearch cluster)
6. LLM Adapters (OpenAI, Ollama, LongChain)
7. Session Store (Redis or In-memory)
8. Logger & Monitoring (timing decorators, config JSON)
9. CI/CD Pipeline & Docker Orchestration
b. Relationships & data flow
– User ↔ Frontend: HTTP/JSX interfaces & WebSocket (if any)
– Frontend → FastAPI: REST calls (/search, /chat)
– FastAPI → Elasticsearch: index/query via db_connectors/elasticdb
– FastAPI → LLM Adapters: generate chat completions, prompt formatting
– FastAPI → Session Store: store/retrieve conversation context
– Data Stream Manager → Indexer → Elasticsearch: ingest & transform source files
– CI/CD (.github) & Docker Stack → deploy Elasticsearch & app
c. Architectural patterns & principles
– Layered architecture: Presentation (UI), Application (API), Data (Elasticsearch, Redis)
– Plugin pattern: data_streams and llms adapters can be extended independently
– Strategy/Factory for selecting search algorithm (cosine, euclidean, hybrid)
– Single-page-application decoupled from the API layer
d. Key technologies
– Frontend: React, Vite, JSX, CSS
– Backend: Python, FastAPI, Uvicorn
– Search: Elasticsearch, Python elastic-client
– LLMs: OpenAI API, Ollama, LangChain
– Session store: Redis or in-memory
– Testing: pytest, tox
– Deployment: Docker stack, GitHub Actions

##### Capturing essential architectural elements
- Do not over-abstract—expose major subcomponents (search strategies, data streams, llm adapters)
- Both runtime data flow (user → search/chat → responses) and background workflows (data ingestion → indexing)
- CI/CD and deployment artifacts to illustrate end-to-end flow

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interface
    User["User (Browser)"]:::ui
    subgraph "Frontend Layer"
        direction TB
        Widget["Embedded Chatbot Widget"]:::ui
        click Widget "https://github.com/vkrypto/semantics-search/blob/main/frontend/src/components/ChatbotWidget.jsx"
        WidgetEntry["Widget Entry"]:::ui
        click WidgetEntry "https://github.com/vkrypto/semantics-search/blob/main/frontend/src/widget.jsx"
        PublicPage["Public HTML"]:::ui
        click PublicPage "https://github.com/vkrypto/semantics-search/blob/main/frontend/public/index.html"
        DemoApp["Standalone App"]:::ui
        click DemoApp "https://github.com/vkrypto/semantics-search/blob/main/frontend/src/App.jsx"
        MainEntry["Main Entry"]:::ui
        click MainEntry "https://github.com/vkrypto/semantics-search/blob/main/frontend/src/main.jsx"
        ViteConfig["Vite Config"]:::service
        click ViteConfig "https://github.com/vkrypto/semantics-search/blob/main/frontend/vite.config.js"
        IndexHTML["Index HTML"]:::ui
        click IndexHTML "https://github.com/vkrypto/semantics-search/blob/main/frontend/index.html"
    end

    %% Backend Layer
    subgraph "Backend Layer"
        direction TB
        APILayer["FastAPI HTTP API"]:::service
        click APILayer "https://github.com/vkrypto/semantics-search/blob/main/backend/app.py"
        subgraph "Routers"
            SearchRouter["/search Router"]:::service
            click SearchRouter "https://github.com/vkrypto/semantics-search/blob/main/backend/routers/search.py"
            ChatRouter["/chat Router"]:::service
            click ChatRouter "https://github.com/vkrypto/semantics-search/blob/main/backend/routers/chat.py"
        end
        subgraph "Data Stream Manager"
            direction TB
            BasePlugin["Base Plugin"]:::service
            click BasePlugin "https://github.com/vkrypto/semantics-search/blob/main/backend/data_streams/base.py"
            FAQStream["FAQ Plugin"]:::service
            click FAQStream "https://github.com/vkrypto/semantics-search/tree/main/backend/data_streams/faq/"
            KafkaStream["Kafka Plugin"]:::service
            click KafkaStream "https://github.com/vkrypto/semantics-search/tree/main/backend/data_streams/kafka/"
            MyntraStream["Myntra Plugin"]:::service
            click MyntraStream "https://github.com/vkrypto/semantics-search/tree/main/backend/data_streams/myntra/"
            OrnazStream["Ornaz Plugin"]:::service
            click OrnazStream "https://github.com/vkrypto/semantics-search/tree/main/backend/data_streams/ornaz/"
        end
        subgraph "Search Engine Integration"
            direction TB
            ESConnector["Elasticsearch Connector"]:::datastore
            click ESConnector "https://github.com/vkrypto/semantics-search/blob/main/backend/db_connectors/elasticdb/connector.py"
            ESModels["ES Models"]:::datastore
            click ESModels "https://github.com/vkrypto/semantics-search/blob/main/backend/db_connectors/elasticdb/models.py"
            subgraph "Search Strategies"
                direction TB
                Cosine["Cosine Search"]:::service
                click Cosine "https://github.com/vkrypto/semantics-search/blob/main/backend/search/cosine_search.py"
                Euclidean["Euclidean Search"]:::service
                click Euclidean "https://github.com/vkrypto/semantics-search/blob/main/backend/search/euclidean_search.py"
                Hybrid["Hybrid Search"]:::service
                click Hybrid "https://github.com/vkrypto/semantics-search/blob/main/backend/search/hybrid_search.py"
                SearchMain["Search Dispatcher"]:::service
                click SearchMain "https://github.com/vkrypto/semantics-search/blob/main/backend/search/main.py"
            end
        end
        subgraph "LLM Adapters"
            direction TB
            GPTAdapter["OpenAI GPT Adapter"]:::external
            click GPTAdapter "https://github.com/vkrypto/semantics-search/tree/main/backend/llms/gpt/"
            OllamaAdapter["Ollama Adapter"]:::external
            click OllamaAdapter "https://github.com/vkrypto/semantics-search/tree/main/backend/llms/ollama/"
            LongChainAdapter["LongChain Adapter"]:::external
            click LongChainAdapter "https://github.com/vkrypto/semantics-search/tree/main/backend/llms/longchain/"
            GPTREST["GPT REST API Adapter"]:::external
            click GPTREST "https://github.com/vkrypto/semantics-search/tree/main/backend/llms/gpt_rest_api_based/"
            OllamaREST["Ollama REST API Adapter"]:::external
            click OllamaREST "https://github.com/vkrypto/semantics-search/tree/main/backend/llms/ollama_rest_api_based/"
        end
        subgraph "Session Store"
            direction TB
            SessionInterface["Session Store Interface"]:::service
            click SessionInterface "https://github.com/vkrypto/semantics-search/blob/main/backend/llms/_session_store/_interface.py"
            MemoryStore["In-Memory Store"]:::datastore
            click MemoryStore "https://github.com/vkrypto/semantics-search/blob/main/backend/llms/_session_store/memory.py"
            RedisStore["Redis Store"]:::datastore
            click RedisStore "https://github.com/vkrypto/semantics-search/blob/main/backend/llms/_session_store/redis.py"
        end
        subgraph "Utilities & Logging"
            direction TB
            LoggerConfig["Logging Config"]:::utility
            click LoggerConfig "https://github.com/vkrypto/semantics-search/blob/main/backend/logger/config.json"
            CommonUtil["Common Utils"]:::utility
            click CommonUtil "https://github.com/vkrypto/semantics-search/blob/main/backend/utils/common.py"
            EncoderUtil["Encoder Utils"]:::utility
            click EncoderUtil "https://github.com/vkrypto/semantics-search/blob/main/backend/utils/encoder.py"
            IndexerUtil["Indexer Utils"]:::utility
            click IndexerUtil "https://github.com/vkrypto/semantics-search/blob/main/backend/utils/indexer.py"
        end
    end

    %% Data Ingestion Flow
    DataIngest["Data Ingestion Pipeline"]:::service

    %% Datastores
    ESCluster["Elasticsearch Cluster"]:::datastore
    RedisDB["Redis"]:::datastore

    %% External Systems
    DockerStack["Docker Orchestration"]:::external
    CI["CI/CD Pipeline"]:::external
    click CI "https://github.com/vkrypto/semantics-search/blob/main/.github/workflows/test.yml"
    click DockerStack "https://github.com/vkrypto/semantics-search/blob/main/docker-stack.yml"

    %% Connections
    User -->|uses| Widget
    Widget -->|renders| WidgetEntry
    WidgetEntry -->|loads| PublicPage
    WidgetEntry -->|fetches| APILayer
    DemoApp -->|uses| MainEntry
    MainEntry -->|builds| APILayer

    APILayer -->|routes to| SearchRouter
    APILayer -->|routes to| ChatRouter
    SearchRouter -->|queries| ESConnector
    ESConnector -->|uses| ESCluster
    SearchRouter -->|dispatch| SearchMain
    SearchMain -->|implements| Cosine
    SearchMain -->|implements| Euclidean
    SearchMain -->|implements| Hybrid

    ChatRouter -->|calls| GPTAdapter
    ChatRouter -->|calls| OllamaAdapter
    ChatRouter -->|calls| LongChainAdapter
    ChatRouter -->|calls| GPTREST
    ChatRouter -->|calls| OllamaREST
    ChatRouter -->|manages| SessionInterface
    SessionInterface -->|stores| MemoryStore
    SessionInterface -->|stores| RedisStore

    DataIngest -->|uses plugins| BasePlugin
    BasePlugin --> FAQStream
    BasePlugin --> KafkaStream
    BasePlugin --> MyntraStream
    BasePlugin --> OrnazStream
    DataIngest -->|indexes to| ESConnector

    APILayer -->|logs via| LoggerConfig
    APILayer -->|uses| CommonUtil
    APILayer -->|uses| EncoderUtil
    APILayer -->|uses| IndexerUtil

    CI -->|deploys| DockerStack
    DockerStack -->|orchestrates| ESCluster
    DockerStack -->|runs| APILayer

    %% Styles
    classDef ui fill:#E0F7FA,stroke:#006064,color:#006064
    classDef service fill:#E8F5E9,stroke:#2E7D32,color:#2E7D32
    classDef datastore fill:#FFF3E0,stroke:#EF6C00,color:#E65100
    classDef external fill:#ECEFF1,stroke:#37474F,color:#37474F
    classDef utility fill:#F3E5F5,stroke:#6A1B9A,color:#4A148C
```
