# nhduong1203/chatbot architecture

> Architecture diagram of the GitHub repository nhduong1203/chatbot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nhduong1203/chatbot
- Repository: https://github.com/nhduong1203/chatbot

## Overview

##### Main Components
- Frontend:
- “app/frontend” component that provides the user interface and initiates stateful WebSocket connections.
- A Docker container for the frontend and note its integration with NGINX if applicable.
- Backend Services:
⁃ The QA service (in “app/backend/qa”) handling conversation initiation, standalone question formation, and LLM calls.
⁃ The Reference Document service (in “app/backend/reference_doc”) that manages document processing, including semantic chunking, embedding (through document_preprocessor, embedder, semantic chunking code), and storage.
- Supporting utility modules and shared components (e.g., “utils.py”) common to both backend modules.
- Data Management:
⁃ Cassandra for historical conversation storage (used by Chat System for stateful conversations).
⁃ Redis for vector database functionalities used in retrieval of context embeddings.
⁃ MinIO used as object storage for uploaded documents.
- “database_manager” modules under each backend (qa and reference_doc) that abstract access to these stores.
- LLM Model Serving:
- Specialized LLM serving module, detailing:
⁃ The process of converting and building the LLM engine using TensorRT-LLM.
⁃ Deployment of the LLM backend using Triton Inference Server (as scripted via “run_triton.sh”).
⁃ Integration with KServe (InferenceService and ClusterServingRuntime) to manage GPU instances and handle model inference requests.
- Infrastructure & Deployment:
- CI/CD workflow:
⁃ Docker-compose build and tagging steps.
- Kubernetes deployment components:
⁃ YAML files under “k8s-yaml” for backend, frontend, database (Cassandra, Redis, MinIO), and NGINX.
⁃ Terraform and Ansible deployment scripts for spinning up GKE clusters, Jenkins integration, and compute instances.
- Observability:
- Observability components from the “observability” directory:
⁃ Metrics and logging with Prometheus, Grafana, Jaeger, and the ELK (Elasticsearch/Kibana) stack.
⁃ Indicate deployment and configuration artifacts (Helm charts, inotify limits, etc).

##### Relationships and Data Flow
- Data flow from the user on the frontend through a stateful WebSocket connection to the backend chat service.
- How user queries trigger the following processes:
- Chat Service processes the query and calls a dedicated “llm_call” module to generate standalone questions.
- The standalone question’s embedding is then used to query Redis (storing the vectorized document chunks) for relevant context.
- The retrieved context, along with the question, is used to generate a response via an OpenAI API call.
- Document uploads being saved in MinIO followed by further processing in the “reference_doc” backend to prepare semantic chunks and embeddings.
- Interactions between the backend services and the LLM model serving pipeline when advanced inference is needed.
- CI/CD pipelines indicating new image builds pushed to Docker Hub then deployed to GKE.

##### Architectural Patterns and Design Principles
- Frontend vs. Backend (further split into QA and Document Reference handling).
- Database storage layers (Cassandra, Redis, and MinIO) as supporting layers.
- A dedicated pipeline for LLM model processing (TensorRT-LLM, Triton Inference Server, KServe).
- Leverage containerization (Docker) and orchestration (Kubernetes) concepts.
- Observability and deployment strategies via dedicated modules.

##### Technologies, Frameworks, and Tools
- Docker for containerization and docker-compose for local builds.
- Kubernetes YAML manifests and Helm charts for deployment.
- Terraform and Ansible for infrastructure provisioning (GKE clusters, Jenkins deployment).
- Programming platform: Python (FastAPI or similar frameworks, indicated by __pycache__ and requirements.txt usage).
- Databases: Cassandra, Redis, MinIO.
- LLM and inference stack: TensorRT-LLM, Triton Inference Server, KServe.
- Observability: Prometheus, Grafana, Jaeger, Elasticsearch/Kibana.

##### Specific Elements
- Each major component should have a clear, descriptive label.
- How a user’s question flows from the frontend into the backend.
- The branching of processing (semantic chunking, embedding, context retrieval).
- The data persistence mechanisms (Storing history in Cassandra, document data in MinIO).
- The interconnectivity between Kubernetes-managed services.
- Optionally, include annotations or legends that identify which components are deployed on GPU-enabled nodes (LLM Node Pool) versus other node pools.
- Integration points with CI/CD pipelines, IaC artifacts (Terraform & Ansible), and observability modules.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Interaction
    subgraph "User Interaction"
        U("User"):::default
    end

    %% Frontend Layer
    subgraph "Frontend Layer"
        FE("Frontend UI"):::frontend
    end

    %% Backend Services
    subgraph "Backend Services"
        QA("QA Service"):::backend
        RD("Reference Document Service"):::backend
        LLM("LLM Inference Module"):::backend
    end

    %% Data Management
    subgraph "Data Management"
        CassandraDB("Cassandra DB"):::database
        CassandraDeploy("Cassandra Deployment"):::infrastructure
        RedisDB("Redis DB"):::database
        RedisDeploy("Redis Deployment"):::infrastructure
        MinIOStorage("MinIO Object Storage"):::database
        MinIODeploy("MinIO Deployment"):::infrastructure
    end

    %% Infrastructure & Deployment
    subgraph "Infrastructure & Deployment"
        DC("Docker Compose"):::infrastructure
        K8s("Kubernetes Manifests"):::infrastructure
        Terraform("Terraform IaC"):::infrastructure
        Ansible("Ansible IaC"):::infrastructure
        Jenkins("Jenkins Setup"):::infrastructure
    end

    %% Observability
    subgraph "Observability"
        Obs("Observability Stack"):::observability
    end

    %% Connections: User to Frontend
    U -->|"WebSocket"| FE

    %% Connections: Frontend to Backend
    FE -->|"sends_query"| QA

    %% Backend internal communications
    QA -->|"calls_llm_call"| LLM
    LLM -->|"returns_response"| QA

    %% Data storage interactions from QA Service
    QA -->|"stores_history"| CassandraDB
    QA -->|"queries_context"| RedisDB

    %% Document service interactions
    RD -->|"uploads_docs"| MinIOStorage
    RD -->|"queries_vector"| RedisDB

    %% Deployment relationships in Data Management
    CassandraDeploy ---|"deploys"| CassandraDB
    RedisDeploy ---|"deploys"| RedisDB
    MinIODeploy ---|"deploys"| MinIOStorage

    %% Infrastructure & Deployment flows
    DC -->|"triggers"| K8s
    Terraform -->|"provisions"| K8s
    Ansible -->|"configures"| K8s
    Jenkins -->|"orchestrates"| DC

    %% Kubernetes deploys Backend and Data components
    K8s -->|"deploys"| QA
    K8s -->|"deploys"| RD
    K8s -->|"deploys"| CassandraDeploy
    K8s -->|"deploys"| RedisDeploy
    K8s -->|"deploys"| MinIODeploy

    %% Observability monitors components
    Obs ---|"monitors"| FE
    Obs ---|"monitors"| QA
    Obs ---|"monitors"| RD
    Obs ---|"monitors"| LLM
    Obs ---|"monitors"| K8s
    Obs ---|"monitors"| CassandraDB
    Obs ---|"monitors"| RedisDB
    Obs ---|"monitors"| MinIOStorage

    %% Click Events for Component Mapping
    click FE "https://github.com/nhduong1203/chatbot/tree/main/app/frontend"
    click QA "https://github.com/nhduong1203/chatbot/tree/main/app/backend/qa"
    click RD "https://github.com/nhduong1203/chatbot/tree/main/app/backend/reference_doc"
    click CassandraDB "https://github.com/nhduong1203/chatbot/blob/main/app/backend/qa/database_manager/cassandra.py"
    click CassandraDeploy "https://github.com/nhduong1203/chatbot/tree/main/k8s-yaml/db/cassandra"
    click RedisDB "https://github.com/nhduong1203/chatbot/blob/main/app/backend/qa/database_manager/redis.py"
    click RedisDeploy "https://github.com/nhduong1203/chatbot/blob/main/k8s-yaml/db/redis.yaml"
    click MinIOStorage "https://github.com/nhduong1203/chatbot/blob/main/app/backend/reference_doc/database_manager/minio.py"
    click MinIODeploy "https://github.com/nhduong1203/chatbot/blob/main/k8s-yaml/db/minio.yaml"
    click LLM "https://github.com/nhduong1203/chatbot/tree/main/app/backend/qa/llm_call"
    click DC "https://github.com/nhduong1203/chatbot/blob/main/docker-compose.yaml"
    click K8s "https://github.com/nhduong1203/chatbot/tree/main/k8s-yaml"
    click Terraform "https://github.com/nhduong1203/chatbot/tree/main/iac/terraform"
    click Ansible "https://github.com/nhduong1203/chatbot/tree/main/iac/ansible"
    click Jenkins "https://github.com/nhduong1203/chatbot/tree/main/jenkins"
    click Obs "https://github.com/nhduong1203/chatbot/tree/main/observability"

    %% Styles
    classDef frontend fill:#CDEBFF,stroke:#1F78C1,stroke-width:2px;
    classDef backend fill:#FFE6E6,stroke:#E31A1C,stroke-width:2px;
    classDef database fill:#DFFFD6,stroke:#33A02C,stroke-width:2px;
    classDef infrastructure fill:#FFF2CC,stroke:#FF8C00,stroke-width:2px;
    classDef observability fill:#F0E68C,stroke:#DAA520,stroke-width:2px;
    classDef default fill:#ffffff,stroke:#000000,stroke-width:1px;
```
