How nhduong1203/chatbot works
Main Components
• Frontend:
• “app/frontend” component that provides the user interface and initiates stateful WebSocket connections.
• A Docker container for the frontend and note its integration with NGINX if applicable.
• Backend Services:
⁃ The QA service (in “app/backend/qa”) handling conversation initiation, standalone question formation, and LLM calls.
⁃ The Reference Document service (in “app/backend/reference_doc”) that manages document processing, including semantic chunking, embedding (through document_preprocessor, embedder, semantic chunking code), and storage.
• Supporting utility modules and shared components (e.g., “utils.py”) common to both backend modules.
• Data Management:
⁃ Cassandra for historical conversation storage (used by Chat System for stateful conversations).
⁃ Redis for vector database functionalities used in retrieval of context embeddings.
⁃ MinIO used as object storage for uploaded documents.
• “database_manager” modules under each backend (qa and reference_doc) that abstract access to these stores.
• LLM Model Serving:
• Specialized LLM serving module, detailing:
⁃ The process of converting and building the LLM engine using TensorRT-LLM.
⁃ Deployment of the LLM backend using Triton Inference Server (as scripted via “run_triton.sh”).
⁃ Integration with KServe (InferenceService and ClusterServingRuntime) to manage GPU instances and handle model inference requests.
• Infrastructure & Deployment:
• CI/CD workflow:
⁃ Docker-compose build and tagging steps.
• Kubernetes deployment components:
⁃ YAML files under “k8s-yaml” for backend, frontend, database (Cassandra, Redis, MinIO), and NGINX.
⁃ Terraform and Ansible deployment scripts for spinning up GKE clusters, Jenkins integration, and compute instances.
• Observability:
• Observability components from the “observability” directory:
⁃ Metrics and logging with Prometheus, Grafana, Jaeger, and the ELK (Elasticsearch/Kibana) stack.
⁃ Indicate deployment and configuration artifacts (Helm charts, inotify limits, etc).
Relationships and Data Flow
• Data flow from the user on the frontend through a stateful WebSocket connection to the backend chat service.
• How user queries trigger the following processes:
• Chat Service processes the query and calls a dedicated “llm_call” module to generate standalone questions.
• The standalone question’s embedding is then used to query Redis (storing the vectorized document chunks) for relevant context.
• The retrieved context, along with the question, is used to generate a response via an OpenAI API call.
• Document uploads being saved in MinIO followed by further processing in the “reference_doc” backend to prepare semantic chunks and embeddings.
• Interactions between the backend services and the LLM model serving pipeline when advanced inference is needed.
• CI/CD pipelines indicating new image builds pushed to Docker Hub then deployed to GKE.
Architectural Patterns and Design Principles
• Frontend vs. Backend (further split into QA and Document Reference handling).
• Database storage layers (Cassandra, Redis, and MinIO) as supporting layers.
• A dedicated pipeline for LLM model processing (TensorRT-LLM, Triton Inference Server, KServe).
• Leverage containerization (Docker) and orchestration (Kubernetes) concepts.
• Observability and deployment strategies via dedicated modules.
Technologies, Frameworks, and Tools
• Docker for containerization and docker-compose for local builds.
• Kubernetes YAML manifests and Helm charts for deployment.
• Terraform and Ansible for infrastructure provisioning (GKE clusters, Jenkins deployment).
• Programming platform: Python (FastAPI or similar frameworks, indicated by __pycache__ and requirements.txt usage).
• Databases: Cassandra, Redis, MinIO.
• LLM and inference stack: TensorRT-LLM, Triton Inference Server, KServe.
• Observability: Prometheus, Grafana, Jaeger, Elasticsearch/Kibana.
Specific Elements
• Each major component should have a clear, descriptive label.
• How a user’s question flows from the frontend into the backend.
• The branching of processing (semantic chunking, embedding, context retrieval).
• The data persistence mechanisms (Storing history in Cassandra, document data in MinIO).
• The interconnectivity between Kubernetes-managed services.
• Optionally, include annotations or legends that identify which components are deployed on GPU-enabled nodes (LLM Node Pool) versus other node pools.
• Integration points with CI/CD pipelines, IaC artifacts (Terraform & Ansible), and observability modules.