How duongngyn0510/deploy-highly-scalable-rag-for-vietnamese-law-qa-system works
Main components
• Embedding Component – Deployed in its own GKE cluster with Terraform and managed via Helm charts located under the “embedding” directory. This component runs a text-embeddings-inference service that pulls models from Huggingface.
• Weaviate Component – Deployed separately on its own GKE cluster (configured via Terraform and managed by Helm charts under “weaviate”). This serves as the vector store (likely a database and search layer) for storing processed embeddings.
• Reranker and LLM Component – Runs on a dedicated GPU server using Docker Compose (found in the “reranker-llm” directory). This component handles heavy model inference tasks like reranking and language model operations.
• RAG Controller Component – A central FastAPI service (with main.py as the entry point) that orchestrates communication between the embedding, retrieval, and LLM services. It is deployed in a GKE cluster (with its own Helm chart and Terraform scripts in “rag-controller”) and also contains various subcomponents (chat, completions, translation, vector store, etc.) within its “src” folder.
• Monitoring & CI/CD – Monitoring is implemented using Prometheus (with Helm charts under “monitoring” and Grafana configurations) and there is a CI/CD pipeline indicated by the Jenkinsfile and Ansible playbooks under “gce”. These show integration with provisioning and deployment tools.
Relationships and interactions between components
• Frontend/API gateway (RAG Controller) interfaces with backend processes and routes traffic. It also contains UI elements (referenced in the README’s UI image) and communicates with the other core service components.
• The RAG Controller interacts with the Embedding Component to obtain text embeddings and with the Weaviate Component to perform vector store queries.
• The Reranker and LLM component (running on the GPU server) likely receives requests from the RAG Controller, providing enhanced language model responses or re-ranking results.
• Monitoring components observe the state of each GKE cluster (via Prometheus installed on each cluster with a remote-write pattern) and feed dashboards in Grafana.
• CI/CD components (Jenkins, Ansible, Terraform, Helm) ensure automated deployment and updates across these clusters.
Architectural patterns and technologies
• Cloud Native / Microservices Architecture: Components are split into distinct Kubernetes clusters or Docker deployments for service isolation and scalability.
• Infrastructure as Code: Terraform scripts are used to provision the clusters, and Helm charts manage deployments for Embedding, Weaviate, and RAG Controller components.
• Containerization: Docker is used across the board (including local development via Docker Compose and deployment with Dockerfiles).
• Automation: CI/CD pipelines leverage Jenkins and Ansible for configuration, testing, and deployments.
• Monitoring: A dedicated monitoring stack (Prometheus + Grafana) collects telemetry from multiple clusters, with remote-write patterns to centralize data.
Need for detail and clarity
• Do not shy away from breaking down the architecture into as many components as you feel necessary. Given the complexity (multiple helm charts, Terraform scripts, several CI/CD pipelines, and diverse service groups), the goal is to ensure that every critical interaction and dependency is visible.