How ayush-patel-10/rag-using-azure-databricks-ci-cd works
Main components
• Azure Infrastructure – the overall cloud environment, resource groups, storage accounts, and Databricks workspaces.
• Databricks Workspace – the central component where development, testing, and production activities occur. Include the Databricks Folder Structure that distinguishes between test, staging, and production.
• CI/CD Pipeline – a dedicated component that automates deployment, testing, and integration from development through production. This will include the GitHub secrets configuration and any scripts that trigger deployments.
• Terraform Module – used for provisioning infrastructure and managing state, which is critical for reproducibility in the Databricks environment.
• Model Management Component – featuring MLFlow for tracking model iterations, evaluating performance, and orchestrating model versioning and rollback.
• Data Storage – Delta Tables for secure and efficient storage of data, ensuring both compliance and performance.
• Chatbot Framework – utilizing the RAG approach to fetch and augment responses based on contextual data.
Relationships and interactions
• The Azure infrastructure underpins the entire system where Databricks and storage components are deployed.
• The Databricks Workspace interacts with Delta Tables for data read/write operations and serves as the environment for training and deploying the RAG chatbot.
• The CI/CD pipeline interacts with the Databricks Workspace to automate model training, tests, and production deployments.
• Terraform scripts manage and provision the Azure and Databricks resources, ensuring that changes to the infrastructure are versioned and reproducible.
• MLFlow monitors and serves as the model registry, capturing model performance data, deploying models to production, and handling version rollback as needed.
• The chatbot accesses the deployed models (via MLFlow) and the underlying data (Delta Tables) to generate and refine responses using the RAG framework.
Architectural patterns and design principles
• Separation of concerns is evident by distinguishing between developmental, staging, and production environments in the Databricks folder structure.
• Infrastructure as Code (IaC) principles via Terraform support repeatable, scalable environment provisioning.
• The continuous integration and deployment pattern ensures seamless transitions through various stages of the ML lifecycle.
• Modular design with clear roles for each component (Databricks workspace, CI/CD, MLFlow, and cost analysis for resource optimization) supports scalability and flexibility.
Relevant technologies and frameworks
• Azure and its associated resources
• Databricks: for distributed data processing and ML model management
• Terraform: for infrastructure provisioning and state management
• MLFlow: for model tracking and lifecycle management
• Delta Tables: for efficient, secure data storage
• CI/CD tools (likely GitHub Actions or another CI tool based on your README references)