# ayush-patel-10/rag-using-azure-databricks-ci-cd architecture

> Architecture diagram of the GitHub repository ayush-patel-10/rag-using-azure-databricks-ci-cd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ayush-patel-10/rag-using-azure-databricks-ci-cd
- Repository: https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd

## Overview

##### Main components
- Azure Infrastructure – the overall cloud environment, resource groups, storage accounts, and Databricks workspaces.
- Databricks Workspace – the central component where development, testing, and production activities occur. Include the Databricks Folder Structure that distinguishes between test, staging, and production.
- CI/CD Pipeline – a dedicated component that automates deployment, testing, and integration from development through production. This will include the GitHub secrets configuration and any scripts that trigger deployments.
- Terraform Module – used for provisioning infrastructure and managing state, which is critical for reproducibility in the Databricks environment.
- Model Management Component – featuring MLFlow for tracking model iterations, evaluating performance, and orchestrating model versioning and rollback.
- Data Storage – Delta Tables for secure and efficient storage of data, ensuring both compliance and performance.
- Chatbot Framework – utilizing the RAG approach to fetch and augment responses based on contextual data.

##### Relationships and interactions
- The Azure infrastructure underpins the entire system where Databricks and storage components are deployed.
- The Databricks Workspace interacts with Delta Tables for data read/write operations and serves as the environment for training and deploying the RAG chatbot.
- The CI/CD pipeline interacts with the Databricks Workspace to automate model training, tests, and production deployments.
- Terraform scripts manage and provision the Azure and Databricks resources, ensuring that changes to the infrastructure are versioned and reproducible.
- MLFlow monitors and serves as the model registry, capturing model performance data, deploying models to production, and handling version rollback as needed.
- The chatbot accesses the deployed models (via MLFlow) and the underlying data (Delta Tables) to generate and refine responses using the RAG framework.

##### Architectural patterns and design principles
- Separation of concerns is evident by distinguishing between developmental, staging, and production environments in the Databricks folder structure.
- Infrastructure as Code (IaC) principles via Terraform support repeatable, scalable environment provisioning.
- The continuous integration and deployment pattern ensures seamless transitions through various stages of the ML lifecycle.
- Modular design with clear roles for each component (Databricks workspace, CI/CD, MLFlow, and cost analysis for resource optimization) supports scalability and flexibility.

##### Relevant technologies and frameworks
- Azure and its associated resources
- Databricks: for distributed data processing and ML model management
- Terraform: for infrastructure provisioning and state management
- MLFlow: for model tracking and lifecycle management
- Delta Tables: for efficient, secure data storage
- CI/CD tools (likely GitHub Actions or another CI tool based on your README references)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Azure Infrastructure
    A["Azure Infrastructure"]:::azure

    %% CI/CD Pipeline
    CICD["CI/CD Pipeline"]:::cicd

    %% Terraform Provisioning
    TF["Terraform Provisioning"]:::terraform

    %% Delta Tables for Data Storage
    DT["Delta Tables (Data Storage)"]:::delta

    %% Model Management: MLFlow & Model Rollback
    ML["MLFlow (Model Registry & Serving)"]:::mlflow
    MR["Model Rollback"]:::mlflow

    %% RAG Chatbot Component
    CB["RAG Chatbot Component"]:::chatbot

    %% Databricks Workspace Subgraph
    subgraph "Databricks Workspace"
        DBW1["Databricks Workspace (Folder Structure)"]:::databricks
        DBW2["Databricks Workflow"]:::databricks
        DBW1 -->|"supports"| DBW2

        subgraph "Environments"
            T["Test"]:::databricks
            S["Staging"]:::databricks
            P["Production"]:::databricks
        end
        DBW2 -->|"defines"| T
        DBW2 -->|"defines"| S
        DBW2 -->|"defines"| P
    end

    %% Relationships
    A -->|"hosts"| DBW1
    A -->|"hosts"| CICD
    A -->|"hosts"| TF
    A -->|"hosts"| DT

    CICD -->|"automates"| DBW1

    TF -->|"provisions"| A
    TF -->|"provisions"| DBW1

    DBW1 -->|"triggers_training"| ML

    ML -->|"manages"| MR
    ML -->|"logs_data"| DT
    ML -->|"serves_models"| CB

    DT -->|"supplies_context"| CB

    %% Click Events
    click A "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/tree/main/README"
    click DBW1 "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_DATABRICKS_FOLDER_STRUCTURE.md"
    click DBW2 "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_DATABRICKS_WORKFLOW.md"
    click CICD "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_CICD_WORKFLOW.md"
    click TF "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_TERRAFORM_TFSTATE.md"
    click ML "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_MLFlow.md"
    click MR "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_MODEL_VERSION_ROLLBACK.md"
    click DT "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README/README_DATABRICKS_CATALOG.md"
    click CB "https://github.com/ayush-patel-10/rag-using-azure-databricks-ci-cd/blob/main/README.md"

    %% Styles
    classDef azure fill:#f4a261,stroke:#000,stroke-width:2px;
    classDef databricks fill:#2a9d8f,stroke:#000,stroke-width:2px;
    classDef cicd fill:#457b9d,stroke:#000,stroke-width:2px;
    classDef terraform fill:#9b5de5,stroke:#000,stroke-width:2px;
    classDef mlflow fill:#f9c74f,stroke:#000,stroke-width:2px;
    classDef delta fill:#6b705c,stroke:#000,stroke-width:2px;
    classDef chatbot fill:#e76f51,stroke:#000,stroke-width:2px;
```
