# kserve/modelmesh-serving architecture

> Architecture diagram of the GitHub repository kserve/modelmesh-serving, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kserve/modelmesh-serving
- Repository: https://github.com/kserve/modelmesh-serving

## Overview

##### Main Components
- Controller/Operator Layer – Represent the ModelMesh Serving controller which is responsible for orchestrating changes across the Kubernetes cluster. Indicate separate controllers within this layer such as:
- Predictor Controller (handles predictor CRDs)
- ServingRuntime Controller (manages ServingRuntime CRDs)
- Autoscaler and HPA Controllers (present under “controllers/autoscaler” and “controllers/hpa”)
- ModelMesh-specific controllers (in “controllers/modelmesh”) that deal with model routing, endpoint management, model puller logic, and TLS/etcd integration.
- API/CRD Definitions – Highlight the API objects defined in the “apis/serving/v1alpha1” directory and the CRD manifests under “config/crd”. These objects (like InferenceService, Predictor, ServingRuntime) are the entry points for users and for the operator to watch changes.
- Configuration & Deployment – Include the “config” directory (with its different subfolders for certmanager, default, dependencies, manager, webhook, etc.) which is used to set up the controller’s runtime environment, managing deployments, webhooks, and RBAC.
- Supporting Libraries & Internal Packages – Show the “pkg” directory which includes configuration logic, GRPC resolvers, and predictor source management; this supports the core controller behavior.
- External Components (Interfacing Systems) – Although part of a larger ecosystem, illustrate the integration points:
- ModelMesh Containers (from the separate “modelmesh” repository responsible for routing and load-balancing model requests)
- Runtime Adapters (the external “modelmesh-runtime-adapter” that interfaces with third-party model servers)
- Third-Party Model Serving Runtimes (e.g., Triton, MLServer, TorchServe, OpenVINO) which are targets for serving models.
- Testing & CI/CD – Optionally, include elements for continuous integration and testing (e.g., GitHub workflows, FVT components under “fvt”) to emphasize the robust testing and deployment automation.

##### Relationships and Data Flow Between Components
- User or automated requests create or update CRDs via the Kubernetes API server.
- The ModelMesh Serving controller watches these CRDs, triggering its controllers (e.g., the predictor and servingruntime controllers).
- Controllers interact with underlying components such as updating configurations (using “pkg/config”), managing endpoints, or instructing the runtime adapter/container (which in turn manages model loading, pulling, or unloading).
- Communication channels such as gRPC (shown in “pkg/mmesh/grpc_resolver.go”) and etcd integration (for service discovery) should be depicted.
- Various control loops, reconciliation processes, and how the operator eventually triggers changes within the Kubernetes cluster (like deployments or webhooks).

##### Architectural Patterns and Design Principles
- Operator Pattern – Show that the system is built around a reconciliation loop pattern typical in Kubernetes operators.
- CRD-driven Configuration – Emphasize the use of Kubernetes CRDs, so that the system is extensible and declarative.
- Modularity and Separation of Concerns – Indicate separate controllers and libraries that encapsulate different responsibilities (scaling, prediction handling, model placement, etc.).
- Integration with external systems – Demonstrate how the controller connects with model serving runtimes and with supporting runtime adapters.

##### Technologies, Frameworks, and Tools
- Use of Go as the primary programming language.
- Kubernetes as the underlying platform (with CRDs, controllers, webhooks).
- Mention integration with CI/CD tools (workflows defined in .github/workflows) and configuration management tools (Kustomize in “config/*” directories).
- Testing frameworks, as indicated by FVT (functional verification tests) and integration tests in controllers and pkg directories.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Input
    UserCRUD["User/CRD Request"]:::user

    %% API/CRD Definitions
    subgraph "API/CRD Definitions"
        API["CRD: API Objects"]:::crd
        CRDM["CRD: Manifests"]:::crd
    end

    %% Controller/Operator Layer
    subgraph "Controller/Operator Layer"
        PRED["Predictor Controller"]:::controller
        SR["ServingRuntime Controller"]:::controller
        AUTO["Autoscaler Controller"]:::controller
        HPA["HPA Controller"]:::controller
        MMCTRL["ModelMesh Controllers"]:::controller
    end

    %% Supporting Libraries & Internal Packages
    subgraph "Supporting Libraries & Internal Packages"
        CORE["Core Config Logic"]:::lib
        MMINT["ModelMesh Integration"]:::lib
        PS["Predictor Source Mgmt"]:::lib
    end

    %% Configuration & Deployment
    subgraph "Configuration & Deployment"
        MGR["Controller Runtime Config"]:::config
        DEFAULT["Default Configurations"]:::config
        WEBHOOK["Webhook Config"]:::config
        RBAC["RBAC & Deployment"]:::config
        ADDM["Additional Manifests"]:::config
    end

    %% External Components & Runtime Adapters
    subgraph "External Components & Runtime Adapters"
        SERVRUNT["Model Serving Runtimes Config"]:::external
        MMAD["ModelMesh Adapter"]:::external
    end

    %% CI/CD & Testing
    subgraph "CI/CD & Testing"
        CICD["CI/CD Pipelines"]:::cicd
        FVT["Functional Verification Tests"]:::cicd
    end

    %% Connections
    UserCRUD --> API
    API --> PRED
    API --> SR
    API --> AUTO
    API --> HPA
    API --> MMCTRL
    API --- CRDM

    %% Controllers to Supporting Libraries
    PRED --> CORE
    SR --> CORE
    AUTO --> CORE
    HPA --> CORE
    MMCTRL --> CORE

    PRED --> MMINT
    SR --> MMINT
    AUTO --> MMINT
    HPA --> MMINT
    MMCTRL --> MMINT

    PRED --> PS
    SR --> PS
    AUTO --> PS
    HPA --> PS
    MMCTRL --> PS

    %% Libraries to Configuration & Deployment
    CORE --> DEFAULT
    CORE --> MGR
    MMINT --> WEBHOOK
    MMINT --> RBAC
    PS --> DEFAULT

    %% Controllers directly influence External Components
    MMCTRL --> SERVRUNT
    MMCTRL --> MMAD

    %% Configuration & Deployment to External Components
    DEFAULT --> SERVRUNT
    MGR --> SERVRUNT
    WEBHOOK --> MMAD
    RBAC --> MMAD
    ADDM --> MMAD

    %% External Components to CI/CD & Testing
    SERVRUNT --> CICD
    MMAD --> CICD
    SERVRUNT --> FVT
    MMAD --> FVT

    %% CI/CD feedback loop
    CICD --> UserCRUD

    %% Click Events
    click PRED "https://github.com/kserve/modelmesh-serving/blob/main/controllers/predictor_controller.go"
    click SR "https://github.com/kserve/modelmesh-serving/blob/main/controllers/servingruntime_controller.go"
    click AUTO "https://github.com/kserve/modelmesh-serving/blob/main/controllers/autoscaler/autoscaler_reconciler.go"
    click HPA "https://github.com/kserve/modelmesh-serving/blob/main/controllers/hpa/hpa_reconciler.go"
    click MMCTRL "https://github.com/kserve/modelmesh-serving/blob/main/controllers/modelmesh/modelmesh.go"
    click API "https://github.com/kserve/modelmesh-serving/blob/main/apis/serving/v1alpha1/predictor_types.go"
    click CRDM "https://github.com/kserve/modelmesh-serving/blob/main/config/crd/bases/serving.kserve.io_predictors.yaml"
    click MGR "https://github.com/kserve/modelmesh-serving/blob/main/config/manager/manager.yaml"
    click DEFAULT "https://github.com/kserve/modelmesh-serving/blob/main/config/default/config-defaults.yaml"
    click WEBHOOK "https://github.com/kserve/modelmesh-serving/blob/main/config/webhook/service.yaml"
    click RBAC "https://github.com/kserve/modelmesh-serving/tree/main/config/rbac/"
    click ADDM "https://github.com/kserve/modelmesh-serving/tree/main/config/certmanager/"
    click CORE "https://github.com/kserve/modelmesh-serving/blob/main/pkg/config/config.go"
    click MMINT "https://github.com/kserve/modelmesh-serving/blob/main/pkg/mmesh/grpc_resolver.go"
    click PS "https://github.com/kserve/modelmesh-serving/blob/main/pkg/predictor_source/predictor_source.go"
    click SERVRUNT "https://github.com/kserve/modelmesh-serving/blob/main/config/runtimes/triton-2.x.yaml"
    click CICD "https://github.com/kserve/modelmesh-serving/blob/main/.github/workflows/build.yml"
    click FVT "https://github.com/kserve/modelmesh-serving/tree/main/fvt/"

    %% Class Definitions for Colors & Styles
    classDef controller fill:#A3BE8C,stroke:#2E3440,stroke-width:2px;
    classDef crd fill:#81A1C1,stroke:#2E3440,stroke-width:2px;
    classDef config fill:#D08770,stroke:#2E3440,stroke-width:2px;
    classDef lib fill:#EBCB8B,stroke:#2E3440,stroke-width:2px;
    classDef external fill:#B48EAD,stroke:#2E3440,stroke-width:2px;
    classDef cicd fill:#BF616A,stroke:#2E3440,stroke-width:2px;
    classDef user fill:#88C0D0,stroke:#2E3440,stroke-width:2px;
```
