How kserve/modelmesh-serving works
Main Components
• Controller/Operator Layer – Represent the ModelMesh Serving controller which is responsible for orchestrating changes across the Kubernetes cluster. Indicate separate controllers within this layer such as:
• Predictor Controller (handles predictor CRDs)
• ServingRuntime Controller (manages ServingRuntime CRDs)
• Autoscaler and HPA Controllers (present under “controllers/autoscaler” and “controllers/hpa”)
• ModelMesh-specific controllers (in “controllers/modelmesh”) that deal with model routing, endpoint management, model puller logic, and TLS/etcd integration.
• API/CRD Definitions – Highlight the API objects defined in the “apis/serving/v1alpha1” directory and the CRD manifests under “config/crd”. These objects (like InferenceService, Predictor, ServingRuntime) are the entry points for users and for the operator to watch changes.
• Configuration & Deployment – Include the “config” directory (with its different subfolders for certmanager, default, dependencies, manager, webhook, etc.) which is used to set up the controller’s runtime environment, managing deployments, webhooks, and RBAC.
• Supporting Libraries & Internal Packages – Show the “pkg” directory which includes configuration logic, GRPC resolvers, and predictor source management; this supports the core controller behavior.
• External Components (Interfacing Systems) – Although part of a larger ecosystem, illustrate the integration points:
• ModelMesh Containers (from the separate “modelmesh” repository responsible for routing and load-balancing model requests)
• Runtime Adapters (the external “modelmesh-runtime-adapter” that interfaces with third-party model servers)
• Third-Party Model Serving Runtimes (e.g., Triton, MLServer, TorchServe, OpenVINO) which are targets for serving models.
• Testing & CI/CD – Optionally, include elements for continuous integration and testing (e.g., GitHub workflows, FVT components under “fvt”) to emphasize the robust testing and deployment automation.
Relationships and Data Flow Between Components
• User or automated requests create or update CRDs via the Kubernetes API server.
• The ModelMesh Serving controller watches these CRDs, triggering its controllers (e.g., the predictor and servingruntime controllers).
• Controllers interact with underlying components such as updating configurations (using “pkg/config”), managing endpoints, or instructing the runtime adapter/container (which in turn manages model loading, pulling, or unloading).
• Communication channels such as gRPC (shown in “pkg/mmesh/grpc_resolver.go”) and etcd integration (for service discovery) should be depicted.
• Various control loops, reconciliation processes, and how the operator eventually triggers changes within the Kubernetes cluster (like deployments or webhooks).
Architectural Patterns and Design Principles
• Operator Pattern – Show that the system is built around a reconciliation loop pattern typical in Kubernetes operators.
• CRD-driven Configuration – Emphasize the use of Kubernetes CRDs, so that the system is extensible and declarative.
• Modularity and Separation of Concerns – Indicate separate controllers and libraries that encapsulate different responsibilities (scaling, prediction handling, model placement, etc.).
• Integration with external systems – Demonstrate how the controller connects with model serving runtimes and with supporting runtime adapters.
Technologies, Frameworks, and Tools
• Use of Go as the primary programming language.
• Kubernetes as the underlying platform (with CRDs, controllers, webhooks).
• Mention integration with CI/CD tools (workflows defined in .github/workflows) and configuration management tools (Kustomize in “config/*” directories).
• Testing frameworks, as indicated by FVT (functional verification tests) and integration tests in controllers and pkg directories.