# seldonio/mlserver architecture

> Architecture diagram of the GitHub repository seldonio/mlserver, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/seldonio/mlserver
- Repository: https://github.com/seldonio/mlserver

## Overview

##### Major Components
- Core Serving Engine – Represented by the mlserver directory.
- REST Interface: See mlserver/rest with endpoints, error handling, and OpenAPI specification.
- GRPC Interface: Found in mlserver/grpc; serves the gRPC endpoints (data plane and model repository protocols).
- CLI & Server Bootstrap: Modules such as mlserver/cli and mlserver/server.py set up and launch the server.
- Inference Processing Pipeline:
- Batch Processing Modules: Under mlserver/batch_processing and parts of mlserver/parallel, which handle adaptive batching, parallel inference, and worker pool management.
- Model Repository: In mlserver/repository – responsible for model discovery, loading, and management.
- Caching & Context Management: See mlserver/cache and mlserver/context, which manage temporary inference results and state.
- Inference Runtimes (Plugin/Adapter Components):
- These are separate packages under the “runtimes” directory (e.g., runtimes/sklearn, runtimes/xgboost, runtimes/mlflow, etc.). Each runtime provides the “glue” to work with specific ML libraries and includes custom codecs, wrappers, and settings.
- Auxiliary Services & Utilities:
- Logging, Metrics, and Tracing: Modules in mlserver/logging, mlserver/metrics, and mlserver/tracing.
- GRPC Interceptors and Custom Handlers: Enhance error handling and monitoring (e.g., in mlserver/grpc/interceptors.py).
- Configuration and Environment: Modules like mlserver/env.py and mlserver/settings.py govern how the engine is built and run.
- External Integration Interfaces:
- Kafka support found in mlserver/kafka indicates integration for streaming or event ingestion.
- The server also supports cloud events and deployment integrations, with additional connector code.

##### Relationships and Flow
- Start from the external client (or deployment framework such as Kubernetes/Seldon-Core/KServe) which issues an inference request.
- The request enters the system through REST or gRPC endpoints.
- The request is then passed to the core MLServer engine where it may be pre-processed and directed to:
- The batching/parallel modules if the request can be grouped or processed concurrently.
- The model repository which locates the proper model and its entry point.
- The inference runtime adapter (from one of the runtimes packages) receives the prepared request, calls the model, and returns an inference result.
- Utility layers such as caching, logging, and metrics wrap around these calls to ensure performance monitoring and error reporting.
- Finally, the response is sent back to the client through the same interface used to receive the request.

##### Important Architectural Patterns and Design Principles
- Modular Plugin Architecture: Inference runtimes are designed as plug‐ins, allowing additional runtimes to be added without changing core MLServer.
- Separation of Concerns: There is a clear separation between communication interfaces (REST/GRPC), the processing pipeline (batching, parallel processing, model repository), and the runtime integrations.
- Scalability & Extensibility: Adaptive batching and parallel dispatching allow the system to handle variable loads.
- Standard Protocols: MLServer adheres to standardized inference protocols (KFServing V2) for interoperability.

##### Putting It All Together
- That the engine is extensible so that adding a new inference runtime requires connecting a new module within the “Runtime Plugins” group.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Client["Client / External Request"]:::external

    %% Interface Layer
    subgraph "Interface Layer"
        REST_API["REST API"]:::interface
        GRPC_API["gRPC API"]:::interface
    end

    %% Core MLServer Engine Components
    subgraph "Core MLServer Engine"
        CoreEngine["Core Engine"]:::core
        BatchProc["Adaptive Batching & Parallel Processing"]:::processing
        ModelRepo["Model Repository & Loader"]:::processing
        Utilities["Caching, Logging & Metrics"]:::processing
        GRPCInt["gRPC Interceptors & Handlers"]:::processing
        Config["Configuration & Environment"]:::processing
    end

    %% Inference Runtimes (Plugin Adapters)
    subgraph "Inference Runtimes (Plugin Adapters)"
        SKRuntime["Sklearn Runtime"]:::runtime
        XGBRuntime["XGBoost Runtime"]:::runtime
        MLFlowRuntime["MLflow Runtime"]:::runtime
    end

    %% Auxiliary Components
    CLI_Admin["CLI & Administration Tools"]:::admin
    Kafka["External Integration (Kafka Support)"]:::external

    %% Flow Connections
    %% Client to Interfaces
    Client -->|"InferenceRequest(JSON)"| REST_API
    Client -->|"InferenceRequest(gRPC)"| GRPC_API

    %% Interfaces to Core Engine
    REST_API -->|"ForwardRequest"| CoreEngine
    GRPC_API -->|"ForwardRequest"| CoreEngine

    %% Inside Core Engine processing pipeline
    CoreEngine -->|"Dispatch"| BatchProc
    CoreEngine -->|"ModelLookup"| ModelRepo
    CoreEngine -->|"UtilitiesWrapper"| Utilities
    CoreEngine -->|"Interceptor"| GRPCInt
    CoreEngine -->|"ConfigLoad"| Config

    %% Core Engine to Inference Runtimes
    ModelRepo -->|"InvokeInference"| SKRuntime
    ModelRepo -->|"InvokeInference"| XGBRuntime
    ModelRepo -->|"InvokeInference"| MLFlowRuntime

    %% Runtimes return inference results to Core Engine
    SKRuntime -->|"Result"| CoreEngine
    XGBRuntime -->|"Result"| CoreEngine
    MLFlowRuntime -->|"Result"| CoreEngine

    %% Core Engine responds to Client
    CoreEngine -->|"Response"| Client

    %% Auxiliary Flow
    CLI_Admin -->|"Manage/Configure"| CoreEngine
    Kafka -->|"StreamEvents"| CoreEngine

    %% Click Events for Component Mapping
    click CoreEngine "https://github.com/seldonio/mlserver/tree/master/mlserver/"
    click REST_API "https://github.com/seldonio/mlserver/tree/master/mlserver/rest/"
    click GRPC_API "https://github.com/seldonio/mlserver/tree/master/mlserver/grpc/"
    click CLI_Admin "https://github.com/seldonio/mlserver/tree/master/mlserver/cli/"
    click CLI_Admin "https://github.com/seldonio/mlserver/blob/master/mlserver/server.py"
    click BatchProc "https://github.com/seldonio/mlserver/blob/master/mlserver/batch_processing.py"
    click BatchProc "https://github.com/seldonio/mlserver/tree/master/mlserver/parallel/"
    click ModelRepo "https://github.com/seldonio/mlserver/tree/master/mlserver/repository/"
    click Utilities "https://github.com/seldonio/mlserver/tree/master/mlserver/cache/"
    click Utilities "https://github.com/seldonio/mlserver/blob/master/mlserver/context.py"
    click Utilities "https://github.com/seldonio/mlserver/blob/master/mlserver/logging.py"
    click Utilities "https://github.com/seldonio/mlserver/tree/master/mlserver/metrics/"
    click Utilities "https://github.com/seldonio/mlserver/blob/master/mlserver/tracing.py"
    click GRPCInt "https://github.com/seldonio/mlserver/blob/master/mlserver/grpc/interceptors.py"
    click GRPCInt "https://github.com/seldonio/mlserver/tree/master/mlserver/handlers/"
    click Config "https://github.com/seldonio/mlserver/blob/master/mlserver/env.py"
    click Config "https://github.com/seldonio/mlserver/blob/master/mlserver/settings.py"
    click SKRuntime "https://github.com/seldonio/mlserver/tree/master/runtimes/sklearn"
    click XGBRuntime "https://github.com/seldonio/mlserver/tree/master/runtimes/xgboost"
    click MLFlowRuntime "https://github.com/seldonio/mlserver/tree/master/runtimes/mlflow"
    click Kafka "https://github.com/seldonio/mlserver/tree/master/mlserver/kafka/"

    %% Styles
    classDef external fill:#E8DAEF,stroke:#6A1B9A,stroke-width:2px;
    classDef interface fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef core fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef processing fill:#D5F5E3,stroke:#117864,stroke-width:2px;
    classDef runtime fill:#FADBD8,stroke:#943126,stroke-width:2px;
    classDef admin fill:#F5EEF8,stroke:#6C3483,stroke-width:2px;
```
