How seldonio/mlserver works
Major Components
• Core Serving Engine – Represented by the mlserver directory.
• REST Interface: See mlserver/rest with endpoints, error handling, and OpenAPI specification.
• GRPC Interface: Found in mlserver/grpc; serves the gRPC endpoints (data plane and model repository protocols).
• CLI & Server Bootstrap: Modules such as mlserver/cli and mlserver/server.py set up and launch the server.
• Inference Processing Pipeline:
• Batch Processing Modules: Under mlserver/batch_processing and parts of mlserver/parallel, which handle adaptive batching, parallel inference, and worker pool management.
• Model Repository: In mlserver/repository – responsible for model discovery, loading, and management.
• Caching & Context Management: See mlserver/cache and mlserver/context, which manage temporary inference results and state.
• Inference Runtimes (Plugin/Adapter Components):
• These are separate packages under the “runtimes” directory (e.g., runtimes/sklearn, runtimes/xgboost, runtimes/mlflow, etc.). Each runtime provides the “glue” to work with specific ML libraries and includes custom codecs, wrappers, and settings.
• Auxiliary Services & Utilities:
• Logging, Metrics, and Tracing: Modules in mlserver/logging, mlserver/metrics, and mlserver/tracing.
• GRPC Interceptors and Custom Handlers: Enhance error handling and monitoring (e.g., in mlserver/grpc/interceptors.py).
• Configuration and Environment: Modules like mlserver/env.py and mlserver/settings.py govern how the engine is built and run.
• External Integration Interfaces:
• Kafka support found in mlserver/kafka indicates integration for streaming or event ingestion.
• The server also supports cloud events and deployment integrations, with additional connector code.
Relationships and Flow
• Start from the external client (or deployment framework such as Kubernetes/Seldon-Core/KServe) which issues an inference request.
• The request enters the system through REST or gRPC endpoints.
• The request is then passed to the core MLServer engine where it may be pre-processed and directed to:
• The batching/parallel modules if the request can be grouped or processed concurrently.
• The model repository which locates the proper model and its entry point.
• The inference runtime adapter (from one of the runtimes packages) receives the prepared request, calls the model, and returns an inference result.
• Utility layers such as caching, logging, and metrics wrap around these calls to ensure performance monitoring and error reporting.
• Finally, the response is sent back to the client through the same interface used to receive the request.
Important Architectural Patterns and Design Principles
• Modular Plugin Architecture: Inference runtimes are designed as plug‐ins, allowing additional runtimes to be added without changing core MLServer.
• Separation of Concerns: There is a clear separation between communication interfaces (REST/GRPC), the processing pipeline (batching, parallel processing, model repository), and the runtime integrations.
• Scalability & Extensibility: Adaptive batching and parallel dispatching allow the system to handle variable loads.
• Standard Protocols: MLServer adheres to standardized inference protocols (KFServing V2) for interoperability.
Putting It All Together
• That the engine is extensible so that adding a new inference runtime requires connecting a new module within the “Runtime Plugins” group.