How siriusinftra/sirius works
Project type and purpose
• This is a full-stack ML system for colocating inference and training workloads on shared GPUs with a custom fast GPU memory handover mechanism.
• It comprises a C++ inference server (with gRPC API, scheduler, model store, resource manager), a C++/Python client workload launcher, Python training scripts, and a PyTorch plugin.
Top-level components (from file structure)
• client/ : C++ command-line benchmark and workload launcher (busy loops, Triton/Sirius implementations).
• server/ : C++ inference server
• control/ : controller for MPS percentage
• grpc/ : gRPC server entry points (proto → service)
• llm/ : LLM support (C++ allocator plugin + Python binding)
• model_store/ : inference model loader & cache (TVM & Triton models)
• schedule/ : job and exec queues, scheduling logic
• resource_manager: orchestrates GPU memory pools
• common/ : shared C++ libraries (tensor abstractions, device manager, GPU memory mpool, inter-process communication via inf_tra_comm)
• pytorch/ : PyTorch plugin “torch_col” (C++/Cython) that intercepts allocator calls, hooks into common library and xsched
• eval/ : Python scripts for automated benchmarking (launch MPS, server, training tasks, inference workloads)
• train/ : standalone Python training scripts using the PyTorch plugin
• util/ : helper scripts (model preparation, ONNX⇄Triton, profiling, data dumps)
• proto/ : gRPC protocol definition for client-server RPC
• third_party/ : external libraries (mpool, xsched scheduler, CLI11, glog, grpc, etc.)
• environment/, scripts/ : Dockerfiles, conda specs, helper shell scripts
Core architectural patterns
• Layered server:
• API layer (gRPC service)
• Control plane (MPS configuration)
• Scheduler (xsched-based job queues)
• Execution layer (CUDA/Triton/TVM/PyTorch)
• Plugin pattern: custom GPU allocator plugin used by both server and PyTorch (via “torch_col”) to implement shared GPU memory handover
• Shared library (common/) for unified tensor abstraction, device management, inter-process synchronization
Final advice
• Every essential component (client, server layers, plugins, hardware) without over-aggregating
• The result will clearly map how inference/training requests travel from client/scripts, through server or PyTorch plugin, to GPUs, and back.