# siriusinftra/sirius architecture

> Architecture diagram of the GitHub repository siriusinftra/sirius, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/siriusinftra/sirius
- Repository: https://github.com/siriusinftra/sirius

## Overview

##### Project type and purpose
- This is a full-stack ML system for colocating inference and training workloads on shared GPUs with a custom fast GPU memory handover mechanism.
- It comprises a C++ inference server (with gRPC API, scheduler, model store, resource manager), a C++/Python client workload launcher, Python training scripts, and a PyTorch plugin.

##### Top-level components (from file structure)
- client/ : C++ command-line benchmark and workload launcher (busy loops, Triton/Sirius implementations).
- server/ : C++ inference server
- control/ : controller for MPS percentage
- grpc/ : gRPC server entry points (proto → service)
- llm/ : LLM support (C++ allocator plugin + Python binding)
- model_store/ : inference model loader & cache (TVM & Triton models)
- schedule/ : job and exec queues, scheduling logic
- resource_manager: orchestrates GPU memory pools
- common/ : shared C++ libraries (tensor abstractions, device manager, GPU memory mpool, inter-process communication via inf_tra_comm)
- pytorch/ : PyTorch plugin “torch_col” (C++/Cython) that intercepts allocator calls, hooks into common library and xsched
- eval/ : Python scripts for automated benchmarking (launch MPS, server, training tasks, inference workloads)
- train/ : standalone Python training scripts using the PyTorch plugin
- util/ : helper scripts (model preparation, ONNX⇄Triton, profiling, data dumps)
- proto/ : gRPC protocol definition for client-server RPC
- third_party/ : external libraries (mpool, xsched scheduler, CLI11, glog, grpc, etc.)
- environment/, scripts/ : Dockerfiles, conda specs, helper shell scripts

##### Core architectural patterns
- Layered server:
- API layer (gRPC service)
- Control plane (MPS configuration)
- Scheduler (xsched-based job queues)
- Execution layer (CUDA/Triton/TVM/PyTorch)
- Plugin pattern: custom GPU allocator plugin used by both server and PyTorch (via “torch_col”) to implement shared GPU memory handover
- Shared library (common/) for unified tensor abstraction, device management, inter-process synchronization

##### Final advice
- Every essential component (client, server layers, plugins, hardware) without over-aggregating
- The result will clearly map how inference/training requests travel from client/scripts, through server or PyTorch plugin, to GPUs, and back.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Processes
    Client["Client Process\n(C++ workload launcher)"]:::external
    Training["Python Training Scripts"]:::external
    EvalHarness["Evaluation Harness"]:::external

    %% Server Box
    subgraph "Server Process (C++)"
        direction TB
        API["gRPC API Layer"]:::server
        Control["Control Plane\n(MPS Config)"]:::server
        Scheduler["Scheduler\n(xsched)"]:::server
        ResourceMgr["Resource Manager\n(GPU-mempool)"]:::server
        ModelStore["Model Store\n(TVM & Triton)"]:::server
        Execution["Execution Engine\n(TVM/Triton/PyTorch)"]:::server
        LLM["LLM Support Plugin"]:::server
    end

    %% Shared/Plugins
    CommonLib["Common C++ Libraries"]:::shared
    GPUAlloc["GPU Memory Allocator\n(common/cuda_allocator)"]:::shared
    PyTorchPlugin["PyTorch Plugin\n(torch_col)"]:::shared

    %% GPU Hardware
    GPU["GPU Hardware\n(CUDA Driver/MPS/Unified Memory)"]:::gpu

    %% Relationships
    Client -->|gRPC| API
    API -->|set MPS %| Control
    API -->|enqueue job| Scheduler
    Scheduler -->|allocate| ResourceMgr
    ResourceMgr -->|use allocator| GPUAlloc
    Scheduler -->|dispatch| Execution
    Execution -->|invoke| ModelStore
    Execution -->|use TVM| TVMSub["TVM Backend"]:::server
    Execution -->|use Triton| TritonSub["Triton Backend"]:::server
    Execution -->|use PyTorch| PyTorchPlugin
    LLM -->|binds to| Execution
    Training -->|via plugin| PyTorchPlugin
    PyTorchPlugin -->|calls| CommonLib
    CommonLib -->|manage| GPUAlloc
    CommonLib -->|IPC sync| Scheduler
    API -->|proto definitions| Proto["colserve.proto"]:::server
    API --> Proto
    ResourceMgr --> GPU
    Execution --> GPU
    Training --> GPUAlloc
    EvalHarness -.->|launch| Client
    EvalHarness -.->|launch| API
    EvalHarness -.->|launch| Training
    EvalHarness -.->|launch| GPU

    %% Click Events
    click Client "https://github.com/siriusinftra/sirius/tree/main/client/src"
    click API "https://github.com/siriusinftra/sirius/tree/main/server/grpc"
    click Proto "https://github.com/siriusinftra/sirius/blob/main/proto/colserve.proto"
    click Control "https://github.com/siriusinftra/sirius/tree/main/server/control"
    click Scheduler "https://github.com/siriusinftra/sirius/tree/main/server/schedule"
    click ResourceMgr "https://github.com/siriusinftra/sirius/blob/main/server/resource_manager.cc"
    click GPUAlloc "https://github.com/siriusinftra/sirius/blob/main/common/cuda_allocator.cc"
    click ModelStore "https://github.com/siriusinftra/sirius/tree/main/server/model_store"
    click TVMSub "https://github.com/siriusinftra/sirius/tree/main/server/tvm"
    click TritonSub "https://github.com/siriusinftra/sirius/tree/main/server/triton_models"
    click LLM "https://github.com/siriusinftra/sirius/tree/main/server/llm"
    click CommonLib "https://github.com/siriusinftra/sirius/tree/main/common"
    click PyTorchPlugin "https://github.com/siriusinftra/sirius/tree/main/pytorch/torch_col"
    click Training "https://github.com/siriusinftra/sirius/tree/main/train"
    click EvalHarness "https://github.com/siriusinftra/sirius/tree/main/eval"

    %% Styles
    classDef external fill:#BBDEFB,stroke:#1E88E5,color:#0D47A1
    classDef server fill:#C8E6C9,stroke:#2E7D32,color:#1B5E20
    classDef shared fill:#FFF9C4,stroke:#FBC02D,color:#F57F17
    classDef gpu fill:#FFE0B2,stroke:#E65100,color:#BF360C
```
