# opencsgs/llm-inference architecture

> Architecture diagram of the GitHub repository opencsgs/llm-inference, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/opencsgs/llm-inference
- Repository: https://github.com/opencsgs/llm-inference

## Overview

##### Main Components

a) Client Layer:
- REST API Interface
- Web UI Interface
- SDK Interface (llmserve/api/sdk.py)

b) Frontend Service:
- Web Interface Components (llmserve/frontend)
- Leaderboard System
- MongoDB Logger for Analytics

c) Backend Service:
- API Server (llmserve/backend/server)
- LLM Engine Manager (llmserve/backend/llm/engines)
- Model Initializers (llmserve/backend/llm/initializers)
- Pipeline Processors (llmserve/backend/llm/pipelines)

d) Ray Cluster:
- Resource Management
- Auto-scaling Components
- GPU Executor (vllm specific)

e) Storage Systems:
- Model Storage (supporting multiple sources)
- OpenCSG Model Hub
- Hugging Face Hub
- S3 Storage
- Local Storage
- MinIO for Object Storage

##### Component Relationships

- Client requests → Frontend/Backend Services
- Backend → Ray Cluster for resource management
- Ray Cluster → Model Storage for model loading
- Backend → Storage Systems for model retrieval
- Frontend → MongoDB for logging and analytics

##### Special Architectural Elements

Specific sections showing:
- Serverless scaling mechanism
- Model Registry system
- Streaming capabilities
- Resource distribution flow

##### Technical Details to Highlight

- Supported frameworks (VLLM, DeepSpeed, GGUF, HF Transformers)
- Pipeline architecture
- Scaling policies
- Resource management flow

##### Additional Elements

Include:
- System boundaries
- Load balancing components
- Monitoring and logging flow
- Configuration management

## Diagram (Mermaid)

```mermaid
graph TB
    %% Client Layer
    subgraph Clients
        SDK["SDK Interface"]:::client
        CLI["CLI Interface"]:::client
        WEB["Web Interface"]:::client
        REST["REST API"]:::client
    end

    %% Frontend Service
    subgraph Frontend
        WEBUI["Web UI Components"]:::frontend
        LB["Leaderboard System"]:::frontend
        MLOG["MongoDB Logger"]:::frontend
    end

    %% Backend Service
    subgraph Backend
        API["API Server"]:::backend
        ENGMGR["LLM Engine Manager"]:::backend
        MODINIT["Model Initializers"]:::backend
        PIPE["Pipeline Processors"]:::backend
    end

    %% Ray Cluster
    subgraph RayCluster
        RESMAN["Resource Management"]:::ray
        AUTOSCALE["Auto-scaling"]:::ray
        GPUEXEC["GPU Executor"]:::ray
    end

    %% Storage Systems
    subgraph Storage
        subgraph ModelStorage
            OPENCSG["OpenCSG Hub"]:::storage
            HF["Hugging Face Hub"]:::storage
            S3["S3 Storage"]:::storage
            LOCAL["Local Storage"]:::storage
        end
        MINIO["MinIO Object Storage"]:::storage
    end

    %% Relationships
    SDK --> API
    CLI --> API
    WEB --> WEBUI
    REST --> API
    
    WEBUI --> API
    WEBUI --> LB
    WEBUI --> MLOG
    
    API --> ENGMGR
    ENGMGR --> MODINIT
    MODINIT --> PIPE
    
    ENGMGR --> RESMAN
    RESMAN --> AUTOSCALE
    AUTOSCALE --> GPUEXEC
    
    MODINIT --> ModelStorage
    PIPE --> MINIO
    
    %% Styles
    classDef client fill:#f9f,stroke:#333
    classDef frontend fill:#9cf,stroke:#333
    classDef backend fill:#9fc,stroke:#333
    classDef ray fill:#ff9,stroke:#333
    classDef storage fill:#fc9,stroke:#333

    %% Click Events
    click SDK "https://github.com/OpenCSGs/llm-inference/blob/main/llmserve/api/sdk.py"
    click CLI "https://github.com/OpenCSGs/llm-inference/blob/main/llmserve/api/cli.py"
    click WEB "https://github.com/OpenCSGs/llm-inference/tree/main/llmserve/frontend/"
    click LB "https://github.com/OpenCSGs/llm-inference/blob/main/llmserve/frontend/leaderboard.py"
    click MLOG "https://github.com/OpenCSGs/llm-inference/blob/main/llmserve/frontend/mongo_logger.py"
    click API "https://github.com/OpenCSGs/llm-inference/tree/main/llmserve/backend/server/"
    click ENGMGR "https://github.com/OpenCSGs/llm-inference/tree/main/llmserve/backend/llm/engines/"
    click MODINIT "https://github.com/OpenCSGs/llm-inference/tree/main/llmserve/backend/llm/initializers/"
    click PIPE "https://github.com/OpenCSGs/llm-inference/tree/main/llmserve/backend/llm/pipelines/"
    click GPUEXEC "https://github.com/OpenCSGs/llm-inference/blob/main/llmserve/backend/llm/engines/vllm/ray_gpu_executor.py"
    click RESMAN "https://github.com/OpenCSGs/llm-inference/tree/main/deploy/ray/"
    click MINIO "https://github.com/OpenCSGs/llm-inference/tree/main/deploy/minio/"

    %% Legend
    subgraph Legend
        C1["Client Component"]:::client
        F1["Frontend Component"]:::frontend
        B1["Backend Component"]:::backend
        R1["Ray Component"]:::ray
        S1["Storage Component"]:::storage
    end
```
