# internlm/lmdeploy architecture

> Architecture diagram of the GitHub repository internlm/lmdeploy, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/internlm/lmdeploy
- Repository: https://github.com/internlm/lmdeploy

## Overview

##### Main Components

a. Core Engines:
- TurboMind Engine (C++/CUDA based)
- PyTorch Engine (Python based)

b. Service Layer:
- API Server
- Proxy Server (for multi-model, multi-machine deployment)
- Gradio Interface

c. Model Support:
- LLM Models
- VLM Models
- Quantization Module

d. Pipeline Components:
- Inference Pipeline
- Chat Pipeline
- Vision-Language Pipeline

##### Component Relationships

a. Data Flow:
- Input Processing → Model Inference → Output Generation
- Request Distribution → Load Balancing → Model Selection
- Image/Text Processing → Multimodal Fusion → Response Generation

b. Integration Points:
- Model Loading Interface
- Quantization Pipeline
- Serving APIs

##### Architectural Patterns

- Modular Architecture
- Pipeline Pattern
- Service-Oriented Architecture
- Plugin Architecture (for model support)

**[Client Layer]**

- ▼
**[Service Layer]**
- API Server
- Proxy Server
- Gradio Interface

- ▼
**[Pipeline Layer]**
- Inference Pipeline
- Chat Pipeline
- Vision-Language Pipeline

- ▼
**[Engine Layer]**
- TurboMind Engine
- CUDA Kernels
- Attention Implementation
- Memory Management

- PyTorch Engine
- Python Components
- Model Implementations
- Runtime Optimizations

- ▼
**[Model Layer]**
- LLM Support
- VLM Support
- Quantization Module

##### Additional Elements

Annotations for:
- Performance Optimizations
- Quantization Methods
- Supported Model Types
- Deployment Options

##### Implementation Details

- CUDA-based optimizations
- Tensor parallelism
- KV cache management
- Quantization techniques (W4A16, KV Cache)
- Multi-GPU support

##### Key Features to Emphasize

- Persistent batch processing
- Blocked KV cache
- Dynamic split & fuse
- Interactive inference mode
- Distributed serving capabilities

##### Technical Specifications

Notes on:
- Supported CUDA versions
- Python version requirements
- Hardware requirements
- Performance metrics

##### Integration Points

Connections with:
- HuggingFace Hub
- ModelScope
- OpenMind Hub
- External monitoring tools

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client
        CL[Client Applications]
    end

    subgraph ServiceLayer
        AS["API Server"]:::service
        PS["Proxy Server"]:::service
        GI["Gradio Interface"]:::service
    end

    subgraph PipelineLayer
        IP["Inference Pipeline"]:::pipeline
        CP["Chat Pipeline"]:::pipeline
        VLP["Vision-Language Pipeline"]:::pipeline
    end

    subgraph EngineLayer
        subgraph TurboMind
            TE["TurboMind Engine"]:::engine
            CK["CUDA Kernels"]:::cuda
            AI["Attention Implementation"]:::cuda
            MM["Memory Management"]:::system
        end
        
        subgraph PyTorch
            PE["PyTorch Engine"]:::engine
            PC["Python Components"]:::python
            MI["Model Implementations"]:::python
            RO["Runtime Optimizations"]:::system
        end
    end

    subgraph ModelLayer
        MLI["Model Loading Interface"]:::model
        LS["LLM Support"]:::model
        VS["VLM Support"]:::model
        QM["Quantization Module"]:::model
    end

    %% Connections
    CL --> AS & PS & GI
    AS & PS & GI --> IP & CP & VLP
    IP & CP & VLP --> TE & PE
    TE --> CK & AI & MM
    PE --> PC & MI & RO
    TE & PE --> MLI
    MLI --> LS & VS & QM

    %% Click Events
    click TE "https://github.com/InternLM/lmdeploy/tree/main/src/turbomind/"
    click PE "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/pytorch/"
    click AS "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/serve/openai/"
    click PS "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/serve/proxy/"
    click GI "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/serve/gradio/"
    click LS "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/turbomind/deploy/source_model/"
    click VS "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/vl/model/"
    click QM "https://github.com/InternLM/lmdeploy/tree/main/lmdeploy/lite/quantization/"
    click CP "https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/pytorch/chat.py"
    click VLP "https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/vl/engine.py"
    click CK "https://github.com/InternLM/lmdeploy/tree/main/src/turbomind/kernels/"
    click AI "https://github.com/InternLM/lmdeploy/tree/main/src/turbomind/kernels/attention/"
    click MM "https://github.com/InternLM/lmdeploy/blob/main/src/turbomind/utils/memory_utils.h"
    click MLI "https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/turbomind/deploy/loader.py"

    %% Styling
    classDef service fill:#90EE90,stroke:#333,stroke-width:2px
    classDef pipeline fill:#FFA500,stroke:#333,stroke-width:2px
    classDef engine fill:#4169E1,stroke:#333,stroke-width:2px
    classDef cuda fill:#6495ED,stroke:#333,stroke-width:2px
    classDef python fill:#87CEEB,stroke:#333,stroke-width:2px
    classDef system fill:#D3D3D3,stroke:#333,stroke-width:2px
    classDef model fill:#DDA0DD,stroke:#333,stroke-width:2px
```
