# modeltc/lightllm architecture

> Architecture diagram of the GitHub repository modeltc/lightllm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/modeltc/lightllm
- Repository: https://github.com/modeltc/lightllm

## Overview

##### Project type and purpose
- LightLLM is a full-stack, Python-based LLM inference and serving framework.
- It provides a lightweight, high-performance engine for loading myriad open-source model implementations (e.g., Llama, InternLM, DeepSeek2), running inference on GPU (via Triton kernels, FlashAttention, quantization), and serving results over HTTP/WebSocket/RPC.

##### High-level component analysis
From the file tree and README, the key top-level directories are:
- lightllm/common: core abstractions (Basemodel, memory managers, CUDA wrappers, kernel configs, quantization)
- lightllm/models: per‐model “plugin” modules implementing model-specific weights, inference logic, Triton kernels
- lightllm/distributed: primitives for multi-GPU / multi-node communication (all-reduce, all-gather, NCCL wrappers)
- lightllm/server: various serving interfaces (HTTP, CLI, audio server, visual server, API adapters), plus routing, queuing, detokenization, caching, metrics, health monitoring
- lightllm/utils: utility functions (config, logging, environment, profiling, device/distributed helpers)
- demos: standalone example applications (e.g., chat server)
- docs: user and developer documentation (English/Chinese)
- format_out, test, unit_tests: formatting and correctness tests

##### Main components and interactions
a. Client layer
- External clients (REST/HTTP, WebSocket, CLI, RPC) send inference requests.
- Demo apps (demos/qa_server) or user applications built atop LightLLM act as consumers.
b. Server layer
- HTTP server (lightllm/server/httpserver) or CLI adapter (api_cli) receives requests.
- Config server (config_server) parses start-up and runtime arguments.
- Router (server/router) manages request queueing, batching, dynamic prompt logic, split across modes (prefill, continue).
- Core inference manager (server/core) holds shared objects, locks, pooling, and interfaces with Basemodel.
- Embed cache (server/embed_cache) for repeated embedding requests.
- Detokenization manager (server/detokenization) converts token IDs to text.
- Metrics and health monitor collect performance data.
c. Inference engine
- Basemodel (common/basemodel) loads weights, constructs inference data structures, manages CUDA graph, layer buffers.
- Triton kernels and quantization (common/triton_kernel, common/quantization) implement optimized ops (bmm, attention, dequantize, etc.).
- Per-model logic (lightllm/models/*) implements model-specific weight loading, layer inference, Triton kernels, rope/embedding.
- Distributed operators (lightllm/distributed) coordinate multi-GPU/multi-node execution.
d. External dependencies
- GPUs with CUDA, Triton, PyTorch for host-side orchestration.
- Docker/container orchestration for deployment.
- Optionally, external storage or Petrel for model artifacts.

##### Architectural patterns and design principles
- Plugin architecture for model support: each model under lightllm/models registers itself via registry.py
- Layered separation: common engine vs. model implementations vs. serving front-ends
- Batch/request routing: abstract “router” separates queuing logic from model invocation
- Kernel abstraction over Triton: wrapper modules translate high-level ops to Triton kernels
- Distributed data-parallel and pipeline strategies via custom NCCL/RPC operators

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients
    subgraph "Clients" 
        direction TB
        REST["REST/HTTP Client"]:::client
        WS["WebSocket Client"]:::client
        CLI["CLI Client"]:::client
        RPC["RPC Client"]:::client
    end

    %% Ingress Layer
    subgraph "Server Ingress" 
        direction TB
        HTTPServer["HTTP Server Interface"]:::server
        CLIAdapter["CLI Adapter"]:::server
        AudioServer["Audio Server"]:::server
        VisualServer["Visual Server"]:::server
        ConfigServer["Config Server"]:::server
    end

    REST -->|"/infer"| HTTPServer
    WS -->|"ws/connect"| HTTPServer
    CLI -->|"cli call"| CLIAdapter
    RPC -->|"rpc call"| HTTPServer
    AudioServer -->|"/audio"| Router
    VisualServer -->|"/visual"| Router
    HTTPServer -->|init| ConfigServer
    CLIAdapter -->|init| ConfigServer

    %% Router and Core
    subgraph "Routing & Dispatch" 
        direction TB
        Router["Router & Queue Manager"]:::server
        CoreManager["Core Inference Manager"]:::server
        EmbedCache["Embed Cache"]:::server
        Detokenization["Detokenization Manager"]:::server
    end

    ConfigServer -->|config| Router
    Router -->|batch request| CoreManager
    CoreManager -->|embed lookup| EmbedCache
    CoreManager -->|post-process| Detokenization

    %% Engine Layer
    subgraph "Inference Engine" 
        direction TB
        Basemodel["Basemodel Engine"]:::engine
        TritonKernel["Triton Kernel Wrappers"]:::kernel
        Quantization["Quantization Engine"]:::kernel
        LayerInfer["Layer Inference Templates"]:::engine
        MemoryManagers["Memory Managers"]:::engine
        Distributed["Distributed Comm Primitives"]:::engine
        ModelPlugins["Model Plugins"]:::plugin
    end

    CoreManager -->|invoke| Basemodel
    Basemodel -->|load plugin| ModelPlugins
    Basemodel --> LayerInfer
    LayerInfer --> TritonKernel
    TritonKernel --> Quantization
    LayerInfer --> MemoryManagers
    Basemodel --> Distributed

    %% GPU
    GPU["GPU (CUDA/Triton)"]:::gpu
    TritonKernel -->|kernels| GPU
    Quantization -->|dequant| GPU

    %% Response Flow
    Detokenization -->|text| HTTPServer
    HTTPServer -->|response| REST
    HTTPServer -->|response| WS
    CLIAdapter -->|response| CLI
    HTTPServer -->|response| RPC

    %% Metrics & Health
    subgraph "Monitoring" 
        direction TB
        Metrics["Metrics Collector"]:::monitor
        Health["Health Monitor"]:::monitor
    end
    CoreManager -.->|metrics| Metrics
    Router -.->|metrics| Metrics
    Basemodel -.->|health| Health

    %% Utilities, Demos, Docs, Docker, Tests
    Utils["Utility Functions"]:::infra
    DemoChat["Demo Chat Application"]:::infra
    Docs["Documentation"]:::infra
    Docker["Docker Scripts"]:::infra
    Tests["Tests & Unit Tests"]:::infra

    ConfigServer --> Utils
    DemoChat --> HTTPServer
    Utils --> CoreManager
    Docs -->|read| REST
    Docker -->|deploy| HTTPServer
    Tests --> CoreManager

    %% Click Events
    click REST "https://github.com/modeltc/lightllm/tree/main/external"
    click WS "https://github.com/modeltc/lightllm/tree/main/external"
    click CLI "https://github.com/modeltc/lightllm/tree/main/external"
    click RPC "https://github.com/modeltc/lightllm/tree/main/external"
    click HTTPServer "https://github.com/modeltc/lightllm/tree/main/lightllm/server/httpserver"
    click CLIAdapter "https://github.com/modeltc/lightllm/blob/main/lightllm/server/api_cli.py"
    click AudioServer "https://github.com/modeltc/lightllm/tree/main/lightllm/server/audioserver"
    click VisualServer "https://github.com/modeltc/lightllm/tree/main/lightllm/server/visualserver"
    click ConfigServer "https://github.com/modeltc/lightllm/tree/main/lightllm/server/config_server"
    click Router "https://github.com/modeltc/lightllm/tree/main/lightllm/server/router"
    click CoreManager "https://github.com/modeltc/lightllm/tree/main/lightllm/server/core"
    click EmbedCache "https://github.com/modeltc/lightllm/tree/main/lightllm/server/embed_cache"
    click Detokenization "https://github.com/modeltc/lightllm/tree/main/lightllm/server/detokenization"
    click Metrics "https://github.com/modeltc/lightllm/tree/main/lightllm/server/metrics"
    click Health "https://github.com/modeltc/lightllm/tree/main/lightllm/server/health_monitor"
    click Basemodel "https://github.com/modeltc/lightllm/blob/main/lightllm/common/basemodel/basemodel.py"
    click LayerInfer "https://github.com/modeltc/lightllm/tree/main/lightllm/common/basemodel/layer_infer"
    click TritonKernel "https://github.com/modeltc/lightllm/tree/main/lightllm/common/basemodel/triton_kernel"
    click Quantization "https://github.com/modeltc/lightllm/tree/main/lightllm/common/quantization"
    click MemoryManagers "https://github.com/modeltc/lightllm/tree/main/lightllm/common"
    click Distributed "https://github.com/modeltc/lightllm/tree/main/lightllm/distributed"
    click ModelPlugins "https://github.com/modeltc/lightllm/tree/main/lightllm/models"
    click Utils "https://github.com/modeltc/lightllm/tree/main/lightllm/utils"
    click DemoChat "https://github.com/modeltc/lightllm/blob/main/demos/qa_server/chat_server.py"
    click Docs "https://github.com/modeltc/lightllm/tree/main/docs"
    click Docker "https://github.com/modeltc/lightllm/tree/main/Dockerfile"
    click Tests "https://github.com/modeltc/lightllm/tree/main/test/"

    %% Styles
    classDef client fill:#f2f2f2,stroke:#333,stroke-width:1px;
    classDef server fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef engine fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef plugin fill:#fff3cd,stroke:#856404,stroke-width:1px;
    classDef kernel fill:#e2d6f3,stroke:#4b0082,stroke-width:1px;
    classDef gpu fill:#ffcdd2,stroke:#b71c1c,stroke-width:1px,shape:cylinder;
    classDef monitor fill:#e2e3e5,stroke:#6c757d,stroke-width:1px,stroke-dasharray: 5 5;
    classDef infra fill:#ffe5b4,stroke:#cc8400,stroke-width:1px;
```
