How modeltc/lightllm works
Project type and purpose
• LightLLM is a full-stack, Python-based LLM inference and serving framework.
• It provides a lightweight, high-performance engine for loading myriad open-source model implementations (e.g., Llama, InternLM, DeepSeek2), running inference on GPU (via Triton kernels, FlashAttention, quantization), and serving results over HTTP/WebSocket/RPC.
High-level component analysis
From the file tree and README, the key top-level directories are:
• lightllm/common: core abstractions (Basemodel, memory managers, CUDA wrappers, kernel configs, quantization)
• lightllm/models: per‐model “plugin” modules implementing model-specific weights, inference logic, Triton kernels
• lightllm/distributed: primitives for multi-GPU / multi-node communication (all-reduce, all-gather, NCCL wrappers)
• lightllm/server: various serving interfaces (HTTP, CLI, audio server, visual server, API adapters), plus routing, queuing, detokenization, caching, metrics, health monitoring
• lightllm/utils: utility functions (config, logging, environment, profiling, device/distributed helpers)
• demos: standalone example applications (e.g., chat server)
• docs: user and developer documentation (English/Chinese)
• format_out, test, unit_tests: formatting and correctness tests
Main components and interactions
a. Client layer
• External clients (REST/HTTP, WebSocket, CLI, RPC) send inference requests.
• Demo apps (demos/qa_server) or user applications built atop LightLLM act as consumers.
b. Server layer
• HTTP server (lightllm/server/httpserver) or CLI adapter (api_cli) receives requests.
• Config server (config_server) parses start-up and runtime arguments.
• Router (server/router) manages request queueing, batching, dynamic prompt logic, split across modes (prefill, continue).
• Core inference manager (server/core) holds shared objects, locks, pooling, and interfaces with Basemodel.
• Embed cache (server/embed_cache) for repeated embedding requests.
• Detokenization manager (server/detokenization) converts token IDs to text.
• Metrics and health monitor collect performance data.
c. Inference engine
• Basemodel (common/basemodel) loads weights, constructs inference data structures, manages CUDA graph, layer buffers.
• Triton kernels and quantization (common/triton_kernel, common/quantization) implement optimized ops (bmm, attention, dequantize, etc.).
• Per-model logic (lightllm/models/*) implements model-specific weight loading, layer inference, Triton kernels, rope/embedding.
• Distributed operators (lightllm/distributed) coordinate multi-GPU/multi-node execution.
d. External dependencies
• GPUs with CUDA, Triton, PyTorch for host-side orchestration.
• Docker/container orchestration for deployment.
• Optionally, external storage or Petrel for model artifacts.
Architectural patterns and design principles
• Plugin architecture for model support: each model under lightllm/models registers itself via registry.py
• Layered separation: common engine vs. model implementations vs. serving front-ends
• Batch/request routing: abstract “router” separates queuing logic from model invocation
• Kernel abstraction over Triton: wrapper modules translate high-level ops to Triton kernels
• Distributed data-parallel and pipeline strategies via custom NCCL/RPC operators