How connlaio/sglang works
SGLang is a high-performance inference and serving framework for language and multimodal models. A typical request enters through the serving interface, is scheduled for model execution, uses runtime attention/cache and kernel facilities, and returns a generated result. The repository also includes multimodal generation and preprocessing, plus separate Rust routing projects. README capabilities inform the high-level runtime branches; the sampled excerpts are partial and span several subsystems, so wiring not established by them is left out.
Main entry points: main.rs SGL Router · ngram_embedding_manager.py Model runner · srt Runtime APIs · ltx_2_pipeline.py Diffusion pipeline
Components 20
Serving runtime
- Serve CLI
python/sglang/cli/serve.py - Engine
python/sglang/srt/entrypoints/engine.py - Runtime APIs
python/sglang/srt - Inference result
Inference execution
- Batch scheduler
python/sglang/srt - Model runner
python/sglang/srt/model_executor/model_runner_components/ngram_embedding_manager.py - GPU kernels
python/sglang/kernels - KV cache
python/sglang/srt/mem_cache/buffer_mode/pipeline.py - Embedding cache
python/sglang/srt/mem_cache/embedding_cache_controller.py
Multimodal workloads
- Media processing
rust/sglang-mm/src/pipeline.rs - Diffusion pipeline
python/sglang/multimodal_gen/runtime/pipelines/ltx_2_pipeline.py - Pipeline composition
python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py - LoRA handling
python/sglang/multimodal_gen/runtime/pipelines_core/lora/pipeline.py
Routing services
- Inference client
- SGL Router
experimental/sgl-router/src/main.rs - Routing policies
experimental/sgl-router/src/policies/mod.rs - Worker discovery
experimental/sgl-router/src/discovery/mod.rs - Worker load state
experimental/sgl-router/src/state/load_monitor/engine_reported_load.rs - gRPC pipeline
sgl-model-gateway/src/routers/grpc/pipeline.rs
Other components
- API client
Connections 20
- API client → Serve CLI: starts serving
- Serve CLI → Engine: configures engine
- API client → Runtime APIs: submits request
- Runtime APIs → Engine: dispatches request
- Engine → Batch scheduler: launches schedulers
- Batch scheduler → Model runner: schedules batches
- Model runner → GPU kernels: executes operators
- Model runner → KV cache: uses KV storage
- Model runner → Embedding cache: uses embeddings
- Runtime APIs → Inference result: returns response
- Inference client → SGL Router: sends request
- SGL Router → Worker discovery: discovers workers
- SGL Router → Routing policies: selects worker
- SGL Router → Worker load state: reads load
- SGL Router → gRPC pipeline: forwards gRPC
- gRPC pipeline → Inference result: returns response
- Media processing → Model runner: provides tensors
- Pipeline composition → Diffusion pipeline: composes stages
- Diffusion pipeline → LoRA handling: supports adapters
- Diffusion pipeline → Inference result: produces media