# connlaio/sglang architecture

> Architecture diagram of the GitHub repository connlaio/sglang, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/connlaio/sglang
- Repository: https://github.com/connlaio/sglang

## Overview

SGLang is a high-performance inference and serving framework for language and multimodal models. A typical request enters through the serving interface, is scheduled for model execution, uses runtime attention/cache and kernel facilities, and returns a generated result. The repository also includes multimodal generation and preprocessing, plus separate Rust routing projects. README capabilities inform the high-level runtime branches; the sampled excerpts are partial and span several subsystems, so wiring not established by them is left out.

Main entry points: [`experimental/sgl-router/src/main.rs`](https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/main.rs) (SGL Router), [`python/sglang/srt/model_executor/model_runner_components/ngram_embedding_manager.py`](https://github.com/connlaio/sglang/blob/main/python/sglang/srt/model_executor/model_runner_components/ngram_embedding_manager.py) (Model runner), [`python/sglang/srt`](https://github.com/connlaio/sglang/tree/main/python/sglang/srt) (Runtime APIs), [`python/sglang/multimodal_gen/runtime/pipelines/ltx_2_pipeline.py`](https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines/ltx_2_pipeline.py) (Diffusion pipeline)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Serving runtime"]
  node_cli["Serve CLI<br/>[serve.py]"]
  node_engine["Engine<br/>[engine.py]"]
  node_runtime_api["Runtime APIs"]
  node_generation_result["Inference result"]
end

subgraph group_execution["Inference execution"]
  node_scheduler["Batch scheduler"]
  node_model_runner["Model runner"]
  node_gpu_kernels["GPU kernels"]
  node_kv_cache["KV cache<br/>[pipeline.py]"]
  node_embedding_cache[("Embedding cache")]
end

subgraph group_multimodal["Multimodal workloads"]
  node_multimodal_driver["Media processing<br/>[pipeline.rs]"]
  node_diffusion_pipeline["Diffusion pipeline<br/>[ltx_2_pipeline.py]"]
  node_composed_pipeline["Pipeline composition"]
  node_lora["LoRA handling<br/>[pipeline.py]"]
end

subgraph group_routing["Routing services"]
  node_router_client(("Inference client"))
  node_router["SGL Router<br/>[main.rs]"]
  node_routing_policy["Routing policies<br/>[mod.rs]"]
  node_worker_discovery["Worker discovery<br/>[mod.rs]"]
  node_load_state[("Worker load state")]
  node_grpc_pipeline["gRPC pipeline<br/>[pipeline.rs]"]
end

node_user(("API client"))

node_user -->|"starts serving"| node_cli
node_cli -->|"configures engine"| node_engine
node_user -->|"submits request"| node_runtime_api
node_runtime_api -->|"dispatches request"| node_engine
node_engine -->|"launches schedulers"| node_scheduler
node_scheduler -->|"schedules batches"| node_model_runner
node_model_runner -->|"executes operators"| node_gpu_kernels
node_model_runner -->|"uses KV storage"| node_kv_cache
node_model_runner -.->|"uses embeddings"| node_embedding_cache
node_runtime_api -->|"returns response"| node_generation_result
node_router_client -->|"sends request"| node_router
node_router -->|"discovers workers"| node_worker_discovery
node_router -->|"selects worker"| node_routing_policy
node_router -->|"reads load"| node_load_state
node_router -.->|"forwards gRPC"| node_grpc_pipeline
node_grpc_pipeline -.->|"returns response"| node_generation_result
node_multimodal_driver -.->|"provides tensors"| node_model_runner
node_composed_pipeline -->|"composes stages"| node_diffusion_pipeline
node_diffusion_pipeline -.->|"supports adapters"| node_lora
node_diffusion_pipeline -->|"produces media"| node_generation_result

click node_cli "https://github.com/connlaio/sglang/blob/main/python/sglang/cli/serve.py"
click node_engine "https://github.com/connlaio/sglang/blob/main/python/sglang/srt/entrypoints/engine.py"
click node_runtime_api "https://github.com/connlaio/sglang/tree/main/python/sglang/srt"
click node_scheduler "https://github.com/connlaio/sglang/tree/main/python/sglang/srt"
click node_model_runner "https://github.com/connlaio/sglang/blob/main/python/sglang/srt/model_executor/model_runner_components/ngram_embedding_manager.py"
click node_gpu_kernels "https://github.com/connlaio/sglang/tree/main/python/sglang/kernels"
click node_kv_cache "https://github.com/connlaio/sglang/blob/main/python/sglang/srt/mem_cache/buffer_mode/pipeline.py"
click node_embedding_cache "https://github.com/connlaio/sglang/blob/main/python/sglang/srt/mem_cache/embedding_cache_controller.py"
click node_multimodal_driver "https://github.com/connlaio/sglang/blob/main/rust/sglang-mm/src/pipeline.rs"
click node_diffusion_pipeline "https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines/ltx_2_pipeline.py"
click node_composed_pipeline "https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py"
click node_lora "https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines_core/lora/pipeline.py"
click node_router "https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/main.rs"
click node_routing_policy "https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/policies/mod.rs"
click node_worker_discovery "https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/discovery/mod.rs"
click node_load_state "https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/state/load_monitor/engine_reported_load.rs"
click node_grpc_pipeline "https://github.com/connlaio/sglang/blob/main/sgl-model-gateway/src/routers/grpc/pipeline.rs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_engine,node_runtime_api,node_generation_result,node_user toneBlue
class node_scheduler,node_model_runner,node_gpu_kernels,node_kv_cache,node_embedding_cache toneAmber
class node_multimodal_driver,node_diffusion_pipeline,node_composed_pipeline,node_lora toneMint
class node_router_client,node_router,node_routing_policy,node_worker_discovery,node_load_state,node_grpc_pipeline toneRose
```

## Components

### Serving runtime

- [Serve CLI](https://github.com/connlaio/sglang/blob/main/python/sglang/cli/serve.py): `python/sglang/cli/serve.py`
- [Engine](https://github.com/connlaio/sglang/blob/main/python/sglang/srt/entrypoints/engine.py): `python/sglang/srt/entrypoints/engine.py`
- [Runtime APIs](https://github.com/connlaio/sglang/tree/main/python/sglang/srt): `python/sglang/srt`
- **Inference result**

### Inference execution

- [Batch scheduler](https://github.com/connlaio/sglang/tree/main/python/sglang/srt): `python/sglang/srt`
- [Model runner](https://github.com/connlaio/sglang/blob/main/python/sglang/srt/model_executor/model_runner_components/ngram_embedding_manager.py): `python/sglang/srt/model_executor/model_runner_components/ngram_embedding_manager.py`
- [GPU kernels](https://github.com/connlaio/sglang/tree/main/python/sglang/kernels): `python/sglang/kernels`
- [KV cache](https://github.com/connlaio/sglang/blob/main/python/sglang/srt/mem_cache/buffer_mode/pipeline.py): `python/sglang/srt/mem_cache/buffer_mode/pipeline.py`
- [Embedding cache](https://github.com/connlaio/sglang/blob/main/python/sglang/srt/mem_cache/embedding_cache_controller.py): `python/sglang/srt/mem_cache/embedding_cache_controller.py`

### Multimodal workloads

- [Media processing](https://github.com/connlaio/sglang/blob/main/rust/sglang-mm/src/pipeline.rs): `rust/sglang-mm/src/pipeline.rs`
- [Diffusion pipeline](https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines/ltx_2_pipeline.py): `python/sglang/multimodal_gen/runtime/pipelines/ltx_2_pipeline.py`
- [Pipeline composition](https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py): `python/sglang/multimodal_gen/runtime/pipelines_core/composed_pipeline_base.py`
- [LoRA handling](https://github.com/connlaio/sglang/blob/main/python/sglang/multimodal_gen/runtime/pipelines_core/lora/pipeline.py): `python/sglang/multimodal_gen/runtime/pipelines_core/lora/pipeline.py`

### Routing services

- **Inference client**
- [SGL Router](https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/main.rs): `experimental/sgl-router/src/main.rs`
- [Routing policies](https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/policies/mod.rs): `experimental/sgl-router/src/policies/mod.rs`
- [Worker discovery](https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/discovery/mod.rs): `experimental/sgl-router/src/discovery/mod.rs`
- [Worker load state](https://github.com/connlaio/sglang/blob/main/experimental/sgl-router/src/state/load_monitor/engine_reported_load.rs): `experimental/sgl-router/src/state/load_monitor/engine_reported_load.rs`
- [gRPC pipeline](https://github.com/connlaio/sglang/blob/main/sgl-model-gateway/src/routers/grpc/pipeline.rs): `sgl-model-gateway/src/routers/grpc/pipeline.rs`

### Other components

- **API client**

## Connections

- API client → Serve CLI: starts serving
- Serve CLI → Engine: configures engine
- API client → Runtime APIs: submits request
- Runtime APIs → Engine: dispatches request
- Engine → Batch scheduler: launches schedulers
- Batch scheduler → Model runner: schedules batches
- Model runner → GPU kernels: executes operators
- Model runner → KV cache: uses KV storage
- Model runner → Embedding cache: uses embeddings
- Runtime APIs → Inference result: returns response
- Inference client → SGL Router: sends request
- SGL Router → Worker discovery: discovers workers
- SGL Router → Routing policies: selects worker
- SGL Router → Worker load state: reads load
- SGL Router → gRPC pipeline: forwards gRPC
- gRPC pipeline → Inference result: returns response
- Media processing → Model runner: provides tensors
- Pipeline composition → Diffusion pipeline: composes stages
- Diffusion pipeline → LoRA handling: supports adapters
- Diffusion pipeline → Inference result: produces media