How katelynnllu/preble works
Preble is a distributed LLM-serving load balancer focused on prefix-cache-aware request placement. Its principal workflow accepts generation requests at the Preble server, selects a runtime using request/cache state, and forwards work to an SGLang or other runtime; it also supports dynamic model loading and simulation. SGLang provides the serving engine beneath that layer, while its separate SGLang language package builds and interprets structured programs against backends. The graph reflects documented and indexed relationships; several runtime details are represented at subsystem level where source was not excerpted.
Main entry points: model_runtime_manager.py Runtime manager · model_rpc.py Model RPC · server/server.py Request server · srt/server.py SGLang API server
Components 22
Preble orchestration
- Request server
preble/server/server.py - Runtime manager
preble/model_runtime_manager.py - Request router
preble/data_parallel_request_cache.py - Global LRU cache
preble/global_lru_cache.py - Time scheduler
preble/global_scheduler_with_time.py - Runtime simulator
preble/simulator.py
SGLang serving
- SGLang API server
python/sglang/srt/server.py - Tokenizer manager
python/sglang/srt/managers/tokenizer_manager.py - Request structures
python/sglang/srt/managers/io_struct.py - Router manager
python/sglang/srt/managers/router/manager.py - Model RPC
python/sglang/srt/managers/router/model_rpc.py - Batch scheduler
python/sglang/srt/managers/router/scheduler.py - Inference batches
python/sglang/srt/managers/router/infer_batch.py - Model runner
python/sglang/srt/managers/router/model_runner.py - Radix cache
python/sglang/srt/managers/router/radix_cache.py - Model families
python/sglang/srt/models
SGLang programming
- SGLang API
python/sglang/api.py - Program IR
python/sglang/lang/ir.py - Interpreter
python/sglang/lang/interpreter.py - Model backends
python/sglang/backend
Other components
- Client
- Model runtime
Connections 22
- Client → Request server: sends requests
README.md - Request server → Runtime manager: dispatches requests
preble/server/server.py - Runtime manager → Request router: selects runtime
preble/model_runtime_manager.py - Runtime manager → Model runtime: forwards requests
README.md - Runtime manager → Runtime simulator: runs simulation
preble/model_runtime_manager.py - Request server → Time scheduler: uses scheduling
preble/server/server.py - Time scheduler → Global LRU cache: consults cache
preble/global_scheduler_with_time.py - Model runtime → SGLang API server: serves generation
- SGLang API server → Tokenizer manager: routes requests
python/sglang/srt/server.py - SGLang API server → Router manager: starts router
python/sglang/srt/server.py - Tokenizer manager → Request structures: builds requests
python/sglang/srt/managers/tokenizer_manager.py - Router manager → Model RPC: steps model client
python/sglang/srt/managers/router/manager.py - Model RPC → Batch scheduler: schedules batches
python/sglang/srt/managers/router/model_rpc.py - Model RPC → Inference batches: forms batches
python/sglang/srt/managers/router/model_rpc.py - Model RPC → Model runner: runs inference
python/sglang/srt/managers/router/model_rpc.py - Inference batches → Radix cache: uses prefix cache
python/sglang/srt/managers/router/infer_batch.py - Model runner → Model families: loads models
python/sglang/srt/models/llava_mistral.py - Client → SGLang API: builds programs
python/sglang/api.py - SGLang API → Program IR: constructs expressions
python/sglang/api.py - SGLang API → Model backends: selects backend
python/sglang/api.py - Model backends → Interpreter: executes programs
python/sglang/backend/base_backend.py - Interpreter → Program IR: interprets expressions
python/sglang/lang/interpreter.py