# katelynnllu/preble architecture

> Architecture diagram of the GitHub repository katelynnllu/preble, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/katelynnllu/preble
- Repository: https://github.com/katelynnllu/preble

## Overview

Preble is a distributed LLM-serving load balancer focused on prefix-cache-aware request placement. Its principal workflow accepts generation requests at the Preble server, selects a runtime using request/cache state, and forwards work to an SGLang or other runtime; it also supports dynamic model loading and simulation. SGLang provides the serving engine beneath that layer, while its separate SGLang language package builds and interprets structured programs against backends. The graph reflects documented and indexed relationships; several runtime details are represented at subsystem level where source was not excerpted.

Main entry points: [`preble/model_runtime_manager.py`](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/model_runtime_manager.py) (Runtime manager), [`python/sglang/srt/managers/router/model_rpc.py`](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/model_rpc.py) (Model RPC), [`preble/server/server.py`](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/server/server.py) (Request server), [`python/sglang/srt/server.py`](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/server.py) (SGLang API server)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_preble["Preble orchestration"]
  node_preble_server["Request server<br/>[server.py]"]
  node_runtime_manager["Runtime manager"]
  node_request_cache["Request router"]
  node_prefix_cache[("Global LRU cache")]
  node_scheduler["Time scheduler"]
  node_simulation["Runtime simulator<br/>[simulator.py]"]
end

subgraph group_serving["SGLang serving"]
  node_sg_server["SGLang API server<br/>[server.py]"]
  node_tokenizer["Tokenizer manager"]
  node_request_types["Request structures<br/>[io_struct.py]"]
  node_router_manager["Router manager<br/>[manager.py]"]
  node_model_rpc["Model RPC<br/>[model_rpc.py]"]
  node_scheduler_core["Batch scheduler<br/>[scheduler.py]"]
  node_inference_batch["Inference batches<br/>[infer_batch.py]"]
  node_model_runner["Model runner<br/>[model_runner.py]"]
  node_radix_cache[("Radix cache<br/>[radix_cache.py]")]
  node_model_families["Model families"]
end

subgraph group_language["SGLang programming"]
  node_sg_api["SGLang API<br/>[api.py]"]
  node_program_ir["Program IR<br/>[ir.py]"]
  node_interpreter["Interpreter<br/>[interpreter.py]"]
  node_backend["Model backends"]
end

node_client(("Client"))
node_runtime_service["Model runtime"]

node_client -->|"sends requests"| node_preble_server
node_preble_server -->|"dispatches requests"| node_runtime_manager
node_runtime_manager -->|"selects runtime"| node_request_cache
node_runtime_manager -->|"forwards requests"| node_runtime_service
node_runtime_manager -->|"runs simulation"| node_simulation
node_preble_server -->|"uses scheduling"| node_scheduler
node_scheduler -->|"consults cache"| node_prefix_cache
node_runtime_service -.->|"serves generation"| node_sg_server
node_sg_server -->|"routes requests"| node_tokenizer
node_sg_server -->|"starts router"| node_router_manager
node_tokenizer -->|"builds requests"| node_request_types
node_router_manager -->|"steps model client"| node_model_rpc
node_model_rpc -->|"schedules batches"| node_scheduler_core
node_model_rpc -->|"forms batches"| node_inference_batch
node_model_rpc -->|"runs inference"| node_model_runner
node_inference_batch -->|"uses prefix cache"| node_radix_cache
node_model_runner -.->|"loads models"| node_model_families
node_client -->|"builds programs"| node_sg_api
node_sg_api -->|"constructs expressions"| node_program_ir
node_sg_api -->|"selects backend"| node_backend
node_backend -->|"executes programs"| node_interpreter
node_interpreter -->|"interprets expressions"| node_program_ir

click node_preble_server "https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/server/server.py"
click node_runtime_manager "https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/model_runtime_manager.py"
click node_request_cache "https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/data_parallel_request_cache.py"
click node_prefix_cache "https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/global_lru_cache.py"
click node_scheduler "https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/global_scheduler_with_time.py"
click node_simulation "https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/simulator.py"
click node_sg_server "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/server.py"
click node_tokenizer "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/tokenizer_manager.py"
click node_request_types "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/io_struct.py"
click node_router_manager "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/manager.py"
click node_model_rpc "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/model_rpc.py"
click node_scheduler_core "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/scheduler.py"
click node_inference_batch "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/infer_batch.py"
click node_model_runner "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/model_runner.py"
click node_radix_cache "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/radix_cache.py"
click node_model_families "https://github.com/katelynnllu/preble/tree/multi_modal_main/python/sglang/srt/models"
click node_sg_api "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/api.py"
click node_program_ir "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/lang/ir.py"
click node_interpreter "https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/lang/interpreter.py"
click node_backend "https://github.com/katelynnllu/preble/tree/multi_modal_main/python/sglang/backend"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_preble_server,node_runtime_manager,node_request_cache,node_prefix_cache,node_scheduler,node_simulation,node_client toneBlue
class node_sg_server,node_tokenizer,node_request_types,node_router_manager,node_model_rpc,node_scheduler_core,node_inference_batch,node_model_runner,node_radix_cache,node_model_families toneAmber
class node_sg_api,node_program_ir,node_interpreter,node_backend toneMint
class node_runtime_service toneIndigo
```

## Components

### Preble orchestration

- [Request server](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/server/server.py): `preble/server/server.py`
- [Runtime manager](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/model_runtime_manager.py): `preble/model_runtime_manager.py`
- [Request router](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/data_parallel_request_cache.py): `preble/data_parallel_request_cache.py`
- [Global LRU cache](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/global_lru_cache.py): `preble/global_lru_cache.py`
- [Time scheduler](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/global_scheduler_with_time.py): `preble/global_scheduler_with_time.py`
- [Runtime simulator](https://github.com/katelynnllu/preble/blob/multi_modal_main/preble/simulator.py): `preble/simulator.py`

### SGLang serving

- [SGLang API server](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/server.py): `python/sglang/srt/server.py`
- [Tokenizer manager](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/tokenizer_manager.py): `python/sglang/srt/managers/tokenizer_manager.py`
- [Request structures](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/io_struct.py): `python/sglang/srt/managers/io_struct.py`
- [Router manager](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/manager.py): `python/sglang/srt/managers/router/manager.py`
- [Model RPC](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/model_rpc.py): `python/sglang/srt/managers/router/model_rpc.py`
- [Batch scheduler](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/scheduler.py): `python/sglang/srt/managers/router/scheduler.py`
- [Inference batches](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/infer_batch.py): `python/sglang/srt/managers/router/infer_batch.py`
- [Model runner](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/model_runner.py): `python/sglang/srt/managers/router/model_runner.py`
- [Radix cache](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/srt/managers/router/radix_cache.py): `python/sglang/srt/managers/router/radix_cache.py`
- [Model families](https://github.com/katelynnllu/preble/tree/multi_modal_main/python/sglang/srt/models): `python/sglang/srt/models`

### SGLang programming

- [SGLang API](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/api.py): `python/sglang/api.py`
- [Program IR](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/lang/ir.py): `python/sglang/lang/ir.py`
- [Interpreter](https://github.com/katelynnllu/preble/blob/multi_modal_main/python/sglang/lang/interpreter.py): `python/sglang/lang/interpreter.py`
- [Model backends](https://github.com/katelynnllu/preble/tree/multi_modal_main/python/sglang/backend): `python/sglang/backend`

### Other components

- **Client**
- **Model runtime**

## Connections

- Client → Request server: sends requests (evidence: `README.md`)
- Request server → Runtime manager: dispatches requests (evidence: `preble/server/server.py`)
- Runtime manager → Request router: selects runtime (evidence: `preble/model_runtime_manager.py`)
- Runtime manager → Model runtime: forwards requests (evidence: `README.md`)
- Runtime manager → Runtime simulator: runs simulation (evidence: `preble/model_runtime_manager.py`)
- Request server → Time scheduler: uses scheduling (evidence: `preble/server/server.py`)
- Time scheduler → Global LRU cache: consults cache (evidence: `preble/global_scheduler_with_time.py`)
- Model runtime → SGLang API server: serves generation
- SGLang API server → Tokenizer manager: routes requests (evidence: `python/sglang/srt/server.py`)
- SGLang API server → Router manager: starts router (evidence: `python/sglang/srt/server.py`)
- Tokenizer manager → Request structures: builds requests (evidence: `python/sglang/srt/managers/tokenizer_manager.py`)
- Router manager → Model RPC: steps model client (evidence: `python/sglang/srt/managers/router/manager.py`)
- Model RPC → Batch scheduler: schedules batches (evidence: `python/sglang/srt/managers/router/model_rpc.py`)
- Model RPC → Inference batches: forms batches (evidence: `python/sglang/srt/managers/router/model_rpc.py`)
- Model RPC → Model runner: runs inference (evidence: `python/sglang/srt/managers/router/model_rpc.py`)
- Inference batches → Radix cache: uses prefix cache (evidence: `python/sglang/srt/managers/router/infer_batch.py`)
- Model runner → Model families: loads models (evidence: `python/sglang/srt/models/llava_mistral.py`)
- Client → SGLang API: builds programs (evidence: `python/sglang/api.py`)
- SGLang API → Program IR: constructs expressions (evidence: `python/sglang/api.py`)
- SGLang API → Model backends: selects backend (evidence: `python/sglang/api.py`)
- Model backends → Interpreter: executes programs (evidence: `python/sglang/backend/base_backend.py`)
- Interpreter → Program IR: interprets expressions (evidence: `python/sglang/lang/interpreter.py`)