How uiuc-mlsys/jitserve works
JITServe is an SLO-aware LLM serving research system: clients submit requests with request types and SLO constraints, the server prepares request metadata and optional predictions, and the vendored vLLM engine serves them under JITServe’s scheduling integration. Collective workloads add staged graph analysis for predicting stage ratios. The README also identifies benchmark clients and trace-generation tools as reproducible-evaluation capabilities. Scheduler wiring and some server-to-analysis calls are not fully shown in the sampled excerpts, so uncertain connections are omitted.
Main entry points: server.py API Server · prediction.py Length Predictor · vllm vLLM Engine · slo_scheduler.py SLO Scheduler
Components 15
Request Serving
- API Server
jitserve/server.py - Request Metadata
jitserve/request_info.py - vLLM Engine
third_party/vllm
Request Analysis
- Length Predictor
jitserve/request_analyzer/prediction.py - Graph Context
jitserve/request_analyzer/graph_context.py - Trace Reader
jitserve/request_analyzer/deepresearch_trace_reader.py - Graph Similarity
jitserve/request_analyzer/similarity.py - QRF Assets
assets
SLO Scheduling
- SLO Scheduler
jitserve/scheduler/slo_scheduler.py - Scheduling Policies
jitserve/scheduler/policy.py - SLO Tracker
jitserve/slo_tracker/slo_tracker.py
Evaluation
- Benchmark Clients
benchmark/trace/client.py - Trace Tools
benchmark/trace/tools - Trace Data
traces
Other components
- Request Client
Connections 12
- Request Client → API Server: submits requests
- API Server → Request Metadata: builds metadata
- API Server → vLLM Engine: serves generation
- API Server → Length Predictor: requests prediction
- Length Predictor → Request Metadata: updates output estimate
- QRF Assets → Length Predictor: loads models
- Graph Context → Graph Similarity: matches graphs
- Trace Reader → Graph Similarity: creates graphs
- SLO Scheduler → Scheduling Policies: gets priorities
- vLLM Engine → SLO Scheduler: uses scheduler integration
- Benchmark Clients → API Server: sends workloads
- Trace Tools → Trace Data: constructs traces