# uiuc-mlsys/jitserve architecture

> Architecture diagram of the GitHub repository uiuc-mlsys/jitserve, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uiuc-mlsys/jitserve
- Repository: https://github.com/uiuc-mlsys/jitserve

## Overview

JITServe is an SLO-aware LLM serving research system: clients submit requests with request types and SLO constraints, the server prepares request metadata and optional predictions, and the vendored vLLM engine serves them under JITServe’s scheduling integration. Collective workloads add staged graph analysis for predicting stage ratios. The README also identifies benchmark clients and trace-generation tools as reproducible-evaluation capabilities. Scheduler wiring and some server-to-analysis calls are not fully shown in the sampled excerpts, so uncertain connections are omitted.

Main entry points: [`jitserve/server.py`](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/server.py) (API Server), [`jitserve/request_analyzer/prediction.py`](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/prediction.py) (Length Predictor), [`third_party/vllm`](https://github.com/uiuc-mlsys/jitserve/tree/main/third_party/vllm) (vLLM Engine), [`jitserve/scheduler/slo_scheduler.py`](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/scheduler/slo_scheduler.py) (SLO Scheduler)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_serving["Request Serving"]
  node_server["API Server<br/>[server.py]"]
  node_requestinfo["Request Metadata<br/>[request_info.py]"]
  node_engine["vLLM Engine"]
end

subgraph group_analysis["Request Analysis"]
  node_predictor["Length Predictor<br/>[prediction.py]"]
  node_graphcontext["Graph Context<br/>[graph_context.py]"]
  node_trace_reader["Trace Reader"]
  node_similarity["Graph Similarity<br/>[similarity.py]"]
  node_predictive_assets["QRF Assets"]
end

subgraph group_scheduling["SLO Scheduling"]
  node_scheduler["SLO Scheduler<br/>[slo_scheduler.py]"]
  node_policy["Scheduling Policies<br/>[policy.py]"]
  node_tracker["SLO Tracker<br/>[slo_tracker.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_benchmark_client["Benchmark Clients<br/>[client.py]"]
  node_trace_tools["Trace Tools"]
  node_traces["Trace Data"]
end

node_client(("Request Client"))

node_client -->|"submits requests"| node_server
node_server -->|"builds metadata"| node_requestinfo
node_server -->|"serves generation"| node_engine
node_server -.->|"requests prediction"| node_predictor
node_predictor -->|"updates output estimate"| node_requestinfo
node_predictive_assets -.->|"loads models"| node_predictor
node_graphcontext -->|"matches graphs"| node_similarity
node_trace_reader -->|"creates graphs"| node_similarity
node_scheduler -->|"gets priorities"| node_policy
node_engine -->|"uses scheduler integration"| node_scheduler
node_benchmark_client -->|"sends workloads"| node_server
node_trace_tools -->|"constructs traces"| node_traces

click node_server "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/server.py"
click node_requestinfo "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_info.py"
click node_engine "https://github.com/uiuc-mlsys/jitserve/tree/main/third_party/vllm"
click node_predictor "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/prediction.py"
click node_graphcontext "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/graph_context.py"
click node_trace_reader "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/deepresearch_trace_reader.py"
click node_similarity "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/similarity.py"
click node_scheduler "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/scheduler/slo_scheduler.py"
click node_policy "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/scheduler/policy.py"
click node_tracker "https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/slo_tracker/slo_tracker.py"
click node_predictive_assets "https://github.com/uiuc-mlsys/jitserve/tree/main/assets"
click node_benchmark_client "https://github.com/uiuc-mlsys/jitserve/blob/main/benchmark/trace/client.py"
click node_trace_tools "https://github.com/uiuc-mlsys/jitserve/tree/main/benchmark/trace/tools"
click node_traces "https://github.com/uiuc-mlsys/jitserve/tree/main/traces"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_server,node_requestinfo,node_engine,node_client toneBlue
class node_predictor,node_graphcontext,node_trace_reader,node_similarity,node_predictive_assets toneAmber
class node_scheduler,node_policy,node_tracker toneMint
class node_benchmark_client,node_trace_tools,node_traces toneRose
```

## Components

### Request Serving

- [API Server](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/server.py): `jitserve/server.py`
- [Request Metadata](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_info.py): `jitserve/request_info.py`
- [vLLM Engine](https://github.com/uiuc-mlsys/jitserve/tree/main/third_party/vllm): `third_party/vllm`

### Request Analysis

- [Length Predictor](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/prediction.py): `jitserve/request_analyzer/prediction.py`
- [Graph Context](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/graph_context.py): `jitserve/request_analyzer/graph_context.py`
- [Trace Reader](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/deepresearch_trace_reader.py): `jitserve/request_analyzer/deepresearch_trace_reader.py`
- [Graph Similarity](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/request_analyzer/similarity.py): `jitserve/request_analyzer/similarity.py`
- [QRF Assets](https://github.com/uiuc-mlsys/jitserve/tree/main/assets): `assets`

### SLO Scheduling

- [SLO Scheduler](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/scheduler/slo_scheduler.py): `jitserve/scheduler/slo_scheduler.py`
- [Scheduling Policies](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/scheduler/policy.py): `jitserve/scheduler/policy.py`
- [SLO Tracker](https://github.com/uiuc-mlsys/jitserve/blob/main/jitserve/slo_tracker/slo_tracker.py): `jitserve/slo_tracker/slo_tracker.py`

### Evaluation

- [Benchmark Clients](https://github.com/uiuc-mlsys/jitserve/blob/main/benchmark/trace/client.py): `benchmark/trace/client.py`
- [Trace Tools](https://github.com/uiuc-mlsys/jitserve/tree/main/benchmark/trace/tools): `benchmark/trace/tools`
- [Trace Data](https://github.com/uiuc-mlsys/jitserve/tree/main/traces): `traces`

### Other components

- **Request Client**

## Connections

- Request Client → API Server: submits requests
- API Server → Request Metadata: builds metadata
- API Server → vLLM Engine: serves generation
- API Server → Length Predictor: requests prediction
- Length Predictor → Request Metadata: updates output estimate
- QRF Assets → Length Predictor: loads models
- Graph Context → Graph Similarity: matches graphs
- Trace Reader → Graph Similarity: creates graphs
- SLO Scheduler → Scheduling Policies: gets priorities
- vLLM Engine → SLO Scheduler: uses scheduler integration
- Benchmark Clients → API Server: sends workloads
- Trace Tools → Trace Data: constructs traces