# casys-kaist/llmservingsim architecture

> Architecture diagram of the GitHub repository casys-kaist/llmservingsim, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/casys-kaist/llmservingsim
- Repository: https://github.com/casys-kaist/llmservingsim

## Overview

LLMServingSim combines a vLLM-based latency profiler with a cycle-level serving simulator and analytical network backend. The main workflow is to acquire per-hardware model timings, then use those profiles and configured workloads to simulate request scheduling, model execution, memory use, and communication, producing serving traces/results. The simulator also models tiered KV memory, MoE, speculative decoding, power, and TP/PP/EP/DP. The map follows documented flow and sampled entry-point references; several subsystem internals are unsampled, so their internal links are omitted.

Main entry points: [`serving/__main__.py`](https://github.com/casys-kaist/llmservingsim/blob/main/serving/__main__.py) (Simulator entry), [`serving/core/trace_generator.py`](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/trace_generator.py) (Trace generation), [`profiler/__main__.py`](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/__main__.py) (Profiler CLI), [`profiler/core/hooks/activity_collector.py`](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/activity_collector.py) (Runtime capture)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_profiling["Latency profiling"]
  node_profiler_cli["Profiler CLI<br/>[__main__.py]"]
  node_vllm_engine["vLLM engine<br/>[engine.py]"]
  node_capture["Runtime capture"]
  node_timings["Timing records<br/>[timings.py]"]
  node_context["Capture contexts"]
  node_component_layouts["Component layouts"]
  node_profile_bundle[("Profile bundle<br/>[bundle_metadata.py]")]
end

subgraph group_serving["Serving simulation"]
  node_sim_cli["Simulator entry<br/>[__main__.py]"]
  node_router["Request router<br/>[router.py]"]
  node_scheduler["Batch scheduler<br/>[scheduler.py]"]
  node_trace_generator["Trace generation<br/>[trace_generator.py]"]
  node_graph["Execution graph<br/>[graph_generator.py]"]
end

subgraph group_modeling["Hardware and model"]
  node_memory["Memory model<br/>[memory_model.py]"]
  node_kv_manager["Tiered KV cache"]
  node_block_pool[("Block pools<br/>[block_pool.py]")]
  node_communication["Communication model<br/>[communication.py]"]
  node_moe["MoE execution<br/>[moe_execution.py]"]
  node_spec_decode["Speculative decoding<br/>[spec_decode.py]"]
  node_power["Power model<br/>[power_model.py]"]
end

subgraph group_inputs["Workload inputs"]
  node_workload_cli["Workload generator CLI<br/>[__main__.py]"]
  node_sharegpt["ShareGPT generator<br/>[sharegpt.py]"]
end

node_user(("Operator"))
node_astra{{"ASTRA-Sim"}}
node_workload["Request workload"]

node_user -->|"profiles models"| node_profiler_cli
node_profiler_cli -->|"starts profiling"| node_vllm_engine
node_vllm_engine -.->|"runs capture"| node_capture
node_capture -.->|"records timings"| node_timings
node_capture -->|"uses contexts"| node_context
node_profiler_cli -->|"uses layouts"| node_component_layouts
node_profiler_cli -.->|"publishes profiles"| node_profile_bundle
node_user -->|"generates workload"| node_workload_cli
node_workload_cli -->|"calls generator"| node_sharegpt
node_sharegpt -.->|"emits requests"| node_workload
node_user -->|"runs simulation"| node_sim_cli
node_workload -.->|"supplies requests"| node_sim_cli
node_profile_bundle -->|"provides timings"| node_sim_cli
node_sim_cli -->|"routes requests"| node_router
node_sim_cli -->|"schedules iterations"| node_scheduler
node_sim_cli -->|"generates traces"| node_trace_generator
node_sim_cli -->|"builds graphs"| node_graph
node_sim_cli -->|"runs backend"| node_astra
node_trace_generator -->|"uses memory model"| node_memory
node_memory -.->|"manages KV cache"| node_kv_manager
node_kv_manager -->|"allocates blocks"| node_block_pool
node_trace_generator -->|"models communication"| node_communication
node_trace_generator -->|"models MoE"| node_moe
node_trace_generator -->|"models power"| node_power
node_sim_cli -->|"configures decoding"| node_spec_decode

click node_profiler_cli "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/__main__.py"
click node_vllm_engine "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/engine.py"
click node_capture "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/activity_collector.py"
click node_timings "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/timings.py"
click node_context "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/callable_context.py"
click node_component_layouts "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/component_layouts.py"
click node_profile_bundle "https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/bundle_metadata.py"
click node_sim_cli "https://github.com/casys-kaist/llmservingsim/blob/main/serving/__main__.py"
click node_router "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/router.py"
click node_scheduler "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/scheduler.py"
click node_trace_generator "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/trace_generator.py"
click node_graph "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/graph_generator.py"
click node_memory "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/memory_model.py"
click node_kv_manager "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/kv_cache_manager.py"
click node_block_pool "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/block_pool.py"
click node_communication "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/communication.py"
click node_moe "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/moe_execution.py"
click node_spec_decode "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/spec_decode.py"
click node_power "https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/power_model.py"
click node_workload_cli "https://github.com/casys-kaist/llmservingsim/blob/main/workloads/generators/__main__.py"
click node_sharegpt "https://github.com/casys-kaist/llmservingsim/blob/main/workloads/generators/sharegpt.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_profiler_cli,node_vllm_engine,node_capture,node_timings,node_context,node_component_layouts,node_profile_bundle toneBlue
class node_sim_cli,node_router,node_scheduler,node_trace_generator,node_graph toneAmber
class node_memory,node_kv_manager,node_block_pool,node_communication,node_moe,node_spec_decode,node_power,node_workload toneMint
class node_workload_cli,node_sharegpt toneRose
class node_user,node_astra toneIndigo
```

## Components

### Latency profiling

- [Profiler CLI](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/__main__.py): `profiler/__main__.py`
- [vLLM engine](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/engine.py): `profiler/core/engine.py`
- [Runtime capture](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/activity_collector.py): `profiler/core/hooks/activity_collector.py`
- [Timing records](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/timings.py): `profiler/core/hooks/timings.py`
- [Capture contexts](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/hooks/callable_context.py): `profiler/core/hooks/callable_context.py`
- [Component layouts](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/component_layouts.py): `profiler/core/component_layouts.py`
- [Profile bundle](https://github.com/casys-kaist/llmservingsim/blob/main/profiler/core/bundle_metadata.py): `profiler/core/bundle_metadata.py`

### Serving simulation

- [Simulator entry](https://github.com/casys-kaist/llmservingsim/blob/main/serving/__main__.py): `serving/__main__.py`
- [Request router](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/router.py): `serving/core/router.py`
- [Batch scheduler](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/scheduler.py): `serving/core/scheduler.py`
- [Trace generation](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/trace_generator.py): `serving/core/trace_generator.py`
- [Execution graph](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/graph_generator.py): `serving/core/graph_generator.py`

### Hardware and model

- [Memory model](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/memory_model.py): `serving/core/memory_model.py`
- [Tiered KV cache](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/kv_cache_manager.py): `serving/core/kv_cache_manager.py`
- [Block pools](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/block_pool.py): `serving/core/block_pool.py`
- [Communication model](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/communication.py): `serving/core/communication.py`
- [MoE execution](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/moe_execution.py): `serving/core/moe_execution.py`
- [Speculative decoding](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/spec_decode.py): `serving/core/spec_decode.py`
- [Power model](https://github.com/casys-kaist/llmservingsim/blob/main/serving/core/power_model.py): `serving/core/power_model.py`

### Workload inputs

- [Workload generator CLI](https://github.com/casys-kaist/llmservingsim/blob/main/workloads/generators/__main__.py): `workloads/generators/__main__.py`
- [ShareGPT generator](https://github.com/casys-kaist/llmservingsim/blob/main/workloads/generators/sharegpt.py): `workloads/generators/sharegpt.py`

### Other components

- **Operator**
- **ASTRA-Sim**
- **Request workload**

## Connections

- Operator → Profiler CLI: profiles models
- Profiler CLI → vLLM engine: starts profiling (evidence: `profiler/__main__.py`)
- vLLM engine → Runtime capture: runs capture
- Runtime capture → Timing records: records timings (evidence: `profiler/core/hooks/activity_collector.py`)
- Runtime capture → Capture contexts: uses contexts (evidence: `profiler/core/hooks/activity_collector.py`)
- Profiler CLI → Component layouts: uses layouts (evidence: `profiler/__main__.py`)
- Profiler CLI → Profile bundle: publishes profiles
- Operator → Workload generator CLI: generates workload
- Workload generator CLI → ShareGPT generator: calls generator (evidence: `workloads/generators/__main__.py`)
- ShareGPT generator → Request workload: emits requests
- Operator → Simulator entry: runs simulation
- Request workload → Simulator entry: supplies requests
- Profile bundle → Simulator entry: provides timings (evidence: `serving/__main__.py`)
- Simulator entry → Request router: routes requests (evidence: `serving/__main__.py`)
- Simulator entry → Batch scheduler: schedules iterations (evidence: `serving/__main__.py`)
- Simulator entry → Trace generation: generates traces (evidence: `serving/__main__.py`)
- Simulator entry → Execution graph: builds graphs (evidence: `serving/__main__.py`)
- Simulator entry → ASTRA-Sim: runs backend
- Trace generation → Memory model: uses memory model (evidence: `serving/core/trace_generator.py`)
- Memory model → Tiered KV cache: manages KV cache (evidence: `serving/core/memory_model.py`)
- Tiered KV cache → Block pools: allocates blocks (evidence: `serving/core/kv_cache_manager.py`)
- Trace generation → Communication model: models communication (evidence: `serving/core/trace_generator.py`)
- Trace generation → MoE execution: models MoE (evidence: `serving/core/trace_generator.py`)
- Trace generation → Power model: models power (evidence: `serving/core/trace_generator.py`)
- Simulator entry → Speculative decoding: configures decoding (evidence: `serving/__main__.py`)