How casys-kaist/llmservingsim works
LLMServingSim combines a vLLM-based latency profiler with a cycle-level serving simulator and analytical network backend. The main workflow is to acquire per-hardware model timings, then use those profiles and configured workloads to simulate request scheduling, model execution, memory use, and communication, producing serving traces/results. The simulator also models tiered KV memory, MoE, speculative decoding, power, and TP/PP/EP/DP. The map follows documented flow and sampled entry-point references; several subsystem internals are unsampled, so their internal links are omitted.
Main entry points: serving/__main__.py Simulator entry · trace_generator.py Trace generation · profiler/__main__.py Profiler CLI · activity_collector.py Runtime capture
Components 24
Latency profiling
- Profiler CLI
profiler/__main__.py - vLLM engine
profiler/core/engine.py - Runtime capture
profiler/core/hooks/activity_collector.py - Timing records
profiler/core/hooks/timings.py - Capture contexts
profiler/core/hooks/callable_context.py - Component layouts
profiler/core/component_layouts.py - Profile bundle
profiler/core/bundle_metadata.py
Serving simulation
- Simulator entry
serving/__main__.py - Request router
serving/core/router.py - Batch scheduler
serving/core/scheduler.py - Trace generation
serving/core/trace_generator.py - Execution graph
serving/core/graph_generator.py
Hardware and model
- Memory model
serving/core/memory_model.py - Tiered KV cache
serving/core/kv_cache_manager.py - Block pools
serving/core/block_pool.py - Communication model
serving/core/communication.py - MoE execution
serving/core/moe_execution.py - Speculative decoding
serving/core/spec_decode.py - Power model
serving/core/power_model.py
Workload inputs
- Workload generator CLI
workloads/generators/__main__.py - ShareGPT generator
workloads/generators/sharegpt.py
Other components
- Operator
- ASTRA-Sim
- Request workload
Connections 25
- Operator → Profiler CLI: profiles models
- Profiler CLI → vLLM engine: starts profiling
profiler/__main__.py - vLLM engine → Runtime capture: runs capture
- Runtime capture → Timing records: records timings
profiler/core/hooks/activity_collector.py - Runtime capture → Capture contexts: uses contexts
profiler/core/hooks/activity_collector.py - Profiler CLI → Component layouts: uses layouts
profiler/__main__.py - Profiler CLI → Profile bundle: publishes profiles
- Operator → Workload generator CLI: generates workload
- Workload generator CLI → ShareGPT generator: calls generator
workloads/generators/__main__.py - ShareGPT generator → Request workload: emits requests
- Operator → Simulator entry: runs simulation
- Request workload → Simulator entry: supplies requests
- Profile bundle → Simulator entry: provides timings
serving/__main__.py - Simulator entry → Request router: routes requests
serving/__main__.py - Simulator entry → Batch scheduler: schedules iterations
serving/__main__.py - Simulator entry → Trace generation: generates traces
serving/__main__.py - Simulator entry → Execution graph: builds graphs
serving/__main__.py - Simulator entry → ASTRA-Sim: runs backend
- Trace generation → Memory model: uses memory model
serving/core/trace_generator.py - Memory model → Tiered KV cache: manages KV cache
serving/core/memory_model.py - Tiered KV cache → Block pools: allocates blocks
serving/core/kv_cache_manager.py - Trace generation → Communication model: models communication
serving/core/trace_generator.py - Trace generation → MoE execution: models MoE
serving/core/trace_generator.py - Trace generation → Power model: models power
serving/core/trace_generator.py - Simulator entry → Speculative decoding: configures decoding
serving/__main__.py