How microsoft/vidur works
Main Components Identification
a) Core Simulation Components:
• Request Generator System
• Scheduler System (Global, Replica, and Stage schedulers)
• Execution Time Predictor
• Event Management System
• Metrics Collection System
b) Configuration Components:
• Model Configurations
• Device SKU Configurations
• Node SKU Configurations
• Simulation Parameters
Component Relationships
• Blue: Core simulation components
• Green: Input/configuration components
• Orange: Scheduling components
• Purple: Monitoring and metrics
• Gray: External integrations
Specific Implementation Details
• Supported model implementations (Llama, CodeLlama, etc.)
• Hardware configurations (A100, H100, multi-GPU setups)
• Scheduler types (SARATHI, VLLM, etc.)
• Profiling capabilities (attention, MLP, network)
Key Features to Highlight
• Pipeline parallelism support
• Tensor parallelism configurations
• Maximum context length handling
• Multi-GPU support
• Trace replay capabilities
Integration Points
External integration points for:
• Wandb logging
• Chrome tracing
• Profiling tools
• Configuration management