How thustorage/medusa works
Main Components
• API/Serverless Inference Layer: This covers the HTTP endpoint(s) implemented using FastAPI (see examples such as “gradio_openai_chatbot_webserver.py”, “openai_chatcompletion_client.py” and the entrypoints under vllm/entrypoints) that accept inference requests.
• LLM Engine and Scheduling Core: This component is built on top of vLLM (the “vllm” directory) and includes submodules such as the engine (both synchronous and asynchronous engines in vllm/engine), the scheduler (vllm/core/scheduler.py) and related modules (e.g., policy, metrics, and ray_utils).
• CUDA Graph Materialization Module: The project puts a heavy emphasis on reducing cold-start latency by saving and recovering CUDA Graphs. Its implementation resides partly in the “csrc” directory (including CUDA kernels, graph-related functions, and custom CUDA utilities) as well as in scripts like “cuda_graph.py”. This component connects with the LLM engine to “materialize” state.
• Persistent Storage & SPDK Integration: The system uses SPDK to interact with high-performance SSDs for quick data retrieval. SPDK-related code (and modifications of PyTorch in the PyTorch-Medusa branch) is integrated through configuration files, build scripts, and support utilities in folders like “spdk_test” and the “data-backup” directories showing high-performance I/O experiments.
• Testing, Benchmarking, and Experimentation: Numerous directories (benchmarks, experiments, breakdowns, tests) support regression testing, performance benchmarking (latency, throughput, GPU utilization) and experimental runs (e.g., traces and overall performance). They help validate the system’s latency improvements and scalability.
Relationships and Data Flow
– Show that these requests are then forwarded to the LLM engine, which manages workloads via its scheduler.
– Highlight that the engine leverages specialized CUDA Graph caching modules to “materialize” state, reducing cold-start latencies. This module communicates with the GPU kernel modules (from “csrc”) to save/restore state.
– Illustrate the integration with SPDK: persistent storage (the high-speed SSDs) is used for quickly saving/restoring model tensors and CUDA Graph state.
– Optionally, indicate that performance monitoring, testing and experimental results feed back into system tuning (through “benchmarks”, “experiments”, and “results”).
Architectural Patterns and Technologies
– The system follows a layered architecture typical of full-stack inference modules where a front-end API/service interface is clearly separated from the compute engine.
– It also features high-performance design patterns by customizing lower-level CUDA operations and interfacing directly with hardware (using SPDK and custom CUDA kernels).
– Use of asynchronous processing (vLLM’s async engine and FastAPI’s async features) is evident.
– Dependencies include PyTorch and vLLM for deep model inference, as well as several GPU-accelerated libraries (cuBLAS, cuDNN, CUDA) and disk I/O libraries (SPDK).
Instructing the Engineer
• From the API layer to the LLM engine.
• From the engine to the CUDA Graph module (for state saving/recovery).
• From the CUDA Graph module to the SPDK-managed storage.
– Optionally, include an “Experiments & Benchmarking” area that feeds metrics back to allow performance tuning.
– Clearly denote external dependencies (such as PyTorch, CUDA, SPDK, and other libraries) as part of the infrastructure.