# thustorage/medusa architecture

> Architecture diagram of the GitHub repository thustorage/medusa, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/thustorage/medusa
- Repository: https://github.com/thustorage/medusa

## Overview

##### Main Components
- API/Serverless Inference Layer: This covers the HTTP endpoint(s) implemented using FastAPI (see examples such as “gradio_openai_chatbot_webserver.py”, “openai_chatcompletion_client.py” and the entrypoints under vllm/entrypoints) that accept inference requests.
- LLM Engine and Scheduling Core: This component is built on top of vLLM (the “vllm” directory) and includes submodules such as the engine (both synchronous and asynchronous engines in vllm/engine), the scheduler (vllm/core/scheduler.py) and related modules (e.g., policy, metrics, and ray_utils).
- CUDA Graph Materialization Module: The project puts a heavy emphasis on reducing cold-start latency by saving and recovering CUDA Graphs. Its implementation resides partly in the “csrc” directory (including CUDA kernels, graph-related functions, and custom CUDA utilities) as well as in scripts like “cuda_graph.py”. This component connects with the LLM engine to “materialize” state.
- Persistent Storage & SPDK Integration: The system uses SPDK to interact with high-performance SSDs for quick data retrieval. SPDK-related code (and modifications of PyTorch in the PyTorch-Medusa branch) is integrated through configuration files, build scripts, and support utilities in folders like “spdk_test” and the “data-backup” directories showing high-performance I/O experiments.
- Testing, Benchmarking, and Experimentation: Numerous directories (benchmarks, experiments, breakdowns, tests) support regression testing, performance benchmarking (latency, throughput, GPU utilization) and experimental runs (e.g., traces and overall performance). They help validate the system’s latency improvements and scalability.

##### Relationships and Data Flow
– Show that these requests are then forwarded to the LLM engine, which manages workloads via its scheduler.
– Highlight that the engine leverages specialized CUDA Graph caching modules to “materialize” state, reducing cold-start latencies. This module communicates with the GPU kernel modules (from “csrc”) to save/restore state.
– Illustrate the integration with SPDK: persistent storage (the high-speed SSDs) is used for quickly saving/restoring model tensors and CUDA Graph state.
– Optionally, indicate that performance monitoring, testing and experimental results feed back into system tuning (through “benchmarks”, “experiments”, and “results”).

##### Architectural Patterns and Technologies
– The system follows a layered architecture typical of full-stack inference modules where a front-end API/service interface is clearly separated from the compute engine.
– It also features high-performance design patterns by customizing lower-level CUDA operations and interfacing directly with hardware (using SPDK and custom CUDA kernels).
– Use of asynchronous processing (vLLM’s async engine and FastAPI’s async features) is evident.
– Dependencies include PyTorch and vLLM for deep model inference, as well as several GPU-accelerated libraries (cuBLAS, cuDNN, CUDA) and disk I/O libraries (SPDK).

##### Instructing the Engineer
- From the API layer to the LLM engine.
- From the engine to the CUDA Graph module (for state saving/recovery).
- From the CUDA Graph module to the SPDK-managed storage.
– Optionally, include an “Experiments & Benchmarking” area that feeds metrics back to allow performance tuning.
– Clearly denote external dependencies (such as PyTorch, CUDA, SPDK, and other libraries) as part of the infrastructure.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "API/Serverless Inference Layer"
        node_api1("Gradio Chatbot (FastAPI)"):::api
        node_api2("Chat Completion Client (FastAPI)"):::api
        node_api3("API Entrypoint (FastAPI)"):::api
    end

    subgraph "LLM Engine and Scheduling Core"
        subgraph "LLM Engine (vLLM)"
            node_engine_sync("LLM Engine (Sync)"):::engine
            node_engine_async("LLM Engine (Async)"):::engine
            node_metrics("Metrics & Utilities"):::engine
        end
        node_scheduler("Scheduler (vLLM Core)"):::engine
    end

    subgraph "CUDA Graph Materialization Module"
        node_cuda_cu("CUDA Graph (csrc)"):::gpu
        node_cuda_script("CUDA Graph Script (Python)"):::gpu
    end

    subgraph "Persistent Storage & SPDK Integration"
        node_spdk_test("SPDK Test (Directory)"):::storage
        node_spdk_util_cpp("SPDK Util (csrc/c++)"):::storage
        node_spdk_util_h("SPDK Util Header (csrc)"):::storage
    end

    subgraph "Model Repository and Preprocessing"
        node_model_offsets("Model Offsets"):::model
        node_model_tokenizers("Model Tokenizers"):::model
    end

    subgraph "Testing, Benchmarking & Experimentation"
        node_benchmarks("Benchmarks"):::testing
        node_experiments("Experiments"):::testing
        node_tests("Tests"):::testing
    end

    subgraph "Build & Deployment Infrastructure"
        node_github(".github"):::deployment
        node_buildkite(".buildkite"):::deployment
        node_dockerfile("Dockerfile"):::deployment
        node_pyproject("pyproject.toml"):::deployment
        node_requirements("requirements.txt"):::deployment
    end

    %% Relationships / Data Flow
    node_api1 -->|"InferenceRequest"| node_engine_sync
    node_api2 -->|"InferenceRequest"| node_engine_async
    node_api3 -->|"InferenceRequest"| node_engine_sync
    node_engine_sync -->|"Schedules"| node_scheduler
    node_engine_async -->|"Schedules"| node_scheduler
    node_engine_sync -->|"Uses"| node_cuda_cu
    node_engine_async -->|"Uses"| node_cuda_script
    node_cuda_cu -->|"SavesState"| node_spdk_test
    node_cuda_script -->|"RecoversState"| node_spdk_test
    node_spdk_util_cpp -->|"Support"| node_spdk_test
    node_spdk_util_h -->|"Support"| node_spdk_test
    node_model_offsets -->|"ProvidesData"| node_engine_sync
    node_model_tokenizers -->|"ProvidesData"| node_engine_async
    node_benchmarks -->|"Monitors"| node_engine_sync
    node_experiments -->|"Optimizes"| node_engine_async
    node_tests -->|"Validates"| node_api3

    %% Styles
    classDef api fill:#FFCCFF,stroke:#333,stroke-width:2px;
    classDef engine fill:#FFFF99,stroke:#333,stroke-width:2px;
    classDef gpu fill:#CCFFCC,stroke:#333,stroke-width:2px;
    classDef storage fill:#CCFFFF,stroke:#333,stroke-width:2px;
    classDef model fill:#FFE5CC,stroke:#333,stroke-width:2px;
    classDef testing fill:#CCCCFF,stroke:#333,stroke-width:2px;
    classDef deployment fill:#CCFFEE,stroke:#333,stroke-width:2px;

    %% Click Events
    click node_api1 "https://github.com/thustorage/medusa/blob/asplos-ae/examples/gradio_openai_chatbot_webserver.py"
    click node_api2 "https://github.com/thustorage/medusa/blob/asplos-ae/examples/openai_chatcompletion_client.py"
    click node_api3 "https://github.com/thustorage/medusa/blob/asplos-ae/vllm/entrypoints/api_server.py"
    click node_engine_sync "https://github.com/thustorage/medusa/blob/asplos-ae/vllm/engine/llm_engine.py"
    click node_engine_async "https://github.com/thustorage/medusa/blob/asplos-ae/vllm/engine/async_llm_engine.py"
    click node_metrics "https://github.com/thustorage/medusa/blob/asplos-ae/vllm/engine/metrics.py"
    click node_scheduler "https://github.com/thustorage/medusa/blob/asplos-ae/vllm/core/scheduler.py"
    click node_cuda_cu "https://github.com/thustorage/medusa/blob/asplos-ae/csrc/cuda_graph.cu"
    click node_cuda_script "https://github.com/thustorage/medusa/blob/asplos-ae/scripts/cuda_graph.py"
    click node_spdk_test "https://github.com/thustorage/medusa/tree/asplos-ae/spdk_test"
    click node_spdk_util_cpp "https://github.com/thustorage/medusa/blob/asplos-ae/csrc/spdk_util.cpp"
    click node_spdk_util_h "https://github.com/thustorage/medusa/blob/asplos-ae/csrc/spdk_util.h"
    click node_model_offsets "https://github.com/thustorage/medusa/tree/asplos-ae/model_offsets"
    click node_model_tokenizers "https://github.com/thustorage/medusa/tree/asplos-ae/model_tokenizers"
    click node_benchmarks "https://github.com/thustorage/medusa/tree/asplos-ae/benchmarks"
    click node_experiments "https://github.com/thustorage/medusa/tree/asplos-ae/experiments"
    click node_tests "https://github.com/thustorage/medusa/tree/asplos-ae/tests"
    click node_github "https://github.com/thustorage/medusa/blob/asplos-ae/.github"
    click node_buildkite "https://github.com/thustorage/medusa/blob/asplos-ae/.buildkite"
    click node_dockerfile "https://github.com/thustorage/medusa/tree/asplos-ae/Dockerfile"
    click node_pyproject "https://github.com/thustorage/medusa/blob/asplos-ae/pyproject.toml"
    click node_requirements "https://github.com/thustorage/medusa/blob/asplos-ae/requirements.txt"
```
