# vllm-project/vllm-gaudi architecture

> Architecture diagram of the GitHub repository vllm-project/vllm-gaudi, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vllm-project/vllm-gaudi
- Repository: https://github.com/vllm-project/vllm-gaudi

## Overview

##### Project Type and Purpose
- Type: Python-based, open-source plugin module for the vLLM inference engine.
- Purpose: Provide a modular interface to Intel Gaudi accelerators by supplying custom kernels, attention backends, quantization ops, distributed communicators, and runtime extensions.

##### File Structure
- Top-level folders:
– vllm_gaudi/: plugin source code, subdivided into attention, ops, distributed, extension, utils, platform, and a v1 variant.
– tests/: unit, full, and upstream tests exercising plugin functionality.
– .github, .jenkins, tools/: CI/CD pipelines, benchmark scripts, pre-commit and lint config.
– docs/: MkDocs-powered documentation—API, design guides, configuration, features.
- Notice patterns:
– “extension/” implements plugin hooks (bucketing, config, runtime, profiler).
– “attention/backends” and “ops” supply HPU-optimized kernels.
– “distributed/device_communicators” abstracts multi-Gaudi communication.
– “v1/” mirrors legacy API compatibility.

##### Extract Architecture Insights from README
- Plugin follows vLLM’s hardware-pluggable RFCs.
- Depends on vLLM core at runtime.
- Leverages HabanaAI’s Python and C++ Habana drivers and PyTorch integration.

##### Main Components & Their Relationships
a. vLLM Core
– The host inference engine, loads plugins via its plugin framework.
b. Plugin Interface Layer (in vLLM Core)
– Defines abstract hooks: attention, ops, distributed communicator, runtime config.
c. vLLM Gaudi Plugin
– Extension Module: config, environment setup, feature flags, bucketing, caching, profiling.
– Attention Backends: HPU-paged attention kernels.
– Ops Suite: quantization (AWQ, GPTQ), fused MoE, layernorm, LoRA, rotary embeddings (all HPU-specific).
– Distributed Communicator: abstracts Habana HPU collectives across devices.
– Platform & Utils: device querying, logging, config loaders.
– Legacy v1 Subpackage: backward compatibility for older vLLM plugin API.
d. External Services & Systems
– HabanaAI Drivers & Libraries (C++/Python bindings)
– PyTorch (for tensor support and dispatcher)
– Intel Gaudi Hardware (HPU)
e. CI/CD & Documentation
– GitHub Actions & Jenkins jobs: linting, type checking, benchmarks, end-to-end tests.
– MkDocs: documentation site.

##### Architectural Patterns & Principles
- Plugin / Strategy Pattern: vLLM core invokes plugin-registered implementations of attention and ops.
- Adapter Pattern: wraps HabanaAI library calls behind a uniform API.
- Separation of Concerns:
– Core inference logic vs. hardware-specific plugin code.
– Extension layer (config, profiling) vs. compute kernels (attention, ops).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Layers
    subgraph "vLLM Core" 
        Core["vLLM Core"]:::core
        PluginInterface["Plugin Interface Layer"]:::core
    end

    subgraph "vLLM Gaudi Plugin" 
        direction TB
        subgraph "Extension Module" 
            ExtMod["Extension Module"]:::plugin
            Config["config.py"]:::plugin
            Env["environment.py"]:::plugin
            Bucketing["bucketing/"]:::plugin
            Cache["cache_ops.py"]:::plugin
            Profiler["profiler.py"]:::plugin
            ExtMod --> Config
            ExtMod --> Env
            ExtMod --> Bucketing
            ExtMod --> Cache
            ExtMod --> Profiler
        end

        subgraph "Attention Backends" 
            AttBack["Attention Backends"]:::plugin
            HPUAttn["hpu_attn.py"]:::plugin
            PagedAttn["hpu_paged_attn.py"]:::plugin
            AttBack --> HPUAttn
            AttBack --> PagedAttn
        end

        subgraph "Ops Suite" 
            Ops["Ops Suite"]:::plugin
            AWQ["hpu_awq.py"]:::plugin
            Compressed["hpu_compressed_tensors.py"]:::plugin
            FP8["hpu_fp8.py"]:::plugin
            Moe["hpu_fused_moe.py"]:::plugin
            GPTQ["hpu_gptq.py"]:::plugin
            LN["hpu_layernorm.py"]:::plugin
            LoRA["hpu_lora.py"]:::plugin
            Rotary["hpu_rotary_embedding.py"]:::plugin
            Ops --> AWQ
            Ops --> Compressed
            Ops --> FP8
            Ops --> Moe
            Ops --> GPTQ
            Ops --> LN
            Ops --> LoRA
            Ops --> Rotary
        end

        subgraph "Distributed Communicator" 
            Dist["Distributed Communicator"]:::plugin
            HPUComm["hpu_communicator.py"]:::plugin
            Dist --> HPUComm
        end

        subgraph "Platform & Utils" 
            Plat["Platform & Utilities"]:::plugin
            PlatformPy["platform.py"]:::plugin
            UtilsPy["utils.py"]:::plugin
            Plat --> PlatformPy
            Plat --> UtilsPy
        end

        subgraph "Legacy v1 Subpackage" 
            V1["v1 Subpackage"]:::plugin
            V1Attn["v1/attention/backends/hpu_attn.py"]:::plugin
            V1WorkerDir["v1/worker/"]:::plugin
            V1 --> V1Attn
            V1 --> V1WorkerDir
        end
    end

    subgraph "External Systems" 
        Habana["HabanaAI Drivers & Libs"]:::external
        PyTorch["PyTorch"]:::external
        GaudiHW["Intel Gaudi HPU"]:::external
    end

    subgraph "CI/CD Pipelines" 
        CI["CI/CD"]:::ci
        GH[".github/workflows/"]:::ci
        JenkinsBench[".jenkins/benchmark/"]:::ci
        JenkinsLM[".jenkins/lm-eval-harness/"]:::ci
        JenkinsVis[".jenkins/vision/"]:::ci
        CI --> GH
        CI --> JenkinsBench
        CI --> JenkinsLM
        CI --> JenkinsVis
    end

    subgraph "Documentation" 
        Docs["Documentation"]:::docs
        APIRef["docs/api/vllm_hpu/"]:::docs
        Design["docs/design/plugin_system.md"]:::docs
        Features["docs/features/quantization/"]:::docs
        ConfigDocs["docs/configuration/"]:::docs
        GettingStarted["docs/getting_started/"]:::docs
        Docs --> APIRef
        Docs --> Design
        Docs --> Features
        Docs --> ConfigDocs
        Docs --> GettingStarted
    end

    %% Connections
    Core --> PluginInterface
    PluginInterface --> ExtMod
    PluginInterface --> AttBack
    PluginInterface --> Ops
    PluginInterface --> Dist
    PluginInterface --> Plat
    PluginInterface --> V1

    ExtMod -->|"invokes hooks"| Habana
    AttBack -->|"calls kernels via PyTorch Tensor API"| PyTorch
    Ops -->|"dispatches ops via PyTorch"| PyTorch
    Dist -->|"Habana collectives RPC"| Habana
    Plat -->|"queries device"| GaudiHW
    PlatformPy -->|"loads drivers"| Habana
    PyTorch -->|"runs on"| GaudiHW

    %% Click Events Extension
    click ExtMod "https://github.com/vllm-project/vllm-gaudi/tree/main/vllm_gaudi/extension/"
    click Config "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/extension/config.py"
    click Env "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/extension/environment.py"
    click Bucketing "https://github.com/vllm-project/vllm-gaudi/tree/main/vllm_gaudi/extension/bucketing/"
    click Cache "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/extension/cache_ops.py"
    click Profiler "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/extension/profiler.py"

    %% Click Events Attention
    click HPUAttn "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/attention/backends/hpu_attn.py"
    click PagedAttn "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/attention/ops/hpu_paged_attn.py"

    %% Click Events Ops Suite
    click AWQ "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_awq.py"
    click Compressed "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_compressed_tensors.py"
    click FP8 "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_fp8.py"
    click Moe "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_fused_moe.py"
    click GPTQ "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_gptq.py"
    click LN "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_layernorm.py"
    click LoRA "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_lora.py"
    click Rotary "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/ops/hpu_rotary_embedding.py"

    %% Click Events Distributed
    click HPUComm "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/distributed/device_communicators/hpu_communicator.py"

    %% Click Events Platform & Utils
    click PlatformPy "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/platform.py"
    click UtilsPy "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/utils.py"

    %% Click Events Legacy v1
    click V1 "https://github.com/vllm-project/vllm-gaudi/tree/main/vllm_gaudi/v1/"
    click V1Attn "https://github.com/vllm-project/vllm-gaudi/blob/main/vllm_gaudi/v1/attention/backends/hpu_attn.py"
    click V1WorkerDir "https://github.com/vllm-project/vllm-gaudi/tree/main/vllm_gaudi/v1/worker/"

    %% Click Events CI/CD
    click GH "https://github.com/vllm-project/vllm-gaudi/tree/main/.github/workflows/"
    click JenkinsBench "https://github.com/vllm-project/vllm-gaudi/tree/main/.jenkins/benchmark/"
    click JenkinsLM "https://github.com/vllm-project/vllm-gaudi/tree/main/.jenkins/lm-eval-harness/"
    click JenkinsVis "https://github.com/vllm-project/vllm-gaudi/tree/main/.jenkins/vision/"

    %% Click Events Documentation
    click APIRef "https://github.com/vllm-project/vllm-gaudi/tree/main/docs/api/vllm_hpu/"
    click Design "https://github.com/vllm-project/vllm-gaudi/blob/main/docs/design/plugin_system.md"
    click Features "https://github.com/vllm-project/vllm-gaudi/tree/main/docs/features/quantization/"
    click ConfigDocs "https://github.com/vllm-project/vllm-gaudi/tree/main/docs/configuration/"
    click GettingStarted "https://github.com/vllm-project/vllm-gaudi/tree/main/docs/getting_started/"

    %% Styles
    classDef core fill:#b3cde0,stroke:#333,stroke-width:1px;
    classDef plugin fill:#ccebc5,stroke:#333,stroke-width:1px;
    classDef external fill:#e5e5e5,stroke:#333,stroke-width:1px;
    classDef ci fill:#decbe4,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5;
    classDef docs fill:#fed9a6,stroke:#333,stroke-width:1px,stroke-dasharray: 2 2;
```
