# w2315516642/vllm-plugin-fl architecture

> Architecture diagram of the GitHub repository w2315516642/vllm-plugin-fl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/w2315516642/vllm-plugin-fl
- Repository: https://github.com/w2315516642/vllm-plugin-fl

## Overview

vllm-plugin-FL extends vLLM inference across heterogeneous accelerator hardware while retaining vLLM’s user-facing workflow. A request enters through vLLM’s worker integration, is prepared and executed by the model runner, and uses platform-specific execution, graph support, and operator adapters before producing inference output. Operator dispatch selects implementations under backend policy, with diagnostics available for operator calls. The repository also includes distributed communication and KV transfer, model-specific integrations, and quantization. Edges rely on README statements and SOURCE INDEX references; unsampled implementation details are not inferred.

Main entry points: [`vllm_fl/dispatch/manager.py`](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/manager.py) (Dispatch manager), [`vllm_fl/worker/model_runner.py`](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/model_runner.py) (Model execution), [`vllm_fl/dispatch/io_dumper.py`](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_dumper.py) (Operator diagnostics), [`vllm_fl/worker/worker.py`](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/worker.py) (Worker lifecycle)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Inference runtime"]
  node_worker["Worker lifecycle<br/>[worker.py]"]
  node_runner["Model execution<br/>[model_runner.py]"]
  node_metadata["Attention metadata"]
  node_graphs["Graph execution<br/>[graph.py]"]
end

subgraph group_dispatch["Operator dispatch"]
  node_dispatch_mgr["Dispatch manager<br/>[manager.py]"]
  node_policy["Backend policy<br/>[policy.py]"]
  node_registry["Implementation registry<br/>[registry.py]"]
  node_discovery["Plugin discovery<br/>[discovery.py]"]
  node_diagnostics["Operator diagnostics<br/>[io_dumper.py]"]
  node_io_common["Diagnostic utilities<br/>[io_common.py]"]
  node_logger["Dispatch logging<br/>[logger_manager.py]"]
end

subgraph group_kernels["Kernels and models"]
  node_flag_gems["FlagGems backend<br/>[attention.py]"]
  node_reference["Reference backend<br/>[reference.py]"]
  node_custom_ops["Custom operators<br/>[custom_ops.py]"]
  node_moe_router["MoE routing<br/>[router.py]"]
  node_moe_utils["MoE utilities<br/>[fused_moe_utils.py]"]
  node_rotary["Rotary embedding"]
  node_model["Model integration<br/>[qwen3_5.py]"]
  node_quant["Quantization<br/>[arm_cpu_w4a8.py]"]
end

subgraph group_hardware["Hardware and distributed"]
  node_platform["Platform integration<br/>[platform.py]"]
  node_communicator["FlagCX communicator<br/>[flagcx.py]"]
  node_kv_transfer["KV transfer"]
  node_kv_stats["Transfer statistics<br/>[flagcx_stats.py]"]
end

node_caller(("vLLM caller"))

node_caller -->|"submits inference"| node_worker
node_worker -->|"executes model"| node_runner
node_runner -->|"uses metadata"| node_metadata
node_runner -->|"uses graph support"| node_graphs
node_runner -->|"queries platform"| node_platform
node_runner -->|"uses diagnostics"| node_diagnostics
node_worker -->|"uses operators"| node_custom_ops
node_custom_ops -->|"imports utilities"| node_moe_utils
node_moe_router -->|"dispatches operators"| node_dispatch_mgr
node_rotary -->|"dispatches operator"| node_dispatch_mgr
node_dispatch_mgr -->|"queries implementations"| node_registry
node_dispatch_mgr -->|"applies policy"| node_policy
node_dispatch_mgr -->|"routes diagnostics"| node_diagnostics
node_diagnostics -->|"uses utilities"| node_io_common
node_diagnostics -->|"uses logging"| node_logger
node_discovery -.->|"registers plugins"| node_dispatch_mgr
node_dispatch_mgr -->|"selects backend"| node_flag_gems
node_dispatch_mgr -->|"supports fallback"| node_reference
node_platform -->|"uses FlagCX"| node_communicator
node_kv_transfer -->|"records statistics"| node_kv_stats

click node_worker "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/worker.py"
click node_runner "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/model_runner.py"
click node_metadata "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/common_attention_metadata.py"
click node_graphs "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/compilation/graph.py"
click node_platform "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/platform.py"
click node_dispatch_mgr "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/manager.py"
click node_policy "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/policy.py"
click node_registry "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/registry.py"
click node_discovery "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/discovery.py"
click node_diagnostics "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_dumper.py"
click node_io_common "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_common.py"
click node_logger "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/logger_manager.py"
click node_flag_gems "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/flaggems/impl/attention.py"
click node_reference "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/reference/reference.py"
click node_custom_ops "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/custom_ops.py"
click node_moe_router "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/fused_moe/router.py"
click node_moe_utils "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/fused_moe/fused_moe_utils.py"
click node_rotary "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/rotary_embedding.py"
click node_model "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/models/qwen3_5.py"
click node_quant "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/quantization/arm_cpu_w4a8.py"
click node_communicator "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/distributed/device_communicators/flagcx.py"
click node_kv_transfer "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_connector.py"
click node_kv_stats "https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_stats.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_worker,node_runner,node_metadata,node_graphs toneBlue
class node_dispatch_mgr,node_policy,node_registry,node_discovery,node_diagnostics,node_io_common,node_logger toneAmber
class node_flag_gems,node_reference,node_custom_ops,node_moe_router,node_moe_utils,node_rotary,node_model,node_quant toneMint
class node_platform,node_communicator,node_kv_transfer,node_kv_stats toneRose
class node_caller toneIndigo
```

## Components

### Inference runtime

- [Worker lifecycle](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/worker.py): `vllm_fl/worker/worker.py`
- [Model execution](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/model_runner.py): `vllm_fl/worker/model_runner.py`
- [Attention metadata](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/worker/common_attention_metadata.py): `vllm_fl/worker/common_attention_metadata.py`
- [Graph execution](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/compilation/graph.py): `vllm_fl/compilation/graph.py`

### Operator dispatch

- [Dispatch manager](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/manager.py): `vllm_fl/dispatch/manager.py`
- [Backend policy](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/policy.py): `vllm_fl/dispatch/policy.py`
- [Implementation registry](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/registry.py): `vllm_fl/dispatch/registry.py`
- [Plugin discovery](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/discovery.py): `vllm_fl/dispatch/discovery.py`
- [Operator diagnostics](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_dumper.py): `vllm_fl/dispatch/io_dumper.py`
- [Diagnostic utilities](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_common.py): `vllm_fl/dispatch/io_common.py`
- [Dispatch logging](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/logger_manager.py): `vllm_fl/dispatch/logger_manager.py`

### Kernels and models

- [FlagGems backend](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/flaggems/impl/attention.py): `vllm_fl/dispatch/backends/flaggems/impl/attention.py`
- [Reference backend](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/reference/reference.py): `vllm_fl/dispatch/backends/reference/reference.py`
- [Custom operators](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/custom_ops.py): `vllm_fl/ops/custom_ops.py`
- [MoE routing](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/fused_moe/router.py): `vllm_fl/ops/fused_moe/router.py`
- [MoE utilities](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/fused_moe/fused_moe_utils.py): `vllm_fl/ops/fused_moe/fused_moe_utils.py`
- [Rotary embedding](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/ops/rotary_embedding.py): `vllm_fl/ops/rotary_embedding.py`
- [Model integration](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/models/qwen3_5.py): `vllm_fl/models/qwen3_5.py`
- [Quantization](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/quantization/arm_cpu_w4a8.py): `vllm_fl/quantization/arm_cpu_w4a8.py`

### Hardware and distributed

- [Platform integration](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/platform.py): `vllm_fl/platform.py`
- [FlagCX communicator](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/distributed/device_communicators/flagcx.py): `vllm_fl/distributed/device_communicators/flagcx.py`
- [KV transfer](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_connector.py): `vllm_fl/distributed/kv_transfer/flagcx_connector.py`
- [Transfer statistics](https://github.com/w2315516642/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_stats.py): `vllm_fl/distributed/kv_transfer/flagcx_stats.py`

### Other components

- **vLLM caller**

## Connections

- vLLM caller → Worker lifecycle: submits inference (evidence: `README.md`)
- Worker lifecycle → Model execution: executes model (evidence: `vllm_fl/worker/worker.py`)
- Model execution → Attention metadata: uses metadata (evidence: `vllm_fl/worker/model_runner.py`)
- Model execution → Graph execution: uses graph support (evidence: `vllm_fl/worker/model_runner.py`)
- Model execution → Platform integration: queries platform (evidence: `vllm_fl/worker/model_runner.py`)
- Model execution → Operator diagnostics: uses diagnostics (evidence: `vllm_fl/worker/model_runner.py`)
- Worker lifecycle → Custom operators: uses operators (evidence: `vllm_fl/worker/worker.py`)
- Custom operators → MoE utilities: imports utilities (evidence: `vllm_fl/ops/custom_ops.py`)
- MoE routing → Dispatch manager: dispatches operators (evidence: `vllm_fl/ops/fused_moe/router.py`)
- Rotary embedding → Dispatch manager: dispatches operator (evidence: `vllm_fl/ops/rotary_embedding.py`)
- Dispatch manager → Implementation registry: queries implementations (evidence: `vllm_fl/dispatch/manager.py`)
- Dispatch manager → Backend policy: applies policy (evidence: `vllm_fl/dispatch/manager.py`)
- Dispatch manager → Operator diagnostics: routes diagnostics (evidence: `vllm_fl/dispatch/manager.py`)
- Operator diagnostics → Diagnostic utilities: uses utilities (evidence: `vllm_fl/dispatch/io_dumper.py`)
- Operator diagnostics → Dispatch logging: uses logging (evidence: `vllm_fl/dispatch/io_dumper.py`)
- Plugin discovery → Dispatch manager: registers plugins
- Dispatch manager → FlagGems backend: selects backend (evidence: `README.md`)
- Dispatch manager → Reference backend: supports fallback (evidence: `README.md`)
- Platform integration → FlagCX communicator: uses FlagCX (evidence: `README.md`)
- KV transfer → Transfer statistics: records statistics (evidence: `vllm_fl/distributed/kv_transfer/flagcx_connector.py`)
