# flagos-ai/vllm-plugin-fl architecture

> Architecture diagram of the GitHub repository flagos-ai/vllm-plugin-fl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/flagos-ai/vllm-plugin-fl
- Repository: https://github.com/flagos-ai/vllm-plugin-fl

## Overview

vllm-plugin-FL extends vLLM inference with multi-chip platform support, model/runtime adaptations, backend-dispatched operators, and optional FlagCX communication. The principal workflow begins with a vLLM inference request, proceeds through the plugin worker and model runner, and uses platform-specific execution and operator implementations to produce model outputs. Dispatch selection, graph execution, model adaptations, quantization, and distributed KV transfer are distinct supporting capabilities. Some subsystem links are documented rather than visible in sampled implementations; those are marked as documented or unverified.

Main entry points: [`vllm_fl/worker/model_runner.py`](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/model_runner.py) (Model runner), [`vllm_fl/worker/worker.py`](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/worker.py) (Worker), [`vllm_fl/dispatch/__init__.py`](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/__init__.py) (Dispatch API), [`vllm_fl/dispatch/manager.py`](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/manager.py) (Dispatch manager)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime integration"]
  node_plugin["Plugin hooks<br/>[__init__.py]"]
  node_platform["Platform adapter<br/>[platform.py]"]
end

subgraph group_execution["Inference execution"]
  node_worker["Worker<br/>[worker.py]"]
  node_runner["Model runner<br/>[model_runner.py]"]
  node_attention_meta["Attention metadata"]
  node_scheduler["Scheduler adaptation<br/>[scheduler_fl.py]"]
  node_graph["Graph wrapper<br/>[graph.py]"]
end

subgraph group_dispatch["Operator dispatch"]
  node_dispatch_api["Dispatch API<br/>[__init__.py]"]
  node_manager["Dispatch manager<br/>[manager.py]"]
  node_policy["Selection policy<br/>[policy.py]"]
  node_registry["Operator registry<br/>[registry.py]"]
  node_discovery["Backend discovery<br/>[discovery.py]"]
  node_diagnostics["I/O diagnostics<br/>[io_dumper.py]"]
end

subgraph group_kernels["Model and kernels"]
  node_flaggems["FlagGems backend<br/>[flaggems.py]"]
  node_reference["Reference backend<br/>[reference.py]"]
  node_custom_ops["Custom operators<br/>[custom_ops.py]"]
  node_moe["Fused MoE routing<br/>[router.py]"]
  node_rotary["Rotary embedding"]
  node_quant["Quantization<br/>[arm_cpu_w4a8.py]"]
  node_model_adapt["Model adaptations<br/>[qwen3_5.py]"]
end

subgraph group_distributed["Distributed runtime"]
  node_communicator["FlagCX communicator<br/>[flagcx.py]"]
  node_kv_transfer["KV transfer connector"]
  node_kv_stats["Transfer statistics<br/>[flagcx_stats.py]"]
end

node_client(("vLLM caller"))
node_hardware{{"Accelerator"}}
node_outputs["Generated outputs"]

node_client -->|"loads plugin"| node_plugin
node_plugin -->|"registers platform"| node_platform
node_client -.->|"submits inference"| node_worker
node_worker -->|"executes model"| node_runner
node_runner -.->|"returns outputs"| node_outputs
node_worker -.->|"uses device adapter"| node_platform
node_runner -->|"uses metadata"| node_attention_meta
node_runner -->|"uses graph wrapper"| node_graph
node_runner -->|"records I/O"| node_dispatch_api
node_worker -.->|"uses scheduling support"| node_scheduler
node_dispatch_api -->|"resolves calls"| node_manager
node_manager -->|"queries implementations"| node_registry
node_manager -->|"selects by policy"| node_policy
node_manager -->|"routes I/O hooks"| node_diagnostics
node_dispatch_api -->|"exposes discovery"| node_discovery
node_registry -.->|"registers implementations"| node_flaggems
node_registry -.->|"registers implementations"| node_reference
node_custom_ops -->|"uses MoE operators"| node_moe
node_custom_ops -->|"uses rotary operator"| node_rotary
node_moe -->|"dispatches routing op"| node_dispatch_api
node_rotary -->|"dispatches rotary op"| node_dispatch_api
node_runner -.->|"uses operator adapters"| node_custom_ops
node_model_adapt -.->|"registered by hooks"| node_plugin
node_quant -.->|"runs on"| node_hardware
node_platform -.->|"targets device"| node_hardware
node_communicator -.->|"communicates across devices"| node_hardware
node_kv_transfer -->|"updates statistics"| node_kv_stats
node_worker -.->|"supports KV transfer"| node_kv_transfer

click node_plugin "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/__init__.py"
click node_platform "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/platform.py"
click node_worker "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/worker.py"
click node_runner "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/model_runner.py"
click node_attention_meta "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/common_attention_metadata.py"
click node_scheduler "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/scheduler_fl.py"
click node_graph "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/compilation/graph.py"
click node_dispatch_api "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/__init__.py"
click node_manager "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/manager.py"
click node_policy "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/policy.py"
click node_registry "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/registry.py"
click node_discovery "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/discovery.py"
click node_diagnostics "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_dumper.py"
click node_flaggems "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/flaggems/flaggems.py"
click node_reference "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/reference/reference.py"
click node_custom_ops "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/ops/custom_ops.py"
click node_moe "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/ops/fused_moe/router.py"
click node_rotary "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/ops/rotary_embedding.py"
click node_quant "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/quantization/arm_cpu_w4a8.py"
click node_model_adapt "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/models/qwen3_5.py"
click node_communicator "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/distributed/device_communicators/flagcx.py"
click node_kv_transfer "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_connector.py"
click node_kv_stats "https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_stats.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_plugin,node_platform toneBlue
class node_worker,node_runner,node_attention_meta,node_scheduler,node_graph toneAmber
class node_dispatch_api,node_manager,node_policy,node_registry,node_discovery,node_diagnostics toneMint
class node_flaggems,node_reference,node_custom_ops,node_moe,node_rotary,node_quant,node_model_adapt toneRose
class node_communicator,node_kv_transfer,node_kv_stats,node_client,node_hardware,node_outputs toneIndigo
```

## Components

### Runtime integration

- [Plugin hooks](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/__init__.py): `vllm_fl/__init__.py`
- [Platform adapter](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/platform.py): `vllm_fl/platform.py`

### Inference execution

- [Worker](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/worker.py): `vllm_fl/worker/worker.py`
- [Model runner](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/model_runner.py): `vllm_fl/worker/model_runner.py`
- [Attention metadata](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/common_attention_metadata.py): `vllm_fl/worker/common_attention_metadata.py`
- [Scheduler adaptation](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/worker/scheduler_fl.py): `vllm_fl/worker/scheduler_fl.py`
- [Graph wrapper](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/compilation/graph.py): `vllm_fl/compilation/graph.py`

### Operator dispatch

- [Dispatch API](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/__init__.py): `vllm_fl/dispatch/__init__.py`
- [Dispatch manager](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/manager.py): `vllm_fl/dispatch/manager.py`
- [Selection policy](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/policy.py): `vllm_fl/dispatch/policy.py`
- [Operator registry](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/registry.py): `vllm_fl/dispatch/registry.py`
- [Backend discovery](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/discovery.py): `vllm_fl/dispatch/discovery.py`
- [I/O diagnostics](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/io_dumper.py): `vllm_fl/dispatch/io_dumper.py`

### Model and kernels

- [FlagGems backend](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/flaggems/flaggems.py): `vllm_fl/dispatch/backends/flaggems/flaggems.py`
- [Reference backend](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/dispatch/backends/reference/reference.py): `vllm_fl/dispatch/backends/reference/reference.py`
- [Custom operators](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/ops/custom_ops.py): `vllm_fl/ops/custom_ops.py`
- [Fused MoE routing](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/ops/fused_moe/router.py): `vllm_fl/ops/fused_moe/router.py`
- [Rotary embedding](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/ops/rotary_embedding.py): `vllm_fl/ops/rotary_embedding.py`
- [Quantization](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/quantization/arm_cpu_w4a8.py): `vllm_fl/quantization/arm_cpu_w4a8.py`
- [Model adaptations](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/models/qwen3_5.py): `vllm_fl/models/qwen3_5.py`

### Distributed runtime

- [FlagCX communicator](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/distributed/device_communicators/flagcx.py): `vllm_fl/distributed/device_communicators/flagcx.py`
- [KV transfer connector](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_connector.py): `vllm_fl/distributed/kv_transfer/flagcx_connector.py`
- [Transfer statistics](https://github.com/flagos-ai/vllm-plugin-fl/blob/main/vllm_fl/distributed/kv_transfer/flagcx_stats.py): `vllm_fl/distributed/kv_transfer/flagcx_stats.py`

### Other components

- **vLLM caller**
- **Accelerator**
- **Generated outputs**

## Connections

- vLLM caller → Plugin hooks: loads plugin (evidence: `README.md`)
- Plugin hooks → Platform adapter: registers platform (evidence: `vllm_fl/__init__.py`)
- vLLM caller → Worker: submits inference (evidence: `README.md`)
- Worker → Model runner: executes model (evidence: `vllm_fl/worker/worker.py`)
- Model runner → Generated outputs: returns outputs (evidence: `README.md`)
- Worker → Platform adapter: uses device adapter
- Model runner → Attention metadata: uses metadata (evidence: `vllm_fl/worker/model_runner.py`)
- Model runner → Graph wrapper: uses graph wrapper (evidence: `vllm_fl/worker/model_runner.py`)
- Model runner → Dispatch API: records I/O (evidence: `vllm_fl/worker/model_runner.py`)
- Worker → Scheduler adaptation: uses scheduling support
- Dispatch API → Dispatch manager: resolves calls (evidence: `vllm_fl/dispatch/__init__.py`)
- Dispatch manager → Operator registry: queries implementations (evidence: `vllm_fl/dispatch/manager.py`)
- Dispatch manager → Selection policy: selects by policy (evidence: `vllm_fl/dispatch/manager.py`)
- Dispatch manager → I/O diagnostics: routes I/O hooks (evidence: `vllm_fl/dispatch/manager.py`)
- Dispatch API → Backend discovery: exposes discovery (evidence: `vllm_fl/dispatch/__init__.py`)
- Operator registry → FlagGems backend: registers implementations
- Operator registry → Reference backend: registers implementations
- Custom operators → Fused MoE routing: uses MoE operators (evidence: `vllm_fl/ops/custom_ops.py`)
- Custom operators → Rotary embedding: uses rotary operator (evidence: `vllm_fl/ops/custom_ops.py`)
- Fused MoE routing → Dispatch API: dispatches routing op (evidence: `vllm_fl/ops/fused_moe/router.py`)
- Rotary embedding → Dispatch API: dispatches rotary op (evidence: `vllm_fl/ops/rotary_embedding.py`)
- Model runner → Custom operators: uses operator adapters
- Model adaptations → Plugin hooks: registered by hooks (evidence: `vllm_fl/__init__.py`)
- Quantization → Accelerator: runs on
- Platform adapter → Accelerator: targets device (evidence: `README.md`)
- FlagCX communicator → Accelerator: communicates across devices (evidence: `README.md`)
- KV transfer connector → Transfer statistics: updates statistics (evidence: `vllm_fl/distributed/kv_transfer/flagcx_connector.py`)
- Worker → KV transfer connector: supports KV transfer (evidence: `README.md`)
