How flagos-ai/vllm-plugin-fl works
vllm-plugin-FL extends vLLM inference with multi-chip platform support, model/runtime adaptations, backend-dispatched operators, and optional FlagCX communication. The principal workflow begins with a vLLM inference request, proceeds through the plugin worker and model runner, and uses platform-specific execution and operator implementations to produce model outputs. Dispatch selection, graph execution, model adaptations, quantization, and distributed KV transfer are distinct supporting capabilities. Some subsystem links are documented rather than visible in sampled implementations; those are marked as documented or unverified.
Main entry points: model_runner.py Model runner · worker.py Worker · __init__.py Dispatch API · manager.py Dispatch manager
Components 26
Runtime integration
- Plugin hooks
vllm_fl/__init__.py - Platform adapter
vllm_fl/platform.py
Inference execution
- Worker
vllm_fl/worker/worker.py - Model runner
vllm_fl/worker/model_runner.py - Attention metadata
vllm_fl/worker/common_attention_metadata.py - Scheduler adaptation
vllm_fl/worker/scheduler_fl.py - Graph wrapper
vllm_fl/compilation/graph.py
Operator dispatch
- Dispatch API
vllm_fl/dispatch/__init__.py - Dispatch manager
vllm_fl/dispatch/manager.py - Selection policy
vllm_fl/dispatch/policy.py - Operator registry
vllm_fl/dispatch/registry.py - Backend discovery
vllm_fl/dispatch/discovery.py - I/O diagnostics
vllm_fl/dispatch/io_dumper.py
Model and kernels
- FlagGems backend
vllm_fl/dispatch/backends/flaggems/flaggems.py - Reference backend
vllm_fl/dispatch/backends/reference/reference.py - Custom operators
vllm_fl/ops/custom_ops.py - Fused MoE routing
vllm_fl/ops/fused_moe/router.py - Rotary embedding
vllm_fl/ops/rotary_embedding.py - Quantization
vllm_fl/quantization/arm_cpu_w4a8.py - Model adaptations
vllm_fl/models/qwen3_5.py
Distributed runtime
- FlagCX communicator
vllm_fl/distributed/device_communicators/flagcx.py - KV transfer connector
vllm_fl/distributed/kv_transfer/flagcx_connector.py - Transfer statistics
vllm_fl/distributed/kv_transfer/flagcx_stats.py
Other components
- vLLM caller
- Accelerator
- Generated outputs
Connections 28
- vLLM caller → Plugin hooks: loads plugin
README.md - Plugin hooks → Platform adapter: registers platform
vllm_fl/__init__.py - vLLM caller → Worker: submits inference
README.md - Worker → Model runner: executes model
vllm_fl/worker/worker.py - Model runner → Generated outputs: returns outputs
README.md - Worker → Platform adapter: uses device adapter
- Model runner → Attention metadata: uses metadata
vllm_fl/worker/model_runner.py - Model runner → Graph wrapper: uses graph wrapper
vllm_fl/worker/model_runner.py - Model runner → Dispatch API: records I/O
vllm_fl/worker/model_runner.py - Worker → Scheduler adaptation: uses scheduling support
- Dispatch API → Dispatch manager: resolves calls
vllm_fl/dispatch/__init__.py - Dispatch manager → Operator registry: queries implementations
vllm_fl/dispatch/manager.py - Dispatch manager → Selection policy: selects by policy
vllm_fl/dispatch/manager.py - Dispatch manager → I/O diagnostics: routes I/O hooks
vllm_fl/dispatch/manager.py - Dispatch API → Backend discovery: exposes discovery
vllm_fl/dispatch/__init__.py - Operator registry → FlagGems backend: registers implementations
- Operator registry → Reference backend: registers implementations
- Custom operators → Fused MoE routing: uses MoE operators
vllm_fl/ops/custom_ops.py - Custom operators → Rotary embedding: uses rotary operator
vllm_fl/ops/custom_ops.py - Fused MoE routing → Dispatch API: dispatches routing op
vllm_fl/ops/fused_moe/router.py - Rotary embedding → Dispatch API: dispatches rotary op
vllm_fl/ops/rotary_embedding.py - Model runner → Custom operators: uses operator adapters
- Model adaptations → Plugin hooks: registered by hooks
vllm_fl/__init__.py - Quantization → Accelerator: runs on
- Platform adapter → Accelerator: targets device
README.md - FlagCX communicator → Accelerator: communicates across devices
README.md - KV transfer connector → Transfer statistics: updates statistics
vllm_fl/distributed/kv_transfer/flagcx_connector.py - Worker → KV transfer connector: supports KV transfer
README.md