How w2315516642/vllm-plugin-fl works
vllm-plugin-FL extends vLLM inference across heterogeneous accelerator hardware while retaining vLLM’s user-facing workflow. A request enters through vLLM’s worker integration, is prepared and executed by the model runner, and uses platform-specific execution, graph support, and operator adapters before producing inference output. Operator dispatch selects implementations under backend policy, with diagnostics available for operator calls. The repository also includes distributed communication and KV transfer, model-specific integrations, and quantization. Edges rely on README statements and SOURCE INDEX references; unsampled implementation details are not inferred.
Main entry points: manager.py Dispatch manager · model_runner.py Model execution · io_dumper.py Operator diagnostics · worker.py Worker lifecycle
Components 24
Inference runtime
- Worker lifecycle
vllm_fl/worker/worker.py - Model execution
vllm_fl/worker/model_runner.py - Attention metadata
vllm_fl/worker/common_attention_metadata.py - Graph execution
vllm_fl/compilation/graph.py
Operator dispatch
- Dispatch manager
vllm_fl/dispatch/manager.py - Backend policy
vllm_fl/dispatch/policy.py - Implementation registry
vllm_fl/dispatch/registry.py - Plugin discovery
vllm_fl/dispatch/discovery.py - Operator diagnostics
vllm_fl/dispatch/io_dumper.py - Diagnostic utilities
vllm_fl/dispatch/io_common.py - Dispatch logging
vllm_fl/dispatch/logger_manager.py
Kernels and models
- FlagGems backend
vllm_fl/dispatch/backends/flaggems/impl/attention.py - Reference backend
vllm_fl/dispatch/backends/reference/reference.py - Custom operators
vllm_fl/ops/custom_ops.py - MoE routing
vllm_fl/ops/fused_moe/router.py - MoE utilities
vllm_fl/ops/fused_moe/fused_moe_utils.py - Rotary embedding
vllm_fl/ops/rotary_embedding.py - Model integration
vllm_fl/models/qwen3_5.py - Quantization
vllm_fl/quantization/arm_cpu_w4a8.py
Hardware and distributed
- Platform integration
vllm_fl/platform.py - FlagCX communicator
vllm_fl/distributed/device_communicators/flagcx.py - KV transfer
vllm_fl/distributed/kv_transfer/flagcx_connector.py - Transfer statistics
vllm_fl/distributed/kv_transfer/flagcx_stats.py
Other components
- vLLM caller
Connections 20
- vLLM caller → Worker lifecycle: submits inference
README.md - Worker lifecycle → Model execution: executes model
vllm_fl/worker/worker.py - Model execution → Attention metadata: uses metadata
vllm_fl/worker/model_runner.py - Model execution → Graph execution: uses graph support
vllm_fl/worker/model_runner.py - Model execution → Platform integration: queries platform
vllm_fl/worker/model_runner.py - Model execution → Operator diagnostics: uses diagnostics
vllm_fl/worker/model_runner.py - Worker lifecycle → Custom operators: uses operators
vllm_fl/worker/worker.py - Custom operators → MoE utilities: imports utilities
vllm_fl/ops/custom_ops.py - MoE routing → Dispatch manager: dispatches operators
vllm_fl/ops/fused_moe/router.py - Rotary embedding → Dispatch manager: dispatches operator
vllm_fl/ops/rotary_embedding.py - Dispatch manager → Implementation registry: queries implementations
vllm_fl/dispatch/manager.py - Dispatch manager → Backend policy: applies policy
vllm_fl/dispatch/manager.py - Dispatch manager → Operator diagnostics: routes diagnostics
vllm_fl/dispatch/manager.py - Operator diagnostics → Diagnostic utilities: uses utilities
vllm_fl/dispatch/io_dumper.py - Operator diagnostics → Dispatch logging: uses logging
vllm_fl/dispatch/io_dumper.py - Plugin discovery → Dispatch manager: registers plugins
- Dispatch manager → FlagGems backend: selects backend
README.md - Dispatch manager → Reference backend: supports fallback
README.md - Platform integration → FlagCX communicator: uses FlagCX
README.md - KV transfer → Transfer statistics: records statistics
vllm_fl/distributed/kv_transfer/flagcx_connector.py