How vllm-project/speculators works
Speculators is an end-to-end framework for preparing hidden-state data, training speculative draft models, converting checkpoints, and deploying them with vLLM. The principal workflow runs from data generation or extracted hidden states through algorithm-specific training to a standardized checkpoint consumed by inference. The README also documents online response regeneration and multi-node hidden-state transfer. Sampled source and the SOURCE INDEX support core training/model/conversion relationships; unsampled feature links are limited to explicit README statements.
Main entry points: trainer.py Trainer · model.py Model loading and base classes · cli.py Training entrypoint · transfer.py State transfer
Components 23
Data preparation
- Training samples
src/speculators/data_generation/offline.py - Hidden-state generation
src/speculators/data_generation/vllm_client.py - Response regeneration
src/speculators/cli/regenerate_responses.py - State transfer
hs_connectors/src/hs_connectors/transfer.py - Mooncake store
hs_connectors/src/hs_connectors/mooncake_store.py
Training
- Training entrypoint
src/speculators/train/cli.py - Training configuration
src/speculators/train/config/schema.py - Training data loader
src/speculators/train/dataloader.py - Trainer
src/speculators/train/trainer.py - Distributed training
src/speculators/train/distributed.py - Training losses
src/speculators/losses/fused.py
Draft models
- DFlash and DSpark
src/speculators/models/dflash/core.py - EAGLE-3 and P-EAGLE
src/speculators/models/eagle3/core.py - MTP model
src/speculators/models/mtp/core.py - Other draft models
src/speculators/models - Greedy proposals
src/speculators/proposals/greedy.py
Formats and loading
- Speculator configuration
src/speculators/config.py - Model loading and base classes
src/speculators/model.py - Checkpoint conversion
src/speculators/convert/entrypoints.py - Standardized checkpoint
Inference integration
- Speculative inference
src/speculators/proposals/greedy.py
Other components
- Model developer
- vLLM service
Connections 33
- Model developer → Hidden-state generation: prepares data
- Hidden-state generation → vLLM service: requests hidden states
- Hidden-state generation → Training samples: saves samples
- Model developer → Response regeneration: regenerates responses
- Response regeneration → vLLM service: requests completions
src/speculators/cli/regenerate_responses.py - Model developer → Training entrypoint: starts training
- Training entrypoint → Training configuration: resolves config
src/speculators/train/cli.py - Training entrypoint → Training data loader: sets up data
src/speculators/train/cli.py - Training entrypoint → Trainer: runs training
src/speculators/train/cli.py - Trainer → Distributed training: uses distributed runtime
src/speculators/train/trainer.py - Trainer → Training losses: computes loss
- Trainer → DFlash and DSpark: trains selected model
- Trainer → EAGLE-3 and P-EAGLE: trains selected model
- Trainer → MTP model: trains selected model
- Trainer → Other draft models: trains supported models
- DFlash and DSpark → Model loading and base classes: extends
src/speculators/models/dflash/core.py - EAGLE-3 and P-EAGLE → Model loading and base classes: extends
src/speculators/models/eagle3/core.py - MTP model → Model loading and base classes: extends
src/speculators/models/mtp/core.py - DFlash and DSpark → Training losses: uses
src/speculators/models/dflash/core.py - EAGLE-3 and P-EAGLE → Training losses: uses
src/speculators/models/eagle3/core.py - MTP model → Greedy proposals: configures proposals
src/speculators/models/mtp/core.py - Model developer → Checkpoint conversion: converts checkpoint
- Checkpoint conversion → Standardized checkpoint: writes format
- Checkpoint conversion → Model loading and base classes: supports model loading
src/speculators/model.py - Speculator configuration → Greedy proposals: configures proposal
src/speculators/config.py - Model loading and base classes → Speculator configuration: loads config
src/speculators/model.py - Model loading and base classes → Standardized checkpoint: loads checkpoint
- Model developer → Speculative inference: deploys model
- Speculative inference → vLLM service: runs in engine
- Model developer → State transfer: enables online transfer
- State transfer → Mooncake store: uses backend
hs_connectors/src/hs_connectors/transfer.py - State transfer → Trainer: transfers hidden states
- vLLM service → State transfer: exchanges hidden states