How microsoft/deepspeed works
DeepSpeed is a framework for scaling deep-learning training and inference. The principal workflow is a model developer configuring distributed training, with the runtime engine coordinating optimization and parallel execution; users can also load models into inference engines and submit token batches for generated outputs. Major branches include ZeRO partitioning, pipeline parallelism, MoE, compilation, checkpoint conversion, and NVMe I/O. The graph uses the provided README and source index; unsampled implementations and wiring are represented only where documented or indexed, so some relationships are intentionally omitted.
Main entry points: engine.py Training Engine · engine_v2.py Inference V2 Engine · comm.py Communication API · torch.py Torch Backend
Components 23
Training Runtime
- Training Engine
deepspeed/runtime/engine.py - ZeRO Sharding
deepspeed/runtime/zero - Precision Optimizers
deepspeed/runtime/bf16_optimizer.py - Pipeline Engine
deepspeed/runtime/pipe/engine.py - Checkpointing
deepspeed/checkpoint
Distributed Execution
- Communication API
deepspeed/comm/comm.py - Torch Backend
deepspeed/comm/torch.py - Parallel Groups
deepspeed/utils/groups.py - MoE Routing
deepspeed/moe/ep_router.py - DeepCompile
deepspeed/compile
Inference
- Inference Engine
deepspeed/inference/engine.py - Inference V2 Factory
deepspeed/inference/v2/engine_factory.py - Inference V2 Engine
deepspeed/inference/v2/engine_v2.py - Model Implementations
deepspeed/inference/v2/model_implementations - Ragged Sequence State
deepspeed/inference/v2/ragged/ragged_manager.py - Inference Kernels
deepspeed/inference/v2/kernels
Systems Features
- NVMe I/O
deepspeed/nvme/io_engine.py - Fast File Writer
deepspeed/io/fast_file_writer.py - Autotuning
deepspeed/autotuning/autotuner.py - Accelerator Support
accelerator - Science Attention
csrc/deepspeed4science/evoformer_attn/attention.cpp
Other components
- Model Developer
- Inference Client
Connections 22
- Model Developer → Training Engine: configures training
- Training Engine → ZeRO Sharding: uses partitioning
deepspeed/runtime/engine.py - Training Engine → Precision Optimizers: uses optimizer
deepspeed/runtime/engine.py - Training Engine → Communication API: uses communication
deepspeed/runtime/engine.py - Training Engine → Pipeline Engine: supports pipeline
deepspeed/runtime/pipe/engine.py - Training Engine → Checkpointing: loads and saves
deepspeed/runtime/engine.py - Communication API → Torch Backend: dispatches operations
deepspeed/comm/comm.py - Torch Backend → Parallel Groups: uses utilities
deepspeed/comm/torch.py - Training Engine → MoE Routing: supports MoE
deepspeed/runtime/engine.py - Model Developer → DeepCompile: compiles training
- Inference Client → Inference Engine: runs inference
- Inference Client → Inference V2 Factory: creates engine
- Inference V2 Factory → Inference V2 Engine: constructs engine
deepspeed/inference/v2/engine_factory.py - Inference V2 Engine → Model Implementations: builds model
deepspeed/inference/v2/engine_v2.py - Inference V2 Engine → Ragged Sequence State: manages sequences
deepspeed/inference/v2/engine_v2.py - Inference V2 Engine → Communication API: initializes distributed
deepspeed/inference/v2/engine_v2.py - Inference Engine → Accelerator Support: uses device APIs
deepspeed/inference/engine.py - Inference Client → Inference V2 Engine: submits token batches
deepspeed/inference/v2/engine_v2.py - Inference V2 Engine → Inference Kernels: executes model kernels
- NVMe I/O → Fast File Writer: writes through
- Training Engine → Autotuning: collects model data
- Model Developer → Science Attention: uses research kernels