How deepspeedai/deepspeed works
Purpose and public entry points
DeepSpeed is a Python/PyTorch systems library for distributed deep-learning training and inference, emphasizing memory partitioning/offload, parallelism, optimized kernels, and large-model serving. Primary user entry points are the deepspeed package (deepspeed/__init__.py), distributed CLI launchers (bin/deepspeed, deepspeed/launcher/runner.py), and environment capability reporting (deepspeed/env_report.py).
Initialization, configuration, and engine control
User models, optimizers, dataloaders, and JSON/dict configuration enter through package initialization and are materialized into the training runtime engine. The engine owns forward/backward/step orchestration, precision, optimizer integration, checkpointing, and optional runtime features. Configuration is a key boundary between application code and DeepSpeed-managed execution.
• deepspeed/runtime/engine.py
• deepspeed/runtime/config.py
• deepspeed/runtime/precision_config.py
Distributed launch and communication
The launcher creates local or multinode worker processes; each process initializes collective communication and process-group abstractions. Runtime communication selects or wraps PyTorch distributed backends, including NCCL, MPI, HCCL, CCL, and compressed/coalesced collectives. This is the boundary to cluster networking and distributed accelerator execution.
• deepspeed/launcher/launch.py
• deepspeed/launcher/multinode_runner.py
• deepspeed/comm/comm.py
• deepspeed/runtime/comm/nccl.py
Memory scaling and parallel training
ZeRO is the central training-memory subsystem: stages 1/2 partition optimizer state and gradients; stage 3 partitions parameters and coordinates their materialization. Offload paths transfer states to CPU/NVMe, while pipeline, tensor, and sequence-parallel subsystems divide model computation across ranks. These mechanisms are composed by the runtime engine and rely on collectives.
• deepspeed/runtime/zero/stage_1_and_2.py
• deepspeed/runtime/zero/stage3.py
• deepspeed/runtime/zero/parameter_offload.py
• deepspeed/runtime/pipe/engine.py
• deepspeed/runtime/tensor_parallel/tp_manager.py
Checkpoint and storage paths
Checkpoint abstractions persist or restore model/training state, including universal checkpoint reshaping across distributed layouts. NVMe/AIO and swap-tensor components support asynchronous storage movement for offload-heavy workloads; GPU Direct Storage has a separate native binding path.
• deepspeed/runtime/checkpoint_engine/checkpoint_engine.py
• deepspeed/checkpoint/universal_checkpoint.py
• deepspeed/runtime/swap_tensor/async_swapper.py
• deepspeed/ops/aio/__init__.py
Inference and model transformation
Inference has a conventional engine plus a v2 architecture. V2 creates an engine from config/checkpoint adapters, maps supported model families into DeepSpeed implementations, selects modules through registries/heuristics, and executes fused/ragged kernels with KV-cache management. Module injection rewrites external PyTorch/Transformers-style models for tensor parallelism or optimized inference.
• deepspeed/inference/engine.py
• deepspeed/inference/v2/engine_factory.py
• deepspeed/inference/v2/engine_v2.py
• deepspeed/inference/v2/ragged/kv_cache.py
• deepspeed/module_inject/replace_module.py
Native operations and accelerator portability
Python operation facades dynamically load or build C++/CUDA/HIP extensions through op builders; source kernels implement fused optimizers, transformer inference, quantization, AIO, and specialized science kernels. The accelerator abstraction selects CPU, CUDA, HPU, MPS, NPU, MLU, SDAA, SUPA, or XPU behavior, insulating most runtime code from device APIs. JIT builds use PyTorch’s extension loader and require toolchains such as Ninja plus nvcc or hipcc.
• op_builder/builder.py
• op_builder/all_ops.py
• accelerator/real_accelerator.py
• csrc/transformer/ds_transformer_cuda.cpp
Supporting integrations and infrastructure
DeepSpeed depends materially on Python, PyTorch, PyTorch distributed, and vendor accelerator runtimes; CUDA/ROCm are primary GPU targets. Optional external integrations include Hugging Face ecosystems, Lightning, TensorBoard, Weights & Biases, and Comet monitoring. CI spans NVIDIA, AMD, CPU, Gaudi, and XPU runners, with GPU CI support from Modal.
• deepspeed/monitor/wandb.py
• deepspeed/monitor/tensorboard.py
• .github/workflows/nv-torch-latest-v100.yml
• docker/Dockerfile
Main entry points: engine.py Training engine · comm.py Collective communication · __init__.py DeepSpeed API · engine_v2.py Inference v2 engine
Components 22
Entry & Control
- DeepSpeed API · Public initialization entry point.
deepspeed/__init__.py - Distributed launcher · Creates local and multinode workers.
deepspeed/launcher/launch.py - Runtime config · Maps user JSON or dict settings into runtime policy.
deepspeed/runtime/config.py
Distributed Training
- Training engine · Orchestrates forward, backward, stepping, precision, and runtime features.
deepspeed/runtime/engine.py - Collective communication · Wraps distributed backends and collectives.
deepspeed/comm/comm.py - NCCL runtime comm
deepspeed/runtime/comm/nccl.py - ZeRO stages 1/2 · Partitions optimizer state and gradients.
deepspeed/runtime/zero/stage_1_and_2.py - ZeRO stage 3 · Partitions and materializes parameters across ranks.
deepspeed/runtime/zero/stage3.py - Parameter offload · Moves model state through CPU or NVMe paths.
deepspeed/runtime/zero/parameter_offload.py - Pipeline and tensor parallel
deepspeed/runtime/pipe/engine.py
State & Storage
- Checkpoint engine · Persists and restores distributed training state.
deepspeed/runtime/checkpoint_engine/checkpoint_engine.py - Universal checkpoint · Adapts checkpoints across distributed layouts.
deepspeed/checkpoint/universal_checkpoint.py - Async swapper · Coordinates asynchronous tensor storage movement.
deepspeed/runtime/swap_tensor/async_swapper.py - AIO operations
deepspeed/ops/aio/__init__.py
Inference
- Inference engine · Conventional optimized inference execution.
deepspeed/inference/engine.py - Inference v2 engine · Builds model-family-specific optimized execution.
deepspeed/inference/v2/engine_v2.py - Ragged KV cache
deepspeed/inference/v2/ragged/kv_cache.py - Module injection · Rewrites external models for optimized or tensor-parallel inference.
deepspeed/module_inject/replace_module.py
Native & Devices
- Operation builders · Dynamically builds or loads native extensions.
op_builder/builder.py - Native kernels · Implements fused transformer and other accelerated operations.
csrc/transformer/ds_transformer_cuda.cpp - Accelerator abstraction · Selects device-specific behavior across supported accelerators.
accelerator/real_accelerator.py
Other components
- PyTorch distributed · Provides process groups and backend implementations.
Connections 27
- DeepSpeed API → Runtime config: accepts
- DeepSpeed API → Training engine: initializes
- Distributed launcher → PyTorch distributed: starts workers for
- Training engine → Runtime config: applies
- Training engine → Collective communication: uses
- Collective communication → PyTorch distributed: wraps
- Collective communication → NCCL runtime comm: selects
- Training engine → ZeRO stages 1/2: composes
- Training engine → ZeRO stage 3: composes
- Training engine → Pipeline and tensor parallel: runs
- ZeRO stages 1/2 → Collective communication: synchronizes through
- ZeRO stage 3 → Collective communication: materializes through
- ZeRO stage 3 → Parameter offload: offloads state via
- Parameter offload → Async swapper: moves tensors with
- Async swapper → AIO operations: uses
- Training engine → Checkpoint engine: saves and restores
- Checkpoint engine → Universal checkpoint: supports
- DeepSpeed API → Inference engine: initializes
- DeepSpeed API → Inference v2 engine: initializes
- Inference engine → Module injection: optimizes models with
- Inference v2 engine → Ragged KV cache: manages
- Inference v2 engine → Module injection: transforms models with
- Training engine → Operation builders: loads operations from
- Inference v2 engine → Operation builders: loads kernels from
- Operation builders → Native kernels: builds or loads
- Training engine → Accelerator abstraction: targets
- Operation builders → Accelerator abstraction: builds for