# deepspeedai/deepspeed architecture

> Architecture diagram of the GitHub repository deepspeedai/deepspeed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deepspeedai/deepspeed
- Repository: https://github.com/deepspeedai/deepspeed

## Overview

#### Purpose and public entry points
DeepSpeed is a Python/PyTorch systems library for distributed deep-learning training and inference, emphasizing memory partitioning/offload, parallelism, optimized kernels, and large-model serving. Primary user entry points are the `deepspeed` package (`deepspeed/__init__.py`), distributed CLI launchers (`bin/deepspeed`, `deepspeed/launcher/runner.py`), and environment capability reporting (`deepspeed/env_report.py`).

#### Initialization, configuration, and engine control
User models, optimizers, dataloaders, and JSON/dict configuration enter through package initialization and are materialized into the training runtime engine. The engine owns forward/backward/step orchestration, precision, optimizer integration, checkpointing, and optional runtime features. Configuration is a key boundary between application code and DeepSpeed-managed execution.
- `deepspeed/runtime/engine.py`
- `deepspeed/runtime/config.py`
- `deepspeed/runtime/precision_config.py`

#### Distributed launch and communication
The launcher creates local or multinode worker processes; each process initializes collective communication and process-group abstractions. Runtime communication selects or wraps PyTorch distributed backends, including NCCL, MPI, HCCL, CCL, and compressed/coalesced collectives. This is the boundary to cluster networking and distributed accelerator execution.
- `deepspeed/launcher/launch.py`
- `deepspeed/launcher/multinode_runner.py`
- `deepspeed/comm/comm.py`
- `deepspeed/runtime/comm/nccl.py`

#### Memory scaling and parallel training
ZeRO is the central training-memory subsystem: stages 1/2 partition optimizer state and gradients; stage 3 partitions parameters and coordinates their materialization. Offload paths transfer states to CPU/NVMe, while pipeline, tensor, and sequence-parallel subsystems divide model computation across ranks. These mechanisms are composed by the runtime engine and rely on collectives.
- `deepspeed/runtime/zero/stage_1_and_2.py`
- `deepspeed/runtime/zero/stage3.py`
- `deepspeed/runtime/zero/parameter_offload.py`
- `deepspeed/runtime/pipe/engine.py`
- `deepspeed/runtime/tensor_parallel/tp_manager.py`

#### Checkpoint and storage paths
Checkpoint abstractions persist or restore model/training state, including universal checkpoint reshaping across distributed layouts. NVMe/AIO and swap-tensor components support asynchronous storage movement for offload-heavy workloads; GPU Direct Storage has a separate native binding path.
- `deepspeed/runtime/checkpoint_engine/checkpoint_engine.py`
- `deepspeed/checkpoint/universal_checkpoint.py`
- `deepspeed/runtime/swap_tensor/async_swapper.py`
- `deepspeed/ops/aio/__init__.py`

#### Inference and model transformation
Inference has a conventional engine plus a v2 architecture. V2 creates an engine from config/checkpoint adapters, maps supported model families into DeepSpeed implementations, selects modules through registries/heuristics, and executes fused/ragged kernels with KV-cache management. Module injection rewrites external PyTorch/Transformers-style models for tensor parallelism or optimized inference.
- `deepspeed/inference/engine.py`
- `deepspeed/inference/v2/engine_factory.py`
- `deepspeed/inference/v2/engine_v2.py`
- `deepspeed/inference/v2/ragged/kv_cache.py`
- `deepspeed/module_inject/replace_module.py`

#### Native operations and accelerator portability
Python operation facades dynamically load or build C++/CUDA/HIP extensions through op builders; source kernels implement fused optimizers, transformer inference, quantization, AIO, and specialized science kernels. The accelerator abstraction selects CPU, CUDA, HPU, MPS, NPU, MLU, SDAA, SUPA, or XPU behavior, insulating most runtime code from device APIs. JIT builds use PyTorch’s extension loader and require toolchains such as Ninja plus `nvcc` or `hipcc`.
- `op_builder/builder.py`
- `op_builder/all_ops.py`
- `accelerator/real_accelerator.py`
- `csrc/transformer/ds_transformer_cuda.cpp`

#### Supporting integrations and infrastructure
DeepSpeed depends materially on Python, PyTorch, PyTorch distributed, and vendor accelerator runtimes; CUDA/ROCm are primary GPU targets. Optional external integrations include Hugging Face ecosystems, Lightning, TensorBoard, Weights & Biases, and Comet monitoring. CI spans NVIDIA, AMD, CPU, Gaudi, and XPU runners, with GPU CI support from Modal.
- `deepspeed/monitor/wandb.py`
- `deepspeed/monitor/tensorboard.py`
- `.github/workflows/nv-torch-latest-v100.yml`
- `docker/Dockerfile`

Main entry points: [`deepspeed/runtime/engine.py`](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/engine.py) (Training engine), [`deepspeed/comm/comm.py`](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/comm/comm.py) (Collective communication), [`deepspeed/__init__.py`](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/__init__.py) (DeepSpeed API), [`deepspeed/inference/v2/engine_v2.py`](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/v2/engine_v2.py) (Inference v2 engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Entry &amp; Control"]
  node_api["DeepSpeed API<br/>Python package<br/>[__init__.py]"]
  node_launcher["Distributed launcher<br/>process launcher<br/>[launch.py]"]
  node_config["Runtime config<br/>configuration<br/>[config.py]"]
end

subgraph group_training["Distributed Training"]
  node_engine["Training engine<br/>runtime engine<br/>[engine.py]"]
  node_comm["Collective communication<br/>distributed communication<br/>[comm.py]"]
  node_nccl["NCCL runtime comm<br/>backend integration<br/>[nccl.py]"]
  node_zero12["ZeRO stages 1/2<br/>memory partitioning<br/>[stage_1_and_2.py]"]
  node_zero3["ZeRO stage 3<br/>parameter partitioning<br/>[stage3.py]"]
  node_offload["Parameter offload<br/>memory offload"]
  node_parallel["Pipeline and tensor parallel<br/>parallel execution<br/>[engine.py]"]
end

subgraph group_storage["State &amp; Storage"]
  node_checkpoint[("Checkpoint engine<br/>checkpoint abstraction")]
  node_universal_ckpt["Universal checkpoint<br/>checkpoint reshaping"]
  node_async_swap["Async swapper<br/>NVMe transfer<br/>[async_swapper.py]"]
  node_aio["AIO operations<br/>native storage facade<br/>[__init__.py]"]
end

subgraph group_inference["Inference"]
  node_inference["Inference engine<br/>inference runtime<br/>[engine.py]"]
  node_inference_v2["Inference v2 engine<br/>inference runtime<br/>[engine_v2.py]"]
  node_kv_cache["Ragged KV cache<br/>inference state<br/>[kv_cache.py]"]
  node_module_inject["Module injection<br/>model transformation<br/>[replace_module.py]"]
end

subgraph group_native["Native &amp; Devices"]
  node_op_builders["Operation builders<br/>extension loader<br/>[builder.py]"]
  node_native_kernels{{"Native kernels<br/>C++/CUDA kernels"}}
  node_accelerator["Accelerator abstraction<br/>device portability"]
end

node_torch_dist(("PyTorch distributed<br/>external runtime"))

node_api -->|"accepts"| node_config
node_api -->|"initializes"| node_engine
node_launcher -->|"starts workers for"| node_torch_dist
node_engine -->|"applies"| node_config
node_engine -->|"uses"| node_comm
node_comm -->|"wraps"| node_torch_dist
node_comm -->|"selects"| node_nccl
node_engine -->|"composes"| node_zero12
node_engine -->|"composes"| node_zero3
node_engine -->|"runs"| node_parallel
node_zero12 -->|"synchronizes through"| node_comm
node_zero3 -->|"materializes through"| node_comm
node_zero3 -->|"offloads state via"| node_offload
node_offload -->|"moves tensors with"| node_async_swap
node_async_swap -->|"uses"| node_aio
node_engine -->|"saves and restores"| node_checkpoint
node_checkpoint -->|"supports"| node_universal_ckpt
node_api -->|"initializes"| node_inference
node_api -->|"initializes"| node_inference_v2
node_inference -->|"optimizes models with"| node_module_inject
node_inference_v2 -->|"manages"| node_kv_cache
node_inference_v2 -->|"transforms models with"| node_module_inject
node_engine -->|"loads operations from"| node_op_builders
node_inference_v2 -->|"loads kernels from"| node_op_builders
node_op_builders -->|"builds or loads"| node_native_kernels
node_engine -->|"targets"| node_accelerator
node_op_builders -->|"builds for"| node_accelerator

click node_api "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/__init__.py"
click node_launcher "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/launcher/launch.py"
click node_config "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/config.py"
click node_engine "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/engine.py"
click node_comm "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/comm/comm.py"
click node_nccl "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/comm/nccl.py"
click node_zero12 "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/zero/stage_1_and_2.py"
click node_zero3 "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/zero/stage3.py"
click node_offload "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/zero/parameter_offload.py"
click node_parallel "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/pipe/engine.py"
click node_checkpoint "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/checkpoint_engine/checkpoint_engine.py"
click node_universal_ckpt "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/checkpoint/universal_checkpoint.py"
click node_async_swap "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/swap_tensor/async_swapper.py"
click node_aio "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/ops/aio/__init__.py"
click node_inference "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/engine.py"
click node_inference_v2 "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/v2/engine_v2.py"
click node_kv_cache "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/v2/ragged/kv_cache.py"
click node_module_inject "https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/module_inject/replace_module.py"
click node_op_builders "https://github.com/deepspeedai/deepspeed/blob/master/op_builder/builder.py"
click node_native_kernels "https://github.com/deepspeedai/deepspeed/blob/master/csrc/transformer/ds_transformer_cuda.cpp"
click node_accelerator "https://github.com/deepspeedai/deepspeed/blob/master/accelerator/real_accelerator.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_launcher,node_config toneBlue
class node_engine,node_comm,node_nccl,node_zero12,node_zero3,node_offload,node_parallel toneAmber
class node_checkpoint,node_universal_ckpt,node_async_swap,node_aio toneMint
class node_inference,node_inference_v2,node_kv_cache,node_module_inject toneRose
class node_op_builders,node_native_kernels,node_accelerator toneIndigo
class node_torch_dist toneNeutral
```

## Components

### Entry & Control

- [DeepSpeed API](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/__init__.py): Python package · Public initialization entry point. · `deepspeed/__init__.py`
- [Distributed launcher](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/launcher/launch.py): process launcher · Creates local and multinode workers. · `deepspeed/launcher/launch.py`
- [Runtime config](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/config.py): configuration · Maps user JSON or dict settings into runtime policy. · `deepspeed/runtime/config.py`

### Distributed Training

- [Training engine](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/engine.py): runtime engine · Orchestrates forward, backward, stepping, precision, and runtime features. · `deepspeed/runtime/engine.py`
- [Collective communication](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/comm/comm.py): distributed communication · Wraps distributed backends and collectives. · `deepspeed/comm/comm.py`
- [NCCL runtime comm](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/comm/nccl.py): backend integration · `deepspeed/runtime/comm/nccl.py`
- [ZeRO stages 1/2](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/zero/stage_1_and_2.py): memory partitioning · Partitions optimizer state and gradients. · `deepspeed/runtime/zero/stage_1_and_2.py`
- [ZeRO stage 3](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/zero/stage3.py): parameter partitioning · Partitions and materializes parameters across ranks. · `deepspeed/runtime/zero/stage3.py`
- [Parameter offload](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/zero/parameter_offload.py): memory offload · Moves model state through CPU or NVMe paths. · `deepspeed/runtime/zero/parameter_offload.py`
- [Pipeline and tensor parallel](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/pipe/engine.py): parallel execution · `deepspeed/runtime/pipe/engine.py`

### State & Storage

- [Checkpoint engine](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/checkpoint_engine/checkpoint_engine.py): checkpoint abstraction · Persists and restores distributed training state. · `deepspeed/runtime/checkpoint_engine/checkpoint_engine.py`
- [Universal checkpoint](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/checkpoint/universal_checkpoint.py): checkpoint reshaping · Adapts checkpoints across distributed layouts. · `deepspeed/checkpoint/universal_checkpoint.py`
- [Async swapper](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/runtime/swap_tensor/async_swapper.py): NVMe transfer · Coordinates asynchronous tensor storage movement. · `deepspeed/runtime/swap_tensor/async_swapper.py`
- [AIO operations](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/ops/aio/__init__.py): native storage facade · `deepspeed/ops/aio/__init__.py`

### Inference

- [Inference engine](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/engine.py): inference runtime · Conventional optimized inference execution. · `deepspeed/inference/engine.py`
- [Inference v2 engine](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/v2/engine_v2.py): inference runtime · Builds model-family-specific optimized execution. · `deepspeed/inference/v2/engine_v2.py`
- [Ragged KV cache](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/inference/v2/ragged/kv_cache.py): inference state · `deepspeed/inference/v2/ragged/kv_cache.py`
- [Module injection](https://github.com/deepspeedai/deepspeed/blob/master/deepspeed/module_inject/replace_module.py): model transformation · Rewrites external models for optimized or tensor-parallel inference. · `deepspeed/module_inject/replace_module.py`

### Native & Devices

- [Operation builders](https://github.com/deepspeedai/deepspeed/blob/master/op_builder/builder.py): extension loader · Dynamically builds or loads native extensions. · `op_builder/builder.py`
- [Native kernels](https://github.com/deepspeedai/deepspeed/blob/master/csrc/transformer/ds_transformer_cuda.cpp): C++/CUDA kernels · Implements fused transformer and other accelerated operations. · `csrc/transformer/ds_transformer_cuda.cpp`
- [Accelerator abstraction](https://github.com/deepspeedai/deepspeed/blob/master/accelerator/real_accelerator.py): device portability · Selects device-specific behavior across supported accelerators. · `accelerator/real_accelerator.py`

### Other components

- **PyTorch distributed**: external runtime · Provides process groups and backend implementations.

## Connections

- DeepSpeed API → Runtime config: accepts
- DeepSpeed API → Training engine: initializes
- Distributed launcher → PyTorch distributed: starts workers for
- Training engine → Runtime config: applies
- Training engine → Collective communication: uses
- Collective communication → PyTorch distributed: wraps
- Collective communication → NCCL runtime comm: selects
- Training engine → ZeRO stages 1/2: composes
- Training engine → ZeRO stage 3: composes
- Training engine → Pipeline and tensor parallel: runs
- ZeRO stages 1/2 → Collective communication: synchronizes through
- ZeRO stage 3 → Collective communication: materializes through
- ZeRO stage 3 → Parameter offload: offloads state via
- Parameter offload → Async swapper: moves tensors with
- Async swapper → AIO operations: uses
- Training engine → Checkpoint engine: saves and restores
- Checkpoint engine → Universal checkpoint: supports
- DeepSpeed API → Inference engine: initializes
- DeepSpeed API → Inference v2 engine: initializes
- Inference engine → Module injection: optimizes models with
- Inference v2 engine → Ragged KV cache: manages
- Inference v2 engine → Module injection: transforms models with
- Training engine → Operation builders: loads operations from
- Inference v2 engine → Operation builders: loads kernels from
- Operation builders → Native kernels: builds or loads
- Training engine → Accelerator abstraction: targets
- Operation builders → Accelerator abstraction: builds for
