# microsoft/deepspeed architecture

> Architecture diagram of the GitHub repository microsoft/deepspeed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/deepspeed
- Repository: https://github.com/microsoft/deepspeed

## Overview

DeepSpeed is a framework for scaling deep-learning training and inference. The principal workflow is a model developer configuring distributed training, with the runtime engine coordinating optimization and parallel execution; users can also load models into inference engines and submit token batches for generated outputs. Major branches include ZeRO partitioning, pipeline parallelism, MoE, compilation, checkpoint conversion, and NVMe I/O. The graph uses the provided README and source index; unsampled implementations and wiring are represented only where documented or indexed, so some relationships are intentionally omitted.

Main entry points: [`deepspeed/runtime/engine.py`](https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/engine.py) (Training Engine), [`deepspeed/inference/v2/engine_v2.py`](https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/engine_v2.py) (Inference V2 Engine), [`deepspeed/comm/comm.py`](https://github.com/microsoft/deepspeed/blob/master/deepspeed/comm/comm.py) (Communication API), [`deepspeed/comm/torch.py`](https://github.com/microsoft/deepspeed/blob/master/deepspeed/comm/torch.py) (Torch Backend)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_training["Training Runtime"]
  node_training_engine["Training Engine<br/>[engine.py]"]
  node_zero["ZeRO Sharding"]
  node_optimizers["Precision Optimizers<br/>[bf16_optimizer.py]"]
  node_pipeline["Pipeline Engine<br/>[engine.py]"]
  node_checkpoint["Checkpointing"]
end

subgraph group_distributed["Distributed Execution"]
  node_communication["Communication API<br/>[comm.py]"]
  node_torch_backend["Torch Backend<br/>[torch.py]"]
  node_groups["Parallel Groups<br/>[groups.py]"]
  node_moe["MoE Routing<br/>[ep_router.py]"]
  node_compile["DeepCompile"]
end

subgraph group_inference["Inference"]
  node_inference_engine["Inference Engine<br/>[engine.py]"]
  node_v2_factory["Inference V2 Factory<br/>[engine_factory.py]"]
  node_v2_engine["Inference V2 Engine<br/>[engine_v2.py]"]
  node_model_impl["Model Implementations"]
  node_ragged_state["Ragged Sequence State<br/>[ragged_manager.py]"]
  node_inference_kernels["Inference Kernels"]
end

subgraph group_systems["Systems Features"]
  node_nvme_io["NVMe I/O<br/>[io_engine.py]"]
  node_fast_writer["Fast File Writer"]
  node_autotuning["Autotuning<br/>[autotuner.py]"]
  node_accelerators["Accelerator Support"]
  node_science_attention["Science Attention<br/>[attention.cpp]"]
end

node_developer(("Model Developer"))
node_inference_user(("Inference Client"))

node_developer -->|"configures training"| node_training_engine
node_training_engine -->|"uses partitioning"| node_zero
node_training_engine -->|"uses optimizer"| node_optimizers
node_training_engine -->|"uses communication"| node_communication
node_training_engine -->|"supports pipeline"| node_pipeline
node_training_engine -->|"loads and saves"| node_checkpoint
node_communication -->|"dispatches operations"| node_torch_backend
node_torch_backend -->|"uses utilities"| node_groups
node_training_engine -->|"supports MoE"| node_moe
node_developer -->|"compiles training"| node_compile
node_inference_user -->|"runs inference"| node_inference_engine
node_inference_user -.->|"creates engine"| node_v2_factory
node_v2_factory -->|"constructs engine"| node_v2_engine
node_v2_engine -->|"builds model"| node_model_impl
node_v2_engine -->|"manages sequences"| node_ragged_state
node_v2_engine -->|"initializes distributed"| node_communication
node_inference_engine -->|"uses device APIs"| node_accelerators
node_inference_user -->|"submits token batches"| node_v2_engine
node_v2_engine -.->|"executes model kernels"| node_inference_kernels
node_nvme_io -->|"writes through"| node_fast_writer
node_training_engine -.->|"collects model data"| node_autotuning
node_developer -->|"uses research kernels"| node_science_attention

click node_training_engine "https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/engine.py"
click node_zero "https://github.com/microsoft/deepspeed/tree/master/deepspeed/runtime/zero"
click node_optimizers "https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/bf16_optimizer.py"
click node_pipeline "https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/pipe/engine.py"
click node_checkpoint "https://github.com/microsoft/deepspeed/tree/master/deepspeed/checkpoint"
click node_communication "https://github.com/microsoft/deepspeed/blob/master/deepspeed/comm/comm.py"
click node_torch_backend "https://github.com/microsoft/deepspeed/blob/master/deepspeed/comm/torch.py"
click node_groups "https://github.com/microsoft/deepspeed/blob/master/deepspeed/utils/groups.py"
click node_moe "https://github.com/microsoft/deepspeed/blob/master/deepspeed/moe/ep_router.py"
click node_compile "https://github.com/microsoft/deepspeed/tree/master/deepspeed/compile"
click node_inference_engine "https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/engine.py"
click node_v2_factory "https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/engine_factory.py"
click node_v2_engine "https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/engine_v2.py"
click node_model_impl "https://github.com/microsoft/deepspeed/tree/master/deepspeed/inference/v2/model_implementations"
click node_ragged_state "https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/ragged/ragged_manager.py"
click node_inference_kernels "https://github.com/microsoft/deepspeed/tree/master/deepspeed/inference/v2/kernels"
click node_nvme_io "https://github.com/microsoft/deepspeed/blob/master/deepspeed/nvme/io_engine.py"
click node_fast_writer "https://github.com/microsoft/deepspeed/blob/master/deepspeed/io/fast_file_writer.py"
click node_autotuning "https://github.com/microsoft/deepspeed/blob/master/deepspeed/autotuning/autotuner.py"
click node_accelerators "https://github.com/microsoft/deepspeed/tree/master/accelerator"
click node_science_attention "https://github.com/microsoft/deepspeed/blob/master/csrc/deepspeed4science/evoformer_attn/attention.cpp"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_training_engine,node_zero,node_optimizers,node_pipeline,node_checkpoint,node_inference_user toneBlue
class node_communication,node_torch_backend,node_groups,node_moe,node_compile toneAmber
class node_inference_engine,node_v2_factory,node_v2_engine,node_model_impl,node_ragged_state,node_inference_kernels toneMint
class node_nvme_io,node_fast_writer,node_autotuning,node_accelerators,node_science_attention toneRose
class node_developer toneIndigo
```

## Components

### Training Runtime

- [Training Engine](https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/engine.py): `deepspeed/runtime/engine.py`
- [ZeRO Sharding](https://github.com/microsoft/deepspeed/tree/master/deepspeed/runtime/zero): `deepspeed/runtime/zero`
- [Precision Optimizers](https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/bf16_optimizer.py): `deepspeed/runtime/bf16_optimizer.py`
- [Pipeline Engine](https://github.com/microsoft/deepspeed/blob/master/deepspeed/runtime/pipe/engine.py): `deepspeed/runtime/pipe/engine.py`
- [Checkpointing](https://github.com/microsoft/deepspeed/tree/master/deepspeed/checkpoint): `deepspeed/checkpoint`

### Distributed Execution

- [Communication API](https://github.com/microsoft/deepspeed/blob/master/deepspeed/comm/comm.py): `deepspeed/comm/comm.py`
- [Torch Backend](https://github.com/microsoft/deepspeed/blob/master/deepspeed/comm/torch.py): `deepspeed/comm/torch.py`
- [Parallel Groups](https://github.com/microsoft/deepspeed/blob/master/deepspeed/utils/groups.py): `deepspeed/utils/groups.py`
- [MoE Routing](https://github.com/microsoft/deepspeed/blob/master/deepspeed/moe/ep_router.py): `deepspeed/moe/ep_router.py`
- [DeepCompile](https://github.com/microsoft/deepspeed/tree/master/deepspeed/compile): `deepspeed/compile`

### Inference

- [Inference Engine](https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/engine.py): `deepspeed/inference/engine.py`
- [Inference V2 Factory](https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/engine_factory.py): `deepspeed/inference/v2/engine_factory.py`
- [Inference V2 Engine](https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/engine_v2.py): `deepspeed/inference/v2/engine_v2.py`
- [Model Implementations](https://github.com/microsoft/deepspeed/tree/master/deepspeed/inference/v2/model_implementations): `deepspeed/inference/v2/model_implementations`
- [Ragged Sequence State](https://github.com/microsoft/deepspeed/blob/master/deepspeed/inference/v2/ragged/ragged_manager.py): `deepspeed/inference/v2/ragged/ragged_manager.py`
- [Inference Kernels](https://github.com/microsoft/deepspeed/tree/master/deepspeed/inference/v2/kernels): `deepspeed/inference/v2/kernels`

### Systems Features

- [NVMe I/O](https://github.com/microsoft/deepspeed/blob/master/deepspeed/nvme/io_engine.py): `deepspeed/nvme/io_engine.py`
- [Fast File Writer](https://github.com/microsoft/deepspeed/blob/master/deepspeed/io/fast_file_writer.py): `deepspeed/io/fast_file_writer.py`
- [Autotuning](https://github.com/microsoft/deepspeed/blob/master/deepspeed/autotuning/autotuner.py): `deepspeed/autotuning/autotuner.py`
- [Accelerator Support](https://github.com/microsoft/deepspeed/tree/master/accelerator): `accelerator`
- [Science Attention](https://github.com/microsoft/deepspeed/blob/master/csrc/deepspeed4science/evoformer_attn/attention.cpp): `csrc/deepspeed4science/evoformer_attn/attention.cpp`

### Other components

- **Model Developer**
- **Inference Client**

## Connections

- Model Developer → Training Engine: configures training
- Training Engine → ZeRO Sharding: uses partitioning (evidence: `deepspeed/runtime/engine.py`)
- Training Engine → Precision Optimizers: uses optimizer (evidence: `deepspeed/runtime/engine.py`)
- Training Engine → Communication API: uses communication (evidence: `deepspeed/runtime/engine.py`)
- Training Engine → Pipeline Engine: supports pipeline (evidence: `deepspeed/runtime/pipe/engine.py`)
- Training Engine → Checkpointing: loads and saves (evidence: `deepspeed/runtime/engine.py`)
- Communication API → Torch Backend: dispatches operations (evidence: `deepspeed/comm/comm.py`)
- Torch Backend → Parallel Groups: uses utilities (evidence: `deepspeed/comm/torch.py`)
- Training Engine → MoE Routing: supports MoE (evidence: `deepspeed/runtime/engine.py`)
- Model Developer → DeepCompile: compiles training
- Inference Client → Inference Engine: runs inference
- Inference Client → Inference V2 Factory: creates engine
- Inference V2 Factory → Inference V2 Engine: constructs engine (evidence: `deepspeed/inference/v2/engine_factory.py`)
- Inference V2 Engine → Model Implementations: builds model (evidence: `deepspeed/inference/v2/engine_v2.py`)
- Inference V2 Engine → Ragged Sequence State: manages sequences (evidence: `deepspeed/inference/v2/engine_v2.py`)
- Inference V2 Engine → Communication API: initializes distributed (evidence: `deepspeed/inference/v2/engine_v2.py`)
- Inference Engine → Accelerator Support: uses device APIs (evidence: `deepspeed/inference/engine.py`)
- Inference Client → Inference V2 Engine: submits token batches (evidence: `deepspeed/inference/v2/engine_v2.py`)
- Inference V2 Engine → Inference Kernels: executes model kernels
- NVMe I/O → Fast File Writer: writes through
- Training Engine → Autotuning: collects model data
- Model Developer → Science Attention: uses research kernels
