# geeeekexplorer/3d-parallel-demo architecture

> Architecture diagram of the GitHub repository geeeekexplorer/3d-parallel-demo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/geeeekexplorer/3d-parallel-demo
- Repository: https://github.com/geeeekexplorer/3d-parallel-demo

## Overview

parallel-demo is a small set of runnable PyTorch distributed-computing demonstrations, rather than a single application. Its main user workflow is to launch a DP, TP, or PP script with torchrun; each initializes distributed execution, builds a model, applies one parallel strategy, and runs forward/backward computation. A shared neural-network model supports the demonstrations. The README also advertises a CUDA stream example in stream.py. Edges reflect calls visible in the excerpts; PyTorch and torchrun are external runtime dependencies.

Main entry points: [`dp.py`](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py) (DP runner), [`tp.py`](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py) (TP runner), [`pp.py`](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py) (PP runner), [`model.py`](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/model.py) (Net)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_examples["Parallel demos"]
  node_dp_runner["DP runner<br/>[dp.py]"]
  node_ddp["DDP wrapper<br/>[dp.py]"]
  node_gradient_buckets["Gradient buckets<br/>[dp.py]"]
  node_tp_runner["TP runner<br/>[tp.py]"]
  node_tp_net["TP network<br/>[tp.py]"]
  node_tp_autograd["TP autograd ops<br/>[tp.py]"]
  node_pp_runner["PP runner<br/>[pp.py]"]
  node_pipe["Pipeline partition<br/>[pp.py]"]
  node_pipe_autograd["Pipeline transfers<br/>[pp.py]"]
end

subgraph group_model["Shared model"]
  node_net["Net<br/>[model.py]"]
end

subgraph group_stream["CUDA streams"]
  node_stream_demo["Stream demo<br/>[stream.py]"]
  node_cuda_stream["CUDA streams"]
end

node_user(("Developer"))
node_torchrun["torchrun"]
node_distributed["PyTorch distributed"]

node_user -->|"launches"| node_torchrun
node_torchrun -->|"runs"| node_dp_runner
node_torchrun -->|"runs"| node_tp_runner
node_torchrun -->|"runs"| node_pp_runner
node_dp_runner -->|"initializes"| node_distributed
node_tp_runner -->|"initializes"| node_distributed
node_pp_runner -->|"initializes"| node_distributed
node_dp_runner -->|"constructs"| node_net
node_tp_runner -->|"constructs"| node_net
node_pp_runner -->|"constructs"| node_net
node_dp_runner -->|"wraps model"| node_ddp
node_ddp -->|"forwards to"| node_net
node_ddp -->|"registers hooks"| node_gradient_buckets
node_gradient_buckets -->|"all-reduces"| node_distributed
node_tp_runner -->|"creates shard"| node_tp_net
node_tp_net -->|"applies"| node_tp_autograd
node_tp_autograd -->|"all-reduces"| node_distributed
node_pp_runner -->|"wraps layers"| node_pipe
node_pipe -->|"transfers activations"| node_pipe_autograd
node_pipe_autograd -->|"sends and receives"| node_distributed
node_stream_demo -->|"uses"| node_cuda_stream

click node_dp_runner "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py"
click node_ddp "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py"
click node_gradient_buckets "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py"
click node_tp_runner "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py"
click node_tp_net "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py"
click node_tp_autograd "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py"
click node_pp_runner "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py"
click node_pipe "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py"
click node_pipe_autograd "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py"
click node_net "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/model.py"
click node_stream_demo "https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/stream.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dp_runner,node_ddp,node_gradient_buckets,node_tp_runner,node_tp_net,node_tp_autograd,node_pp_runner,node_pipe,node_pipe_autograd toneBlue
class node_net toneAmber
class node_stream_demo,node_cuda_stream toneMint
class node_user,node_torchrun,node_distributed toneIndigo
```

## Components

### Parallel demos

- [DP runner](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py): `dp.py`
- [DDP wrapper](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py): `dp.py`
- [Gradient buckets](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/dp.py): `dp.py`
- [TP runner](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py): `tp.py`
- [TP network](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py): `tp.py`
- [TP autograd ops](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/tp.py): `tp.py`
- [PP runner](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py): `pp.py`
- [Pipeline partition](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py): `pp.py`
- [Pipeline transfers](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/pp.py): `pp.py`

### Shared model

- [Net](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/model.py): `model.py`

### CUDA streams

- [Stream demo](https://github.com/geeeekexplorer/3d-parallel-demo/blob/main/stream.py): `stream.py`
- **CUDA streams**

### Other components

- **Developer**
- **torchrun**
- **PyTorch distributed**

## Connections

- Developer → torchrun: launches (evidence: `README.md`)
- torchrun → DP runner: runs (evidence: `README.md`)
- torchrun → TP runner: runs (evidence: `README.md`)
- torchrun → PP runner: runs (evidence: `README.md`)
- DP runner → PyTorch distributed: initializes (evidence: `dp.py`)
- TP runner → PyTorch distributed: initializes (evidence: `tp.py`)
- PP runner → PyTorch distributed: initializes (evidence: `pp.py`)
- DP runner → Net: constructs (evidence: `dp.py`)
- TP runner → Net: constructs (evidence: `tp.py`)
- PP runner → Net: constructs (evidence: `pp.py`)
- DP runner → DDP wrapper: wraps model (evidence: `dp.py`)
- DDP wrapper → Net: forwards to (evidence: `dp.py`)
- DDP wrapper → Gradient buckets: registers hooks (evidence: `dp.py`)
- Gradient buckets → PyTorch distributed: all-reduces (evidence: `dp.py`)
- TP runner → TP network: creates shard (evidence: `tp.py`)
- TP network → TP autograd ops: applies (evidence: `tp.py`)
- TP autograd ops → PyTorch distributed: all-reduces (evidence: `tp.py`)
- PP runner → Pipeline partition: wraps layers (evidence: `pp.py`)
- Pipeline partition → Pipeline transfers: transfers activations (evidence: `pp.py`)
- Pipeline transfers → PyTorch distributed: sends and receives (evidence: `pp.py`)
- Stream demo → CUDA streams: uses (evidence: `stream.py`)
