How geeeekexplorer/3d-parallel-demo works
parallel-demo is a small set of runnable PyTorch distributed-computing demonstrations, rather than a single application. Its main user workflow is to launch a DP, TP, or PP script with torchrun; each initializes distributed execution, builds a model, applies one parallel strategy, and runs forward/backward computation. A shared neural-network model supports the demonstrations. The README also advertises a CUDA stream example in stream.py. Edges reflect calls visible in the excerpts; PyTorch and torchrun are external runtime dependencies.
Main entry points: dp.py DP runner · tp.py TP runner · pp.py PP runner · model.py Net
Components 15
Parallel demos
- DP runner
dp.py - DDP wrapper
dp.py - Gradient buckets
dp.py - TP runner
tp.py - TP network
tp.py - TP autograd ops
tp.py - PP runner
pp.py - Pipeline partition
pp.py - Pipeline transfers
pp.py
Shared model
- Net
model.py
CUDA streams
- Stream demo
stream.py - CUDA streams
Other components
- Developer
- torchrun
- PyTorch distributed
Connections 21
- Developer → torchrun: launches
README.md - torchrun → DP runner: runs
README.md - torchrun → TP runner: runs
README.md - torchrun → PP runner: runs
README.md - DP runner → PyTorch distributed: initializes
dp.py - TP runner → PyTorch distributed: initializes
tp.py - PP runner → PyTorch distributed: initializes
pp.py - DP runner → Net: constructs
dp.py - TP runner → Net: constructs
tp.py - PP runner → Net: constructs
pp.py - DP runner → DDP wrapper: wraps model
dp.py - DDP wrapper → Net: forwards to
dp.py - DDP wrapper → Gradient buckets: registers hooks
dp.py - Gradient buckets → PyTorch distributed: all-reduces
dp.py - TP runner → TP network: creates shard
tp.py - TP network → TP autograd ops: applies
tp.py - TP autograd ops → PyTorch distributed: all-reduces
tp.py - PP runner → Pipeline partition: wraps layers
pp.py - Pipeline partition → Pipeline transfers: transfers activations
pp.py - Pipeline transfers → PyTorch distributed: sends and receives
pp.py - Stream demo → CUDA streams: uses
stream.py