How microsoft/nnscaler works
Project Type and Purpose
• nnScaler is an open-source parallelization engine / compiler for Deep Neural Network (DNN) training.
• It takes a single-GPU PyTorch model and emits parallelized code that runs across multiple GPUs, exploring data, model, and pipeline parallelism.
Key Architectural Layers and Directories
• CLI Layer (nnscaler/cli, launch_torchrun.py): Parses user arguments, configures Trainer, and kicks off compilation and execution.
• Frontend / Graph Tracing (nnscaler/graph/tracer): Hooks into PyTorch to trace user model code into a computation graph.
• Graph Parsing & IR (nnscaler/graph/parser, nnscaler/ir): Converts the traced graph into a platform-agnostic intermediate representation (IR).
• Parallelization Planning (nnscaler/autodist, nnscaler/execplan):
• Constraint-guided plan generation (dp_solver, spmd_solver, pipeline_solver)
• ExecPlan passes (fusion, grouping) that refine how operators are partitioned
• Code Generation (nnscaler/codegen): Lowers IR + parallelization plan into executable PyTorch code or C++ kernels.
• Runtime Engine (nnscaler/runtime):
• Executor that dispatches generated plan across devices
• Adapters for communication collectives, optimizer integration, mixed precision
• Integration Layer (nnscaler/integration): Hooks for PyTorch Lightning Fabric / Trainer compatibility.
• Profiler & Cost Database (nnscaler/profiler, nnscaler/autodist/cost_database): Collects performance metrics to guide planning.
• Utilities & Configuration (nnscaler/utils.py, nnscaler/flags.py, nnscaler/version.py).