# microsoft/nnscaler architecture

> Architecture diagram of the GitHub repository microsoft/nnscaler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/nnscaler
- Repository: https://github.com/microsoft/nnscaler

## Overview

##### Project Type and Purpose
- nnScaler is an open-source parallelization engine / compiler for Deep Neural Network (DNN) training.
- It takes a single-GPU PyTorch model and emits parallelized code that runs across multiple GPUs, exploring data, model, and pipeline parallelism.

##### Key Architectural Layers and Directories
- CLI Layer (`nnscaler/cli`, `launch_torchrun.py`): Parses user arguments, configures Trainer, and kicks off compilation and execution.
- Frontend / Graph Tracing (`nnscaler/graph/tracer`): Hooks into PyTorch to trace user model code into a computation graph.
- Graph Parsing & IR (`nnscaler/graph/parser`, `nnscaler/ir`): Converts the traced graph into a platform-agnostic intermediate representation (IR).
- Parallelization Planning (`nnscaler/autodist`, `nnscaler/execplan`):
- Constraint-guided plan generation (dp_solver, spmd_solver, pipeline_solver)
- ExecPlan passes (fusion, grouping) that refine how operators are partitioned
- Code Generation (`nnscaler/codegen`): Lowers IR + parallelization plan into executable PyTorch code or C++ kernels.
- Runtime Engine (`nnscaler/runtime`):
- Executor that dispatches generated plan across devices
- Adapters for communication collectives, optimizer integration, mixed precision
- Integration Layer (`nnscaler/integration`): Hooks for PyTorch Lightning Fabric / Trainer compatibility.
- Profiler & Cost Database (`nnscaler/profiler`, `nnscaler/autodist/cost_database`): Collects performance metrics to guide planning.
- Utilities & Configuration (`nnscaler/utils.py`, `nnscaler/flags.py`, `nnscaler/version.py`).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Subgraphs by layer
    subgraph "Frontend Layer"
        CLI["CLI Layer"]:::frontend
        Tracer["Graph Tracer"]:::frontend
    end

    subgraph "Mid-end Layer"
        Parser["Graph Parser"]:::mid
        IR["Intermediate Representation (IR)"]:::mid
        PlannerAD["AutoDist Planner"]:::mid
        PlannerEP["ExecPlan Planner"]:::mid
        Profiler["Profiler"]:::mid
        CostDB["Cost Database"]:::mid
    end

    subgraph "Back-end Layer"
        CodeGen["Code Generation Backend"]:::backend
    end

    subgraph "Runtime Layer"
        Runtime["Runtime Engine"]:::runtime
    end

    subgraph "Integration Layer"
        Integration["Integration Adapters"]:::integration
    end

    subgraph "Utilities & Config"
        Utils["Utilities & Configuration"]:::util
    end

    %% External Dependencies
    PyTorch["PyTorch FX / TorchScript"]:::external
    NCCL["CUDA / NCCL / cuBLAS"]:::external

    %% Connections
    CLI -->|"parse args & kickoff"| Tracer
    Tracer -->|"trace model graph"| Parser
    Parser -->|"produce IR"| IR
    IR -->|"IR format"| PlannerAD
    IR -->|"IR format"| PlannerEP
    Profiler -->|"collect metrics"| CostDB
    CostDB -->|"provide cost profiles"| PlannerAD
    PlannerAD -->|"plan JSON"| CodeGen
    PlannerEP -->|"plan JSON"| CodeGen
    CodeGen -->|"generated code"| Runtime
    Runtime -->|"calls APIs"| PyTorch
    Runtime -->|"collectives"| NCCL
    Integration <-->|"hooks / callbacks"| Runtime
    Integration <-->|"CLI extensions"| CLI
    Utils --> CLI
    Utils --> Tracer
    Utils --> Parser
    Utils --> PlannerAD
    Utils --> PlannerEP
    Utils --> CodeGen
    Utils --> Runtime
    Utils --> Integration

    %% Click Events
    click CLI "https://github.com/microsoft/nnscaler/tree/main/nnscaler/cli/"
    click Tracer "https://github.com/microsoft/nnscaler/tree/main/nnscaler/graph/tracer/"
    click Parser "https://github.com/microsoft/nnscaler/tree/main/nnscaler/graph/parser/"
    click IR "https://github.com/microsoft/nnscaler/tree/main/nnscaler/ir/"
    click PlannerAD "https://github.com/microsoft/nnscaler/tree/main/nnscaler/autodist/"
    click PlannerEP "https://github.com/microsoft/nnscaler/tree/main/nnscaler/execplan/"
    click CodeGen "https://github.com/microsoft/nnscaler/tree/main/nnscaler/codegen/"
    click Runtime "https://github.com/microsoft/nnscaler/tree/main/nnscaler/runtime/"
    click Integration "https://github.com/microsoft/nnscaler/tree/main/nnscaler/integration/"
    click Profiler "https://github.com/microsoft/nnscaler/tree/main/nnscaler/profiler/"
    click CostDB "https://github.com/microsoft/nnscaler/blob/main/nnscaler/autodist/cost_database.py"
    click Utils "https://github.com/microsoft/nnscaler/blob/main/nnscaler/utils.py"
    click Utils "https://github.com/microsoft/nnscaler/blob/main/nnscaler/flags.py"
    click Utils "https://github.com/microsoft/nnscaler/blob/main/nnscaler/version.py"

    %% Styles
    classDef frontend fill:#D0E8FF,stroke:#0366d6,stroke-width:1px
    classDef mid fill:#D0FFD6,stroke:#28a745,stroke-width:1px
    classDef backend fill:#FFE0A3,stroke:#d87c00,stroke-width:1px
    classDef runtime fill:#FFCCCC,stroke:#d73a49,stroke-width:1px
    classDef integration fill:#E4D0FF,stroke:#6f42c1,stroke-width:1px
    classDef util fill:#F0F0F0,stroke:#6a737d,stroke-width:1px
    classDef external fill:#FFFFFF,stroke:#999999,stroke-dasharray: 5 5,stroke-width:1px
```
