# nvidia/cutlass architecture

> Architecture diagram of the GitHub repository nvidia/cutlass, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/cutlass
- Repository: https://github.com/nvidia/cutlass

## Overview

##### Major Components
- “Core CUTLASS Library” – This is represented by the “include/” directory. Inside this, you have the main namespaces such as:
- cutlass/arch (architecture-specific abstractions for memory, MMA, and low-level instructions),
- cutlass/conv (convolution-specific implementations using implicit GEMM),
- cutlass/gemm (GEMM implementations with many strategies),
- cutlass/epilogue (postprocessing of GEMM outputs, such as bias, activation, etc.),
- cutlass/layout (definitions for different data layouts (row-major, column-major, interleaved, etc.)),
- cutlass/reduction, cutlass/transform, cutlass/thread and others.
- “CuTe Library” – Found in the “cute/” directory. This is the abstraction for tensor and thread layouts, tiling operations, copy and MMA “atom” primitives – these form the low-level building blocks that are also used by CUTLASS.
- “SDK Examples” – Located in the “examples/” directory. Provide sample use cases such as basic GEMM, convolution variants and emerging architectures like Hopper and Blackwell.
- “Tools” – This has two major subcomponents:
- The Library instantiations (in tools/library), which generate specific kernel instances that are then usable by an application.
- The Profiler (in tools/profiler) which not only runs performance tests but is also responsible for validating correctness; it can be configured via command-line to selectively build specific kernel variants (using kernel-name wildcards, etc.).
- “Tests” – The “test/” folder contains unit tests (mostly built via Google Test) that validate every level of CUTLASS (core numerical operations, kernel instantiations, GEMM, convolution, etc.). This also includes self-contained tests and targets for NVRTC and other experimental aspects.
- “Python Bindings and Documentation” – Notice the “python/” directory which contains the Python packages (bindings and higher level APIs) that allow CUTLASS kernels to be invoked from Python. Also the “docs/” and “python/docs/” directories contain the auto-generated Doxygen documentation as well as the in-depth usage guides and API documents.

##### Relationships and Dependencies
- The layered design includes a compile‐time hierarchy. The “CUTLASS Library” includes the “cute” (CuTe) core primitives. In particular, the abstractions in CUTLASS rely on CuTe for defining and transforming tensor layouts.
- The CUTLASS Template Library is used as the building block for all GPU kernels (GEMM, convolution, trmm, etc.). These are designed using architecture‐specific subcomponents (e.g., the “arch” files) that directly depend on the target CUDA architecture.
- Tools and tests depend on the CUTLASS Library and use the instantiation provided in tools/library to launch the kernels.
- The build system uses CMake to set factors such as “CUTLASS_NVCC_ARCHS” that control which GPU architectures are targeted.
- The documentation and Python bindings wrap around the core library to facilitate a higher level user interface.

##### Important Architectural Patterns
- Hierarchical decomposition: The project organizes the computation in multiple layers. For instance, a GEMM kernel is broken down into an “epilogue”, a “MMA” (matrix multiply-accumulate) stage, tiling strategies, and per-thread tiles. Each of these layers is separately parameterized.
- Modularity and composability: The design separates abstraction of data layouts, computation “atoms” (MMA / Copy primitives), grouping of compute into warp-level and threadblock-level operations, and then fuses them into complete kernels.
- Customization via policy and dispatch: Users and the CUTLASS profiler can select a specific kernel variant by providing kernel name patterns and architectural constraints.
- Integration with external frameworks: The library is integrated with CUDA (especially through the CUDA Toolkit’s compilation processes and NVRTC) as well as providing Python bindings.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Build System
    BS["Build System (CMake)"]:::build

    %% External Dependencies
    CUDA["CUDA Toolkit"]:::external
    GPU["GPU Hardware"]:::external
    Compiler["C++17 Compiler"]:::external
    BLAS["cuBLAS/cuDNN"]:::external

    %% Top Layer: User Integration
    subgraph "User Integration"
        SDKExamples["SDK Examples"]:::user
        PythonBindings["Python Bindings"]:::user
        PythonDocs["Documentation"]:::user
    end

    %% Middle Layer: Core Libraries
    subgraph "Core Libraries"
        CuTeLibrary["CuTe Library"]:::core
        CUTLASSCoreLibrary["CUTLASS Core Library"]:::core
        Submodules["Submodules: arch, conv, gemm, epilogue, layout, reduction, transform, thread,..."]:::coreText
    end

    %% Lower Layer: Supporting Tools and Tests
    subgraph "Support Tools"
        ToolsLibrary["Tools: Library Instantiation"]:::tool
        ToolsProfiler["Tools: Profiler"]:::tool
        Tests["Tests (Google Test)"]:::tool
    end

    %% Build system configuration
    BS --> CUTLASSCoreLibrary
    BS --> CuTeLibrary
    BS --> SDKExamples
    BS --> PythonBindings
    BS --> PythonDocs
    BS --> ToolsLibrary
    BS --> ToolsProfiler
    BS --> Tests

    %% External dependencies for core
    CUTLASSCoreLibrary --> CUDA
    CUTLASSCoreLibrary --> GPU
    CUTLASSCoreLibrary --> Compiler
    CUTLASSCoreLibrary --> BLAS

    %% CuTe integration into CUTLASS
    CuTeLibrary --> CUTLASSCoreLibrary

    %% Core Library usage by User Integration
    CUTLASSCoreLibrary --> SDKExamples
    CUTLASSCoreLibrary --> PythonBindings
    CUTLASSCoreLibrary --> PythonDocs

    %% Core Library usage by Tools and Tests
    CUTLASSCoreLibrary --> ToolsLibrary
    CUTLASSCoreLibrary --> ToolsProfiler
    CUTLASSCoreLibrary --> Tests

    %% Annotation for core submodules
    CUTLASSCoreLibrary --- Submodules

    %% Class Definitions for colour and style
    classDef build fill:#f9d,stroke:#d14,stroke-width:2px;
    classDef external fill:#ffd,stroke:#da2,stroke-width:2px;
    classDef core fill:#ccf,stroke:#55a,stroke-width:2px;
    classDef tool fill:#cfc,stroke:#5a5,stroke-width:2px;
    classDef user fill:#fcc,stroke:#a55,stroke-width:2px;
    classDef coreText fill:#eef,stroke:#99f,stroke-width:0px,stroke-dasharray: 5 5;

    %% Click Events
    click CUTLASSCoreLibrary "https://github.com/nvidia/cutlass/tree/main/include/cutlass"
    click CuTeLibrary "https://github.com/nvidia/cutlass/tree/main/include/cute"
    click SDKExamples "https://github.com/nvidia/cutlass/tree/main/examples"
    click ToolsLibrary "https://github.com/nvidia/cutlass/tree/main/tools/library"
    click ToolsProfiler "https://github.com/nvidia/cutlass/tree/main/tools/profiler"
    click Tests "https://github.com/nvidia/cutlass/tree/main/test"
    click PythonBindings "https://github.com/nvidia/cutlass/tree/main/python"
    click PythonDocs "https://github.com/nvidia/cutlass/tree/main/python/docs"
```
