# deepseek-ai/deepgemm architecture

> Architecture diagram of the GitHub repository deepseek-ai/deepgemm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/deepseek-ai/deepgemm
- Repository: https://github.com/deepseek-ai/deepgemm

## Overview

##### Main Components
- API & Interface Layer: Although minimal, the public API exposed via Python calls (e.g., gemm functions and utility functions in deep_gemm/__init__.py) forms the entry point. This is where end users call functions like gemm_fp8_fp8_bf16_nt.
- Core Kernel Components: The core GEMM kernels reside in the deep_gemm/include/deep_gemm directory. These include the primary kernel (fp8_gemm.cuh), utility functions, TMA support utilities, and the scheduler (scheduler.cuh).
- JIT Kernels and Tuning: Further abstraction and tuning are done in deep_gemm/jit_kernels. This module includes specialized kernel code for both normal and grouped GEMM operations, along with support utilities (like the tuner in tuner.py and grouping kernels in m_grouped_gemm.py).
- Utility Functions & Environment Control: Several utility routines (like deep_gemm/utils.py) and environmental variable setups (DG_CACHE_DIR, DG_NVCC_COMPILER, etc.) are provided for performance tuning and configuration.
- External Dependencies and Submodules: Note that CUTLASS (located in third-party/cutlass) is included as a submodule and is used for inspiration and selective functionality.
- Testing & Development: The tests folder contains unit and integration tests that validate the JIT compilation and the GEMM implementations.
- Build & Installation: The setup.py script provides instructions for installing the library and linking with required third-party includes.

##### Relationships and Interactions
- The API layer forwards GEMM computation requests to the underlying kernel functions.
- The JIT compiler uses source templates (deep_gemm/jit/template.py) and performs potential modifications (e.g., FFMA interleaving via deep_gemm/jit/interleave_ffma.py) before load-time compilation.
- The compiled kernels are then executed on the GPU using CUDA, where the deep GEMM functions operate using kernel-level scheduling (as seen in scheduler.cuh) to maximize hardware utilization.
- Utility functions and environment variables allow fine-grained control over settings such as alignment, scaling factors, and debugging outputs.

##### Incorporate Architectural Patterns & Design Principles
- Modular Design: Separate modules for JIT, kernel implementations, testing, and utilities.
- Dynamic Optimization: The use of JIT compilation represents a design where performance-critical decisions (like block sizing, unrolling, and resource allocation) are computed at runtime.
- GPU-specific Optimizations: Special modules (like scheduler and interleave scripts) directly manipulate low-level CUDA and PTX instructions to enhance throughput.
- Clean Separation between Compilation and Execution: There is a clear boundary between what is compiled (JIT modules) and what is executed (compiled GEMM kernels).

##### Details
- Be very detailed: try to break down the structure into as many components as possible as outlined above.
- Not only logical modules but also their interactions, especially the key sequence from API call → JIT compilation (including templating and FFMA interleaving) → kernel execution with TMA and scheduler assistance.

## Diagram (Mermaid)

```mermaid
graph TD
    %% API Layer
    subgraph "API Layer"
        API["API Interface\n(Python)"]:::api
    end

    %% JIT Module Layer
    subgraph "JIT Module"
        JIT["JIT Compiler"]:::jit
        Template["Kernel Templates & FFMA Interleaving"]:::jit
    end

    %% Core Kernel Components
    subgraph "Core Kernels"
        Kernels["Core GEMM Kernels"]:::kernel
        JITK["JIT Kernels & Tuning"]:::kernel
        Scheduler["Scheduler (Kernel Scheduling)"]:::kernel
    end

    %% Execution Layer
    subgraph "Execution"
        GPU["CUDA GPU Execution"]:::exec
    end

    %% Utility and External Dependencies
    subgraph "Utilities"
        Utils["Utility Functions & Environment Control"]:::util
        ExDeps["External Dependencies\n(CUTLASS)"]:::util
    end

    %% Build & Testing
    subgraph "Build & Testing"
        Test["Testing & Development"]:::test
        Build["Build & Installation"]:::build
    end

    %% Relationships
    API -->|"initiates"| JIT
    JIT -->|"uses_Templates"| Template
    Template -->|"compiles"| Kernels
    JIT -->|"optimizes"| JITK
    JITK -->|"integrates_with"| Scheduler
    Scheduler -->|"executes_on"| GPU

    Utils -->|"configures"| JIT
    Utils -->|"configures"| Kernels
    ExDeps -->|"assists"| Kernels

    Test -->|"validates"| API
    Test -->|"validates"| JIT
    Build -->|"installs"| API

    %% Styles
    classDef api fill:#ccf,stroke:#333,stroke-width:2px;
    classDef jit fill:#cfc,stroke:#333,stroke-width:2px;
    classDef kernel fill:#ffc,stroke:#333,stroke-width:2px;
    classDef exec fill:#fcc,stroke:#333,stroke-width:2px;
    classDef util fill:#eef,stroke:#333,stroke-width:2px;
    classDef test fill:#cce,stroke:#333,stroke-width:2px;
    classDef build fill:#efe,stroke:#333,stroke-width:2px;

    %% Click Events
    click API "https://github.com/deepseek-ai/deepgemm/blob/main/deep_gemm/__init__.py"
    click Kernels "https://github.com/deepseek-ai/deepgemm/tree/main/deep_gemm/include/deep_gemm"
    click JIT "https://github.com/deepseek-ai/deepgemm/tree/main/deep_gemm/jit"
    click JITK "https://github.com/deepseek-ai/deepgemm/tree/main/deep_gemm/jit_kernels"
    click Utils "https://github.com/deepseek-ai/deepgemm/blob/main/deep_gemm/utils.py"
    click ExDeps "https://github.com/deepseek-ai/deepgemm/tree/main/third-party/cutlass"
    click Test "https://github.com/deepseek-ai/deepgemm/tree/main/tests"
    click Build "https://github.com/deepseek-ai/deepgemm/blob/main/setup.py"
```
