How deepseek-ai/deepgemm works
Main Components
• API & Interface Layer: Although minimal, the public API exposed via Python calls (e.g., gemm functions and utility functions in deep_gemm/__init__.py) forms the entry point. This is where end users call functions like gemm_fp8_fp8_bf16_nt.
• Core Kernel Components: The core GEMM kernels reside in the deep_gemm/include/deep_gemm directory. These include the primary kernel (fp8_gemm.cuh), utility functions, TMA support utilities, and the scheduler (scheduler.cuh).
• JIT Kernels and Tuning: Further abstraction and tuning are done in deep_gemm/jit_kernels. This module includes specialized kernel code for both normal and grouped GEMM operations, along with support utilities (like the tuner in tuner.py and grouping kernels in m_grouped_gemm.py).
• Utility Functions & Environment Control: Several utility routines (like deep_gemm/utils.py) and environmental variable setups (DG_CACHE_DIR, DG_NVCC_COMPILER, etc.) are provided for performance tuning and configuration.
• External Dependencies and Submodules: Note that CUTLASS (located in third-party/cutlass) is included as a submodule and is used for inspiration and selective functionality.
• Testing & Development: The tests folder contains unit and integration tests that validate the JIT compilation and the GEMM implementations.
• Build & Installation: The setup.py script provides instructions for installing the library and linking with required third-party includes.
Relationships and Interactions
• The API layer forwards GEMM computation requests to the underlying kernel functions.
• The JIT compiler uses source templates (deep_gemm/jit/template.py) and performs potential modifications (e.g., FFMA interleaving via deep_gemm/jit/interleave_ffma.py) before load-time compilation.
• The compiled kernels are then executed on the GPU using CUDA, where the deep GEMM functions operate using kernel-level scheduling (as seen in scheduler.cuh) to maximize hardware utilization.
• Utility functions and environment variables allow fine-grained control over settings such as alignment, scaling factors, and debugging outputs.
Incorporate Architectural Patterns & Design Principles
• Modular Design: Separate modules for JIT, kernel implementations, testing, and utilities.
• Dynamic Optimization: The use of JIT compilation represents a design where performance-critical decisions (like block sizing, unrolling, and resource allocation) are computed at runtime.
• GPU-specific Optimizations: Special modules (like scheduler and interleave scripts) directly manipulate low-level CUDA and PTX instructions to enhance throughput.
• Clean Separation between Compilation and Execution: There is a clear boundary between what is compiled (JIT modules) and what is executed (compiled GEMM kernels).
Details
• Be very detailed: try to break down the structure into as many components as possible as outlined above.
• Not only logical modules but also their interactions, especially the key sequence from API call → JIT compilation (including templating and FFMA interleaving) → kernel execution with TMA and scheduler assistance.