How hazyresearch/thunderkittens works
The ThunderKittens project is a high-performance, CUDA-based framework focused on writing fast deep learning kernels by leveraging tile primitives and fine control of GPU resources. Its design is both low-level – dealing directly with CUDA, shared memory, and tensor cores – and modular, with various demos, kernels, tests, and utility components that validate and showcase its capabilities.
Main Components
• Core Library: At the heart is the ThunderKittens “kittens.cuh” header file and the rich collection of kernel primitives, data types (register tiles, shared tiles, and vectors), and operations that perform low-level math and memory operations.
• Kernels: The “kernels/” directory contains various CUDA kernels (e.g., attention, FFT conv, layernorm, matmul, rotary) and related Makefiles and test scripts. These kernels are the building blocks of the framework’s fast deep learning computations.
• Files & Low-Level Operations: The “include/” directory encapsulates core CUDA C++ implementations, including low-level operations (e.g., tile operations, memory handling) and definitions related to common types and operations.
• Demos & Examples: Multiple demos (in “demos/”) illustrate practical usage. These examples (based on “based_demo”, “llama_demo”, “lolcats_demo”, “qwen_demo”, etc.) show how to integrate the kernels with training and inference pipelines, sometimes with PyTorch bindings.
• Configuration & Build Tools: Files like “config.py”, “configs/”, “setup.py”, and various Makefiles underpin how the kernels are compiled and installed.
• Tests: The “tests/” folder (with both Python and unit-level CUDA tests) validates functionality and correctness in multiple areas (attention, matrix multiplication, layernorm, etc.).
• Documentation: The README, Doxyfile, and docs provide context about usage, installation instructions, and details on GPU programming model and kernel design.
Relationships and Interactions
• The “include” component (low-level C++/CUDA implementations and helper utilities).
• The “kernels” component where various high-performance operations (e.g., attention, matmul) are instantiated.
• Demos and examples in “demos” showcase how to use the library:
• They depend on the core library (include files and kernels) and illustrate integration with frameworks like PyTorch.
• The “tests” component, which interacts with both the library and its kernel functionalities to ensure correctness.
• The build and configuration paths (setup.py, config.py, various Makefiles) that tie together the core library, kernels, and demos during compilation and deployment.
• External Environment:
• The GPU hardware, with its CUDA programming model (threads, warps, blocks, grid), which is abstracted by ThunderKittens.
• Dependencies such as CUDA (version 12.3+), and compiler requirements (C++20, gcc-11 or clang-11).
Important Architectural Patterns and Principles
• Modularity: The project is organized into distinct modules (kernels, demos, tests, include) with clear separation of concerns.
• Low-Level Abstraction: The design encapsulates GPU programming arrays, through types (tile primitives) and operations that follow NICELY specified functional interfaces (destination-first semantics, layout enforcement).
• Extensibility: The library design inherently supports custom kernel development, as seen from the demos and configuration options. It allows integrating with other deep learning frameworks.
• Performance-Centric Design: Emphasizes using shared memory, asynchronous load/stores, tensor core operations, and warp and warpgroup optimizations.
• Header-Only Library: Since ThunderKittens is header-only, the primary dependency is the inclusion of the library files, with build configuration centralized in Makefiles and setup scripts.
Final Steps
• Revisit the README details that stress principles (simplicity, extensibility, speed) and ensure those principles are represented visually (e.g., layers demonstrating abstraction boundaries).