# hazyresearch/thunderkittens architecture

> Architecture diagram of the GitHub repository hazyresearch/thunderkittens, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-01). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hazyresearch/thunderkittens
- Repository: https://github.com/hazyresearch/thunderkittens

## Overview

The ThunderKittens project is a high-performance, CUDA-based framework focused on writing fast deep learning kernels by leveraging tile primitives and fine control of GPU resources. Its design is both low-level – dealing directly with CUDA, shared memory, and tensor cores – and modular, with various demos, kernels, tests, and utility components that validate and showcase its capabilities.

##### Main Components
- Core Library: At the heart is the ThunderKittens “kittens.cuh” header file and the rich collection of kernel primitives, data types (register tiles, shared tiles, and vectors), and operations that perform low-level math and memory operations.
- Kernels: The “kernels/” directory contains various CUDA kernels (e.g., attention, FFT conv, layernorm, matmul, rotary) and related Makefiles and test scripts. These kernels are the building blocks of the framework’s fast deep learning computations.
- Files & Low-Level Operations: The “include/” directory encapsulates core CUDA C++ implementations, including low-level operations (e.g., tile operations, memory handling) and definitions related to common types and operations.
- Demos & Examples: Multiple demos (in “demos/”) illustrate practical usage. These examples (based on “based_demo”, “llama_demo”, “lolcats_demo”, “qwen_demo”, etc.) show how to integrate the kernels with training and inference pipelines, sometimes with PyTorch bindings.
- Configuration & Build Tools: Files like “config.py”, “configs/”, “setup.py”, and various Makefiles underpin how the kernels are compiled and installed.
- Tests: The “tests/” folder (with both Python and unit-level CUDA tests) validates functionality and correctness in multiple areas (attention, matrix multiplication, layernorm, etc.).
- Documentation: The README, Doxyfile, and docs provide context about usage, installation instructions, and details on GPU programming model and kernel design.

##### Relationships and Interactions
- The “include” component (low-level C++/CUDA implementations and helper utilities).
- The “kernels” component where various high-performance operations (e.g., attention, matmul) are instantiated.
- Demos and examples in “demos” showcase how to use the library:
- They depend on the core library (include files and kernels) and illustrate integration with frameworks like PyTorch.
- The “tests” component, which interacts with both the library and its kernel functionalities to ensure correctness.
- The build and configuration paths (setup.py, config.py, various Makefiles) that tie together the core library, kernels, and demos during compilation and deployment.
- External Environment:
- The GPU hardware, with its CUDA programming model (threads, warps, blocks, grid), which is abstracted by ThunderKittens.
- Dependencies such as CUDA (version 12.3+), and compiler requirements (C++20, gcc-11 or clang-11).

##### Important Architectural Patterns and Principles
- Modularity: The project is organized into distinct modules (kernels, demos, tests, include) with clear separation of concerns.
- Low-Level Abstraction: The design encapsulates GPU programming arrays, through types (tile primitives) and operations that follow NICELY specified functional interfaces (destination-first semantics, layout enforcement).
- Extensibility: The library design inherently supports custom kernel development, as seen from the demos and configuration options. It allows integrating with other deep learning frameworks.
- Performance-Centric Design: Emphasizes using shared memory, asynchronous load/stores, tensor core operations, and warp and warpgroup optimizations.
- Header-Only Library: Since ThunderKittens is header-only, the primary dependency is the inclusion of the library files, with build configuration centralized in Makefiles and setup scripts.

##### Final Steps
- Revisit the README details that stress principles (simplicity, extensibility, speed) and ensure those principles are represented visually (e.g., layers demonstrating abstraction boundaries).

## Diagram (Mermaid)

```mermaid
flowchart TD
    Root("ThunderKittens GPU Kernel Framework"):::root

    %% Core Framework Subgraph
    subgraph "Core Framework"
        CL("Core Library (kittens.cuh)"):::core
        LLO("Low-Level CUDA Operations"):::core
        K("Kernel Implementations"):::core
    end

    %% Demos & Tests Subgraph
    subgraph "Demos & Tests"
        D("Demos & Examples"):::demo
        T("Tests"):::test
    end

    %% Build & Documentation Subgraph
    subgraph "Build & Documentation"
        subgraph "Build & Config"
            Cfg("config.py"):::build
            Setup("setup.py"):::build
            Dcfg("demos/configs/"):::build
        end
        subgraph "Documentation"
            Readme("README.md"):::doc
            Doxy("Doxyfile"):::doc
            Docs("Documentation (docs/)"):::doc
        end
    end

    %% External Environment
    GPU("GPU Hardware/CUDA Runtime<br/>(CUDA Threads/Warps layering)"):::external

    %% Connections
    Root --> CL
    CL --> LLO
    CL --> K

    D -->|"depends_on"| CL
    D -->|"depends_on"| K

    T -->|"validates"| CL
    T -->|"validates"| K

    Cfg -->|"configures"| CL
    Cfg -->|"configures"| K

    Setup -->|"builds"| CL
    Setup -->|"builds"| K

    Dcfg -->|"configures_demo"| D

    Root --> GPU

    Root --> Cfg
    Root --> Setup
    Root --> Dcfg
    Root --> Readme
    Root --> Doxy
    Root --> Docs

    %% Styles
    classDef root fill:#e6e6fa,stroke:#4b0082,stroke-width:2px;
    classDef core fill:#AED6F1,stroke:#1B4F72,stroke-width:2px;
    classDef demo fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef test fill:#F5CBA7,stroke:#A04000,stroke-width:2px;
    classDef build fill:#D5F5E3,stroke:#1E8449,stroke-width:2px;
    classDef doc fill:#FADBD8,stroke:#922B21,stroke-width:2px;
    classDef external fill:#F0B27A,stroke:#7D6608,stroke-width:2px;

    %% Click Events
    click CL "https://github.com/hazyresearch/thunderkittens/blob/main/include/kittens.cuh"
    click LLO "https://github.com/hazyresearch/thunderkittens/tree/main/include/"
    click K "https://github.com/hazyresearch/thunderkittens/tree/main/kernels/"
    click D "https://github.com/hazyresearch/thunderkittens/tree/main/demos/"
    click Cfg "https://github.com/hazyresearch/thunderkittens/blob/main/config.py"
    click Setup "https://github.com/hazyresearch/thunderkittens/blob/main/setup.py"
    click Dcfg "https://github.com/hazyresearch/thunderkittens/tree/main/demos/configs/"
    click T "https://github.com/hazyresearch/thunderkittens/tree/main/tests/"
    click Readme "https://github.com/hazyresearch/thunderkittens/blob/main/README.md"
    click Doxy "https://github.com/hazyresearch/thunderkittens/tree/main/Doxyfile"
    click Docs "https://github.com/hazyresearch/thunderkittens/tree/main/docs/"
```
