# shi-labs/natten architecture

> Architecture diagram of the GitHub repository shi-labs/natten, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shi-labs/natten
- Repository: https://github.com/shi-labs/natten

## Overview

##### Project Type and Purpose
- This is a hybrid language library (Python + C++/CUDA) that implements Neighborhood Attention and Fused Neighborhood Attention (FNA) kernels as a PyTorch extension.
- High-level goal: expose fast, multi-dimensional sliding-window self-attention ops to Python users, with both CPU and GPU backends, packaged for pip.

##### Top-Level Components
A. Python Layer
- src/natten: the public API, Python wrappers for ops, autodiff integration, autotuner.
- setup.py, setup.cfg, Makefile: drive build, install, and wheel packaging.
- dev/packaging/*, dev/docker: scripts and Dockerfiles to build/test wheels for many CUDA/CPU variants.
- docs/, webpage/: user-facing documentation (Markdown) and static site assets.
- tests/: unit tests for Python and extension behaviors.

B. Native Extension Layer (csrc/)
- csrc/CMakeLists.txt + setup.py → builds a shared library (“natten” extension).
- csrc/include/: public & internal headers; PyTorch binding headers in include/natten/pytorch.
- csrc/src/: glue code (.cpp/.cu) registering forward/backward kernels with PyTorch.
- csrc/autogen/: kernel implementations generated by Python scripts in scripts/*.py.

C. Backends and Kernels
- CPU naive kernels (csrc/autogen/src/cpu/naive + csrc/include/natten/cpu).
- CUDA kernels:
– Naive, GEMM-based, and FNA variants under csrc/autogen/src/cuda/* and csrc/include/natten/cuda.
– Third-party CUTLASS library as a submodule for GEMM support (third_party/cutlass).

D. Tooling and Support
- scripts/autogen_*.py: generate specialized CUDA/C++ kernel code.
- tools/: Python scripts for profiling and benchmarking.
- docs/, docs/fna/: architecture and usage guides.

##### Key Relationships and Data Flows
User Python code
↓ calls
src/natten API (Python)
↓ dispatch (based on device & config)
PyTorch C++ extension (natten.cpp + CMake build)
↓ invokes
– CPU kernel or
– CUDA kernel (naive / gemm / fna)
↓ executes on target hardware
↓ returns tensor to Python
↓ backward pass follows same flow in reverse (registered backward kernels)

Autotuner (Python)
↔ reads/writes config
↔ benchmarks different kernel variants
↔ influences dispatch choices in API

##### Architectural Patterns and Principles
- Layered separation:
– Python API layer ←→ Native extension layer ←→ Hardware kernels
- Plug-in/backends: CPU vs. CUDA vs. GEMM vs. FNA → selected at runtime.
- Code generation: scripts auto-generate optimized, architecture-specific kernels.
- Polymorphic dispatch: use PyTorch’s registration mechanism to select forward/backward.
- Modular packaging: build scripts orchestrate multi-platform wheel builds.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User & API Layer
    subgraph "User & API Layer"
        User["User Python Code"]:::python
        API["natten Python API\n(src/natten)"]:::python
        click API "https://github.com/shi-labs/natten/tree/main/src/natten"
        subgraph "Public Python Ops"
            na1d["na1d.py"]:::python
            na2d["na2d.py"]:::python
            na3d["na3d.py"]:::python
            nat1d["natten1d.py"]:::python
            nat2d["natten2d.py"]:::python
            nat3d["natten3d.py"]:::python
            ops["ops.py"]:::python
            click na1d "https://github.com/shi-labs/natten/blob/main/src/natten/na1d.py"
            click na2d "https://github.com/shi-labs/natten/blob/main/src/natten/na2d.py"
            click na3d "https://github.com/shi-labs/natten/blob/main/src/natten/na3d.py"
            click nat1d "https://github.com/shi-labs/natten/blob/main/src/natten/natten1d.py"
            click nat2d "https://github.com/shi-labs/natten/blob/main/src/natten/natten2d.py"
            click nat3d "https://github.com/shi-labs/natten/blob/main/src/natten/natten3d.py"
            click ops "https://github.com/shi-labs/natten/blob/main/src/natten/ops.py"
        end
        subgraph "Autotuner Module"
            autotuner["autotuner\n(src/natten/autotuner)"]:::python
            click autotuner "https://github.com/shi-labs/natten/tree/main/src/natten/autotuner"
        end
    end

    %% Build & Tooling Layer
    subgraph "Tooling & Packaging"
        setuppy["setup.py\nsetup.cfg\nMakefile"]:::tool
        devpkg["dev/packaging/*"]:::tool
        devdocker["Dockerfile.CUDA\n(dev/docker)"]:::tool
        scripts["scripts/autogen_*.py"]:::tool
        tools["tools/profile_*.py"]:::tool
        tests["tests/"]:::tool
        docs["docs/"]:::tool
        webpage["webpage/"]:::tool
        reqdev["requirements-dev.txt"]:::tool
        reqdev2["scripts/dev/requirements.txt"]:::tool

        click setuppy "https://github.com/shi-labs/natten/blob/main/setup.py"
        click devpkg "https://github.com/shi-labs/natten/tree/main/dev/packaging/*"
        click devdocker "https://github.com/shi-labs/natten/blob/main/dev/docker/Dockerfile.CUDA"
        click scripts "https://github.com/shi-labs/natten/blob/main/scripts/autogen_cpu_naive.py"
        click tools "https://github.com/shi-labs/natten/blob/main/tools/profile_*.py"
        click tests "https://github.com/shi-labs/natten/tree/main/tests/"
        click docs "https://github.com/shi-labs/natten/tree/main/docs/"
        click webpage "https://github.com/shi-labs/natten/tree/main/webpage/"
        click reqdev "https://github.com/shi-labs/natten/blob/main/requirements-dev.txt"
        click reqdev2 "https://github.com/shi-labs/natten/blob/main/scripts/dev/requirements.txt"
    end

    %% Native Extension Layer
    subgraph "Native Extension Layer"
        CMake["CMakeLists.txt\n(csrc/)"]:::ext
        nattencpp["natten.cpp"]:::ext
        click CMake "https://github.com/shi-labs/natten/blob/main/csrc/CMakeLists.txt"
        click nattencpp "https://github.com/shi-labs/natten/blob/main/csrc/natten.cpp"
        subgraph "Glue Code"
            pt_src["csrc/src/pytorch\n(.cpp/.cu)"]:::ext
            pt_inc["csrc/include/natten/pytorch"]:::ext
            click pt_src "https://github.com/shi-labs/natten/tree/main/csrc/src/pytorch"
            click pt_inc "https://github.com/shi-labs/natten/tree/main/csrc/include/natten/pytorch"
        end
    end

    %% Generated Kernels
    subgraph "Generated Kernels"
        subgraph CPU
            cpu_src["csrc/autogen/src/cpu/naive"]:::gen
            cpu_inc["csrc/autogen/include/natten_autogen/cpu/naive"]:::gen
            click cpu_src "https://github.com/shi-labs/natten/tree/main/csrc/autogen/src/cpu/naive"
            click cpu_inc "https://github.com/shi-labs/natten/tree/main/csrc/autogen/include/natten_autogen/cpu/naive"
        end
        subgraph CUDA
            cuda_naive_src["csrc/autogen/src/cuda/naive"]:::gen
            cuda_naive_inc["csrc/autogen/include/natten_autogen/cuda/naive"]:::gen
            cuda_gemm_src["csrc/autogen/src/cuda/gemm"]:::gen
            cuda_gemm_inc["csrc/autogen/include/natten_autogen/cuda/gemm"]:::gen
            cuda_fna_src["csrc/autogen/src/cuda/fna"]:::gen
            cuda_fna_inc["csrc/autogen/include/natten_autogen/cuda/fna"]:::gen
            click cuda_naive_src "https://github.com/shi-labs/natten/tree/main/csrc/autogen/src/cuda/naive"
            click cuda_naive_inc "https://github.com/shi-labs/natten/tree/main/csrc/autogen/include/natten_autogen/cuda/naive"
            click cuda_gemm_src "https://github.com/shi-labs/natten/tree/main/csrc/autogen/src/cuda/gemm"
            click cuda_gemm_inc "https://github.com/shi-labs/natten/tree/main/csrc/autogen/include/natten_autogen/cuda/gemm"
            click cuda_fna_src "https://github.com/shi-labs/natten/tree/main/csrc/autogen/src/cuda/fna"
            click cuda_fna_inc "https://github.com/shi-labs/natten/tree/main/csrc/autogen/include/natten_autogen/cuda/fna"
        end
    end

    %% Headers & Third-Party
    subgraph "Headers"
        core_h["csrc/include/natten/*.h"]:::ext
        cpu_h["csrc/include/natten/cpu"]:::ext
        cuda_h["csrc/include/natten/cuda"]:::ext
        click core_h "https://github.com/shi-labs/natten/blob/main/csrc/include/natten/*.h"
        click cpu_h "https://github.com/shi-labs/natten/tree/main/csrc/include/natten/cpu"
        click cuda_h "https://github.com/shi-labs/natten/tree/main/csrc/include/natten/cuda"
    end
    cutlass["CUTLASS Library\n(third_party/cutlass)"]:::third
    click cutlass "https://github.com/shi-labs/natten/tree/main/third_party/cutlass"

    %% Runtime & Build Flows
    User -->|"calls"| API
    API -->|"dispatch"| nattencpp
    nattencpp -->|"registers"| pt_src
    pt_src -->|"invokes"| cpu_src
    pt_src -->|"invokes"| cuda_naive_src
    pt_src -->|"invokes"| cuda_gemm_src
    pt_src -->|"invokes"| cuda_fna_src
    scripts ---|"generates"| cpu_src
    scripts ---|"generates"| cuda_naive_src
    scripts ---|"generates"| cuda_gemm_src
    scripts ---|"generates"| cuda_fna_src
    autotuner <-->|"benchmarks"| scripts
    autotuner <-->|"influences"| API

    setuppy <-->|"build"| CMake
    setuppy <-->|"docker build"| devdocker
    devdocker -->|uses| CMake
    CMake -->|produces| nattencpp

    %% Legend
    classDef python fill:#D0E8FF,stroke:#0366D6,color:#0366D6
    classDef ext fill:#FFE7D1,stroke:#D66300,color:#D66300
    classDef gen fill:#DFFFD1,stroke:#28A745,color:#28A745
    classDef third fill:#F0F0F0,stroke:#666666,color:#666666
    classDef tool fill:#F3D9FF,stroke:#9C27B0,color:#9C27B0
```
