How shi-labs/natten works
Project Type and Purpose
• This is a hybrid language library (Python + C++/CUDA) that implements Neighborhood Attention and Fused Neighborhood Attention (FNA) kernels as a PyTorch extension.
• High-level goal: expose fast, multi-dimensional sliding-window self-attention ops to Python users, with both CPU and GPU backends, packaged for pip.
Top-Level Components
A. Python Layer
• src/natten: the public API, Python wrappers for ops, autodiff integration, autotuner.
• setup.py, setup.cfg, Makefile: drive build, install, and wheel packaging.
• dev/packaging/*, dev/docker: scripts and Dockerfiles to build/test wheels for many CUDA/CPU variants.
• docs/, webpage/: user-facing documentation (Markdown) and static site assets.
• tests/: unit tests for Python and extension behaviors.
B. Native Extension Layer (csrc/)
• csrc/CMakeLists.txt + setup.py → builds a shared library (“natten” extension).
• csrc/include/: public & internal headers; PyTorch binding headers in include/natten/pytorch.
• csrc/src/: glue code (.cpp/.cu) registering forward/backward kernels with PyTorch.
• csrc/autogen/: kernel implementations generated by Python scripts in scripts/*.py.
C. Backends and Kernels
• CPU naive kernels (csrc/autogen/src/cpu/naive + csrc/include/natten/cpu).
• CUDA kernels:
– Naive, GEMM-based, and FNA variants under csrc/autogen/src/cuda/* and csrc/include/natten/cuda.
– Third-party CUTLASS library as a submodule for GEMM support (third_party/cutlass).
D. Tooling and Support
• scripts/autogen_*.py: generate specialized CUDA/C++ kernel code.
• tools/: Python scripts for profiling and benchmarking.
• docs/, docs/fna/: architecture and usage guides.
Key Relationships and Data Flows
User Python code
↓ calls
src/natten API (Python)
↓ dispatch (based on device & config)
PyTorch C++ extension (natten.cpp + CMake build)
↓ invokes
– CPU kernel or
– CUDA kernel (naive / gemm / fna)
↓ executes on target hardware
↓ returns tensor to Python
↓ backward pass follows same flow in reverse (registered backward kernels)
Autotuner (Python)
↔ reads/writes config
↔ benchmarks different kernel variants
↔ influences dispatch choices in API
Architectural Patterns and Principles
• Layered separation:
– Python API layer ←→ Native extension layer ←→ Hardware kernels
• Plug-in/backends: CPU vs. CUDA vs. GEMM vs. FNA → selected at runtime.
• Code generation: scripts auto-generate optimized, architecture-specific kernels.
• Polymorphic dispatch: use PyTorch’s registration mechanism to select forward/backward.
• Modular packaging: build scripts orchestrate multi-platform wheel builds.