How nvlabs/nvdiffrast works
1. What this repo is
- nvdiffrast is a PyTorch extension library for high-performance differentiable rendering.
- Its core purpose is to expose a small set of GPU-accelerated primitives for rasterization-based rendering workflows.
2. Top-level architecture
- The repo is split into:
- Python package layer: nvdiffrast/
- Torch binding layer: csrc/torch/
- Shared native rendering core: csrc/common/
- Examples and documentation: samples/, docs/
- The Python side is thin; most heavy lifting happens in compiled C++/CUDA code.
3. Public Python API boundary
- The package entry point is nvdiffrast/torch/ops.py and nvdiffrast/torch/__init__.py.
- This layer presents the operations used by training code, while hiding the native extension details.
- The library is specifically positioned as a PyTorch-first API, so the main consumer is model code that wants differentiable rendering in a deep learning loop.
4. Torch binding layer
- csrc/torch/torch_bindings.cpp and related files connect Python/Torch calls to native implementations.
- Separate binding translation units exist for the main ops:
- torch_rasterize.cpp
- torch_interpolate.cpp
- torch_texture.cpp
- torch_antialias.cpp
- This indicates a modular op-by-op extension design rather than one monolithic kernel interface.
5. Core rendering primitives
- The compiled backend in csrc/common/ implements the actual rendering primitives:
- rasterization: rasterize.cu
- interpolation: interpolate.cu
- texture sampling: texture.cpp, texture_kernel.cu
- antialiasing: antialias.cu
- These are the computational kernels that the Torch bindings expose to Python.
- The primitives are designed to be combined into a rendering pipeline rather than a full engine.
6. CUDA rasterization subsystem
- csrc/common/cudaraster/ contains a dedicated rasterization implementation with its own internal structure:
- raster implementation logic in impl/
- kernel code in RasterImpl_kernel.cu
- setup, coarse/fine/bin raster stages, buffer management, and utility helpers
- This is the main low-level GPU rasterizer backend and likely the most performance-critical subsystem in the repo.
7. Data flow through the system
- Typical flow is:
- Python/Torch code calls a high-level op
- Torch binding marshals tensors and parameters into native code
- CUDA/C++ core rasterizes geometry and computes auxiliary data
- downstream ops interpolate attributes, sample textures, and optionally antialias
- results are returned as Torch tensors for autograd-based training
- The architecture is centered on tensor in / tensor out GPU computation.
8. Differentiability boundary
- Because the repo is for differentiable rendering, the important boundary is not just rendering, but rendering with gradients.
- The native ops are packaged as PyTorch extension operations so they can participate in training workflows.
- The repo therefore sits between classic graphics rasterization and ML autograd infrastructure.
9. Build and packaging
- setup.py, pyproject.toml, and MANIFEST.in define packaging and extension build integration.
- Dockerfile suggests a reproducible environment for CUDA/PyTorch builds.
- The build is native-extension oriented, meaning deployment depends on compiling the C++/CUDA code against the local PyTorch/CUDA toolchain.
10. Examples and validation
- samples/torch/ provides runnable example pipelines:
- triangle.py
- pose.py
- cube.py
- earth.py
- envphong.py
- These are important as architectural references because they show how the primitives are composed in actual rendering tasks.
- samples/data/ holds the assets used by the demos.
11. Documentation and external interface
- docs/index.html and docs/img/ provide the user-facing documentation site and visuals.
- The README positions the library as a research-grade NVIDIA release and points to external docs for detailed API usage.
12. Key architectural takeaway
- The repo is a layered GPU extension:
- Python/Torch API on top
- Torch C++ bindings in the middle
- CUDA/C++ rendering kernels at the bottom
- The system is not a full renderer application; it is a modular primitive library meant to be embedded inside larger PyTorch-based differentiable rendering systems.
Main entry points: torch_bindings.cpp Main bindings · common.cpp Common core · ops.py Torch API · setup.py Build
Components 20
Python API
Public PyTorch-facing package layer.
- nvdiffrast · Top-level package entry point.
nvdiffrast/__init__.py - Torch API · Public PyTorch-facing operations.
nvdiffrast/torch/ops.py - Torch init · Package export surface for torch integration.
nvdiffrast/torch/__init__.py
Torch bindings
C++ bridge between Python and native ops.
- Main bindings · Entry point that registers native ops with PyTorch.
csrc/torch/torch_bindings.cpp - Raster bind · Torch wrapper for rasterization.
csrc/torch/torch_rasterize.cpp - Interp bind · Torch wrapper for interpolation.
csrc/torch/torch_interpolate.cpp - Texture bind · Torch wrapper for texture sampling.
csrc/torch/torch_texture.cpp - AA bind · Torch wrapper for antialiasing.
csrc/torch/torch_antialias.cpp
Native core
GPU rendering primitives and rasterizer backend.
- Common core · Shared native helpers and glue code.
csrc/common/common.cpp - Rasterize · Geometry rasterization primitive.
csrc/common/rasterize.cu - Interpolate · Attribute interpolation primitive.
csrc/common/interpolate.cu - Texture · Texture sampling implementation.
csrc/common/texture.cpp - Texture kernel · Low-level texture sampling kernel.
csrc/common/texture_kernel.cu - Antialias · Edge smoothing and cleanup primitive.
csrc/common/antialias.cu - CUDA raster · Dedicated low-level rasterization subsystem.
csrc/common/cudaraster/CudaRaster.hpp - Raster impl · Internal raster pipeline and orchestration.
csrc/common/cudaraster/impl/RasterImpl.cpp - Raster kernel · Core rasterizer kernel implementation.
csrc/common/cudaraster/impl/RasterImpl_kernel.cu
Connections 23
- nvdiffrast → Torch API: exposes · Package surface routes users to torch ops.
- Torch init → Torch API: exports · Initialization layer re-exports public ops.
- Torch API → Main bindings: calls · Python API dispatches into registered extension ops.
- Main bindings → Raster bind: routes · Main registration layer forwards raster op calls.
- Main bindings → Interp bind: routes · Main registration layer forwards interpolation op calls.
- Main bindings → Texture bind: routes · Main registration layer forwards texture op calls.
- Main bindings → AA bind: routes · Main registration layer forwards antialias op calls.
- Raster bind → Rasterize: invokes · Torch raster binding reaches native rasterization core.
- Interp bind → Interpolate: invokes · Torch interpolation binding reaches native core.
- Texture bind → Texture: invokes · Torch texture binding reaches sampling core.
- AA bind → Antialias: invokes · Torch antialias binding reaches native primitive.
- Common core → Rasterize: supports · Shared helpers underpin rasterization and adjacent ops.
- Common core → Interpolate: supports · Shared helpers underpin interpolation.
- Common core → Texture: supports · Shared helpers underpin texture sampling.
- Common core → Antialias: supports · Shared helpers underpin antialiasing.
- Texture → Texture kernel: uses · Texture wrapper delegates to low-level GPU kernel.
- CUDA raster → Raster impl: contains · Raster backend is organized around internal implementation.
- Raster impl → Raster kernel: drives · Raster implementation coordinates the core kernel.
- Samples → Torch API: demonstrates · Examples show how to compose the public API.
- Docs → Torch API: documents · Documentation explains the exposed torch-facing interface.
- Build → Main bindings: builds · Packaging compiles and links the native extension.
- Build → Common core: builds · Packaging includes shared native rendering core.
- Build → CUDA raster: builds · Packaging includes the CUDA raster backend.