How tile-ai/tilelang works
1. TileLang is a Python-first kernel DSL and compiler stack, not a general application framework. The primary user entrypoint is the tilelang package, where developers write annotated Python functions that describe tiled GPU/CPU kernels such as GEMM, attention, quantization, and sparse operators.
2. The frontend lives in tilelang/language and tilelang/ir. It provides the DSL syntax and AST/IR nodes for tensors, loops, allocations, copies, reductions, pipelining, warp/group primitives, and target-specific annotations. This layer is what turns “Pythonic kernel code” into a structured compiler IR.
3. Compilation is centered in tilelang/engine, tilelang/transform, and tilelang/backend. The engine performs semantic checks and lowering orchestration; the transform passes rewrite and legalize the IR; the backend chooses a target-specific codegen/execution path. This is the main compiler pipeline boundary.
4. The C++ core in src/ implements the lower-level compiler and runtime pieces. src/op contains generic operator implementations and helpers, src/transform contains IR passes, and src/runtime contains logging/error utilities. This code is the performance-critical compiler/runtime substrate beneath the Python API.
5. Backend specialization is explicit. There are separate stacks for cuda, rocm, cpu, metal, and webgpu, each with its own codegen, target utilities, runtime hooks, and operator implementations. That means the architecture is multi-backend by design rather than “CUDA-only with portability shims.”
6. NVIDIA-specific lowering is the most extensive backend. src/cuda and tilelang/cuda include CUDA codegen, PTX/intrinsic rules, runtime stubs, warp-group/TMA/WGMMA handling, and transforms for Hopper/Blackwell-era features. There is also a tilelang/contrib/cutedsl path and src/cuda/codegen/codegen_cutedsl.cc, showing an alternate code generation route to CUTLASS CuTe DSL.
7. AMD support is a parallel architecture, not an afterthought. src/rocm, tilelang/rocm, and 3rdparty/hip-headers provide HIP/ROCm codegen and MFMA/WMMA intrinsic support. The repo also ships AMD-specific examples and tests, so AMD compilation, lowering, and runtime behavior are first-class.
8. The Python runtime and JIT layer sits between user kernels and compiled shared libraries. tilelang/jit contains adapters for NVRTC, Cython, CuTeDSL, Torch, and TVM FFI, plus kernel caching and wrapper generation. This layer is responsible for compiling on demand, loading the result, and presenting callable kernels back to Python.
9. Execution is cached and target-aware. tilelang/cache and the JIT adapter cache modules store compiled artifacts, while tilelang/backend/execution_backend.py and target modules manage how kernels are launched. The data flow is: user kernel → IR/lowers → backend codegen → compiled artifact → cached loader/execution.
10. Tile-specific performance modeling and operator synthesis live in tilelang/carver and tilelang/tileop. carver analyzes shapes/architectures and suggests schedules or hints; tileop holds reusable operator templates/registries such as GEMM and sparse GEMM. These subsystems support autotuning and template-driven specialization.
11. Higher-level algorithm support is organized into reusable operator families in tilelang/cuda/op, tilelang/rocm/op, tilelang/cpu/op, tilelang/metal/op, and the corresponding Python wrappers. This separates “what the operator does” from “how it is lowered for each device class,” which is important for diagramming the architecture cleanly.
12. The repository includes a substantial verification and regression layer. testing/ contains compiler-pass tests, backend tests, runtime tests, JIT tests, and issue regressions; examples/ contains end-to-end operator demonstrations and benchmarks; benchmark/ holds reproducible performance scripts. These directories form the quality and performance validation boundary around the compiler.
13. Build and distribution are managed as a mixed Python/C++ project. CMakeLists.txt, the cmake/ helpers, pyproject.toml, and the CI/docker files coordinate native compilation, Python packaging, wheel builds, docs, and backend-specific environment setup. The vendored 3rdparty/tvm, CUTLASS, composable kernel, and HIP headers are foundational dependencies that materially shape the compiler and backend architecture.
Main entry points: backend Backend API · cuda CUDA Core · rocm ROCm Core · CMakeLists.txt Build System
Components 23
Frontend DSL
Python-facing kernel language, AST/IR, and layout abstractions.
- tilelang · Top-level Python API for authoring tiled kernels and invoking compilation/execution.
tilelang/__init__.py - Language DSL · Annotated kernel syntax, parser, AST helpers, loop/copy/reduce primitives, and eager/TIR entrypoints.
tilelang/language - IR & Layout · Structured kernel IR plus tensor/layout abstractions used by the compiler and backends.
tilelang/ir.py - TileOps · Reusable operator registries and templates such as GEMM and sparse GEMM.
tilelang/tileop - Carver · Performance analysis, shape/arch reasoning, and schedule hint generation for autotuning.
tilelang/carver
Compiler Pipeline
Semantic checks and lowering/rewriting passes that turn DSL kernels into target IR.
- Engine · Semantic checks, lowering entrypoints, and callback hooks for compilation.
tilelang/engine - Python Passes · High-level IR transforms and legality rewrites before backend specialization.
tilelang/transform - Backend API · Target selection, device codegen glue, and execution backend interfaces.
tilelang/backend
Backends
Target-specific codegen, execution, intrinsics, and operator families.
- CUDA Stack · Python CUDA codegen, execution backend, intrinsics, and CUDA operators.
tilelang/cuda - ROCm Stack · Python ROCm codegen, execution backend, intrinsics, and HIP operators.
tilelang/rocm - CPU Stack · Python CPU codegen, execution backend, and CPU operator implementations.
tilelang/cpu - Metal Stack · Python Metal codegen, execution backend, and Metal operator implementations.
tilelang/metal - WebGPU Stack · Python WebGPU codegen and pipeline hooks.
tilelang/webgpu
JIT and Runtime
On-demand compilation, caching, loading, and execution wrappers.
- JIT Layer · Adapters for NVRTC, Cython, CuTeDSL, Torch, and TVM FFI plus wrapper generation and kernel loading.
tilelang/jit - Kernel Cache · Stores compiled artifacts and compiled-kernel metadata for reuse.
tilelang/cache
Native Core
C++ compiler/runtime substrate and backend-specific lowering implementations.
- Core Ops · Shared operator implementations and helpers used across targets.
src/op - Core Transforms · Lower-level IR rewrites, legalization, pipeline planning, and layout inference in C++.
src/transform - CUDA Core · CUDA codegen, intrinsics, transforms, runtime stubs, and CUDA operator lowerings.
src/cuda - ROCm Core · ROCm codegen, intrinsics, transforms, runtime stubs, and HIP operator lowerings.
src/rocm - Runtime Core · Logging, error helpers, and support utilities for generated code/runtime execution.
src/runtime
Validation
Tests, benchmarks, and examples that exercise the stack end to end.
- Examples & Tests · End-to-end examples, regressions, backend tests, and performance benchmarks.
examples - Build System · CMake, Python packaging, CI, and container setup for mixed Python/C++ builds.
CMakeLists.txt - Vendored Deps · Embedded dependencies shaping compiler and backend behavior, including TVM, CUTLASS, composable kernel, and HIP headers.
3rdparty
Connections 26
- tilelang → Language DSL: uses DSL · Users author kernels through the Python language layer.
- Language DSL → IR & Layout: builds IR · The frontend turns Pythonic kernel code into structured IR and layouts.
- IR & Layout → Engine: lowered by · IR is handed to engine semantic checks and lowering.
- Engine → Python Passes: applies passes · The engine orchestrates high-level rewriting and legalization passes.
- Python Passes → Backend API: targets backends · Transforms prepare the IR for backend-specific codegen and execution.
- Backend API → JIT Layer: executes via · Backends feed compiled artifacts into the JIT/load path.
- JIT Layer → Kernel Cache: stores artifacts · Compiled kernels are cached for reuse across invocations.
- Kernel Cache → Backend API: reuses target code · Cached artifacts remain target-aware and are loaded through backend execution.
- Backend API → CUDA Stack: dispatches · CUDA is a primary specialized backend path.
- Backend API → ROCm Stack: dispatches · ROCm is a first-class specialized backend path.
- Backend API → CPU Stack: dispatches · CPU codegen and execution are supported as a separate backend.
- Backend API → Metal Stack: dispatches · Metal support is integrated through its own backend stack.
- Backend API → WebGPU Stack: dispatches · WebGPU support follows its own codegen path.
- TileOps → Carver: guided by · Operator templates can be analyzed and tuned with schedule hints.
- TileOps → Engine: specialized by · TileOps feed reusable operator structure into compilation.
- Core Ops → Core Transforms: consumed by · Core operator semantics are rewritten by native transform passes.
- Core Transforms → CUDA Core: specialized in · CUDA-specific lowering adds async copy, barriers, tensor cores, and warp-group features.
- Core Transforms → ROCm Core: specialized in · ROCm-specific lowering adds HIP/MFMA/WMMA handling.
- CUDA Core → Core Ops: implements · CUDA operator lowerings back shared op abstractions.
- ROCm Core → Core Ops: implements · ROCm operator lowerings back shared op abstractions.
- CUDA Core → Runtime Core: uses runtime · CUDA runtime hooks rely on common runtime support.
- ROCm Core → Runtime Core: uses runtime · ROCm runtime hooks rely on common runtime support.
- Examples & Tests → tilelang: exercises · Examples and tests validate the public Python API and end-to-end compilation.
- Build System → Vendored Deps: pulls in · Build and packaging are shaped by vendored native dependencies.
- Build System → CUDA Core: builds · Native build configuration compiles backend-specific C++ code.
- Build System → ROCm Core: builds · Native build configuration compiles ROCm-specific C++ code.