# nvidia/fuser architecture

> Architecture diagram of the GitHub repository nvidia/fuser, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/fuser
- Repository: https://github.com/nvidia/fuser

## Overview

##### Project Type and Purpose
- nvFuser is a specialized compiler or code generator designed to fuse operations and generate optimized GPU kernels (specifically for NVIDIA GPUs).
- It serves as the backend for efficiently lowering high-level (often Python-initiated) operations (e.g., from PyTorch) into optimized CUDA kernels.
- The project is multi-faceted, featuring core C++ components, a Python frontend (bindings and support scripts), a CUDA runtime layer, extensive benchmarks, and testing suites.

##### Analyzing the File Structure
- Core Source Directories:
- “csrc”: Contains the bulk of the C++ source code including IR construction (csrc/ir), optimization passes (csrc/device_lower, csrc/preseg_passes, csrc/scheduler), code generation (csrc/codegen), and other supporting utilities.
- “runtime”: Holds CUDA kernels and runtime utilities that manage kernel launching, memory, and device-specific operations.
- “nvfuser” (top level): Provides Python package files (nvfuser/__init__.py, etc.) that interface with the C++ backend.
- “csrc/python_frontend”: Implements bindings and translation layers for Python consumption.
- Supporting Directories:
- “benchmarks” and “tests”: For performance benchmarking and regression testing (both C++ and Python tests).
- “lib/dynamic_type”: Contains a library that appears generic for dynamic type handling which may be used in type dispatching or runtime handling.
- “tools”, “doc”, and “third_party”: Supporting documentation, examples, CI/CD configurations, and external dependency integrations.

##### Main Components to Highlight
a. Frontend / User Interface:
- Python Frontend (nvfuser Python package and csrc/python_frontend): Handles API calls from user code (potentially through PyTorch integration).
- Externally, it interacts with a high-level framework (e.g. user’s PyTorch code) and translates fusion operations to the core engine.
b. Fusion Compiler Core:
- IR Construction and Transformation: Represented by directories such as csrc/ir, which builds and manages the intermediate representation of the fusion operations.
- Optimization and Scheduling: Directories like csrc/scheduler, csrc/preseg_passes, and csrc/device_lower implement algorithms for scheduling, fusion, lowering, and other optimization passes.
- Code Generation: Components such as csrc/codegen produce the final CUDA kernel code from the optimized IR.
c. Runtime / Execution Environment:
- CUDA Runtime & Kernel Executor: Provided by the “runtime” directory with CUDA (.cu) files handling kernel invocation, allocation, synchronization, and low-level device operations.
d. Benchmarks and Testing:
- Benchmark Suites (benchmarks folder) and Test Suites (tests folder in both C++ and Python) that validate performance and correctness.

##### Relationships and Interactions
- These calls invoke the core C++ engine, which constructs the IR, applies multiple optimization and scheduling passes, and then transitions into code generation.
- The generated CUDA kernels are passed to the runtime layer where they are compiled/executed.
- Tools and CI/CD scripts (GitHub workflows, linter configurations, etc.) are part of the development “ecosystem” and can be depicted to show how code quality and automated testing support the core components.

##### Architectural Patterns and Design Principles
- Layered Architecture: Separate layers for the high-level frontend, the mid-level compiler/optimizer engine, and the low-level runtime.
- Modular / Pass-based Compiler Design: Emphasize the various compiler passes (IR simplification, scheduling, lowering, code generation) that mirror patterns in modern compilers.
- Bindings and Interoperability: The design clearly separates language integration (Python frontend) from performance-critical C++ and CUDA code.
- Extensive Testing & Benchmarking: Illustrate how benchmark and testing modules ensure correctness and performance, feeding back into the iterative development process.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        F1["nvFuser Package"]:::frontend
        F2["Python Bindings"]:::frontend
    end

    %% Compiler Core Layer
    subgraph "Compiler Core"
        %% IR Construction
        IR1["IR Construction"]:::core
        
        %% Optimization & Scheduling
        subgraph "Optimization & Scheduling"
            S1["Scheduler"]:::core
            S2["Preseg Passes"]:::core
            S3["Device Lower"]:::core
        end
        
        %% Code Generation
        subgraph "Code Generation"
            CG1["codegen.cpp"]:::core
            CG2["codegen.h"]:::core
        end
    end

    %% Runtime Environment
    R1["CUDA Runtime & Kernel Executor"]:::runtime

    %% Testing & Benchmarking
    subgraph "Support & Testing"
        T1["Benchmark Suites"]:::testing
        T2["Test Suites"]:::testing
    end

    %% Connections
    F1 -->|"API_call"| IR1
    F2 -->|"API_call"| IR1
    IR1 -->|"build_IR"| S1
    S1 -->|"optimize"| S2
    S2 -->|"schedule"| S3
    S3 -->|"generate_code"| CG1
    CG1 -->|"header_dependency"| CG2
    CG2 -->|"final_kernel"| R1
    R1 -->|"performance_data"| T1
    R1 -->|"test_results"| T2
    T1 -->|"feedback"| IR1
    T2 -->|"feedback"| IR1

    %% Click Events
    click F1 "https://github.com/nvidia/fuser/blob/main/nvfuser/__init__.py"
    click F2 "https://github.com/nvidia/fuser/tree/main/csrc/python_frontend"
    click IR1 "https://github.com/nvidia/fuser/tree/main/csrc/ir"
    click S1 "https://github.com/nvidia/fuser/tree/main/csrc/scheduler"
    click S2 "https://github.com/nvidia/fuser/tree/main/csrc/preseg_passes"
    click S3 "https://github.com/nvidia/fuser/tree/main/csrc/device_lower"
    click CG1 "https://github.com/nvidia/fuser/blob/main/csrc/codegen.cpp"
    click CG2 "https://github.com/nvidia/fuser/blob/main/csrc/codegen.h"
    click R1 "https://github.com/nvidia/fuser/tree/main/runtime"
    click T1 "https://github.com/nvidia/fuser/tree/main/benchmarks"
    click T2 "https://github.com/nvidia/fuser/tree/main/tests"

    %% Styles
    classDef frontend fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef runtime fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef testing fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
