# intel/intel-extension-for-pytorch architecture

> Architecture diagram of the GitHub repository intel/intel-extension-for-pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intel/intel-extension-for-pytorch
- Repository: https://github.com/intel/intel-extension-for-pytorch

## Overview

##### Project Type and Purpose
- This is an open-source extension library (“Intel® Extension for PyTorch”) that plugs into PyTorch to deliver CPU/GPU performance optimizations, JIT compilation enhancements, and LLM-specific improvements on Intel hardware.

##### High-Level Architecture Components
a. Python Layer
- torch (core PyTorch module)
- intel_extension_for_pytorch package
– Subpackages: _dynamo, _inductor, llm, quantization, cpu.runtime, cpu.tpp, cpu.hypertune, distributed, xpu etc.
- Examples & User Scripts – Illustrate how end users invoke ipex APIs (e.g., ipex.optimize, ipex.llm.frontend).
b. Native Extension Layer
- C++ source (csrc, intel_extension_for_pytorch/csrc)
– Python bindings (pybind11)
– Modules: CPU kernels (aten), JIT codegen (LLGA, oneDNN fuser), runtime task executor, tpp kernels, quantization utils
- Build System
– CMake scripts (cmake/, CMakeLists.txt)
– Dockerfiles for pre-build environments
c. External Libraries & Hardware
- Dependencies: oneDNN, oneMKL, oneCCL, libxsmm, Sleef
- Hardware targets: Intel CPUs (AVX-512, AMX), Intel discrete GPUs (XPU/XMX)

##### Component Interactions & Data Flow
1. User Python code imports torch and ipex.
2. ipex._dynamo/_inductor intercepts Torch FX graphs → applies fusions & lowerings → generates optimized kernels.
3. Python APIs call into C++ extension via pybind11.
4. C++ extension dispatches to:
– CPU runtime (TaskExecutor → CPUPool → threaded tasks)
– Kernel implementations (csrc/cpu/aten, csrc/cpu/tpp, csrc/cpu/jit/codegen)
– External math libraries (oneDNN, MKL)
5. Results flow back up through Python boundary to user.

##### Architectural Patterns & Design Principles
- Layered modularity: clear separation between Python APIs, JIT/graph layers, native kernel implementations, and hardware interface
- Plugin/extension model: uses pybind11 to inject optimized operations into the PyTorch dispatcher
- Graph optimization pipeline: FX tracing → fusion passes → codegen → native runtime
- Parameterizable backends: dynamic dispatch (dyndisp), hardware-aware kernel selection, hypertuning

##### Capturing Essential Elements
- To include:
- Python API entry points (optimize, llm, quantization)
- JIT pipeline stages and where they live in the file tree
- Native kernel directories (csrc/cpu/aten, csrc/cpu/jit, csrc/cpu/tpp)
- Build & packaging flow (setup.py, CMake, Dockerfiles)
- External dependencies and hardware targets

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Python Layer
    subgraph "Python Layer" 
        direction TB
        Torch["torch (PyTorch core)"]:::python
        IPEX["intel_extension_for_pytorch/__init__.py"]:::python
        Scripts["Examples & User Scripts"]:::scripts
        subgraph "ipex._dynamo (JIT Interception)" 
            Dynamo["_dynamo"]:::python
        end
        subgraph "ipex._inductor (JIT Interception)" 
            Inductor["_inductor"]:::python
        end
        subgraph "ipex.llm" 
            LLM_FE["frontend.py"]:::python
            LLM_FN["functional & modules"]:::python
        end
        Quant["quantization"]:::python
        CRuntime["cpu/runtime"]:::python
        CTpp["cpu/tpp"]:::python
        CHTune["cpu/hypertune"]:::python
        Dist["distributed/dist.py"]:::python
        XPU["xpu"]:::python
    end

    Torch -->|imports| IPEX
    Scripts -->|calls ipex APIs| IPEX
    IPEX --> Dynamo
    IPEX --> Inductor
    IPEX --> LLM_FE
    IPEX --> Quant
    IPEX --> CRuntime
    IPEX --> CTpp
    IPEX --> CHTune
    IPEX --> Dist
    IPEX --> XPU

    %% Boundary to Native via pybind11
    Dynamo -->|FX graph interception| JIT
    Inductor -->|FX graph interception| JIT
    LLM_FE -->|calls| CExt
    Quant -->|calls| CExt
    CRuntime -->|calls| CExt
    CTpp -->|calls| CExt

    %% Native C++ Layer
    subgraph "Native C++ Layer" 
        direction TB
        CExt["pybind11 Extension"]:::native
        subgraph "csrc/cpu/aten" 
            ATEN["ATen CPU Kernels"]:::native
        end
        subgraph "csrc/cpu/jit" 
            JIT["JIT Codegen & Fusion"]:::native
        end
        subgraph "csrc/cpu/runtime" 
            Runtime["TaskExecutor & CPUPool"]:::native
        end
        subgraph "csrc/cpu/tpp" 
            TPP["TPP Kernels"]:::native
        end
        subgraph "Build System" 
            CMake["CMakeLists.txt & cmake/"]:::native
            Docker["docker/Dockerfile.compile & Dockerfile.prebuilt"]:::native
        end
    end

    JIT --> ATEN
    Runtime --> ATEN
    Runtime --> TPP
    Runtime -->|uses| External
    ATEN -->|links| External
    TPP -->|links| External

    CExt --> ATEN
    CExt --> JIT
    CExt --> Runtime
    CExt --> TPP

    %% External Dependencies
    subgraph "External Libraries" 
        direction TB
        oneDNN["oneDNN"]:::external
        oneMKL["oneMKL"]:::external
        oneCCL["oneCCL"]:::external
        libxsmm["libxsmm"]:::external
        Sleef["Sleef"]:::external
    end

    External[[External Math Libraries]]:::external
    External --> oneDNN
    External --> oneMKL
    External --> oneCCL
    External --> libxsmm
    External --> Sleef

    %% Hardware Targets
    subgraph "Hardware Targets" 
        direction TB
        CPU["Intel CPUs (AVX-512, AMX)"]:::hardware
        GPU["Intel GPUs (XPU/XMX)"]:::hardware
    end

    oneDNN --> CPU
    oneMKL --> CPU
    libxsmm --> CPU
    Sleef --> CPU
    External --> GPU

    %% Click Events
    click IPEX "https://github.com/intel/intel-extension-for-pytorch/blob/main/intel_extension_for_pytorch/__init__.py"
    click Dynamo "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/_dynamo"
    click Inductor "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/_inductor"
    click LLM_FE "https://github.com/intel/intel-extension-for-pytorch/blob/main/intel_extension_for_pytorch/llm/frontend.py"
    click LLM_FN "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/llm/functional"
    click Quant "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/quantization"
    click CRuntime "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/cpu/runtime"
    click CTpp "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/cpu/tpp"
    click CHTune "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/cpu/hypertune"
    click Dist "https://github.com/intel/intel-extension-for-pytorch/blob/main/intel_extension_for_pytorch/distributed/dist.py"
    click XPU "https://github.com/intel/intel-extension-for-pytorch/tree/main/intel_extension_for_pytorch/xpu"
    click ATEN "https://github.com/intel/intel-extension-for-pytorch/tree/main/csrc/cpu/aten"
    click JIT "https://github.com/intel/intel-extension-for-pytorch/tree/main/csrc/cpu/jit"
    click Runtime "https://github.com/intel/intel-extension-for-pytorch/tree/main/csrc/cpu/runtime"
    click TPP "https://github.com/intel/intel-extension-for-pytorch/tree/main/csrc/cpu/tpp"
    click CMake "https://github.com/intel/intel-extension-for-pytorch/blob/main/CMakeLists.txt"
    click Docker "https://github.com/intel/intel-extension-for-pytorch/blob/main/docker/Dockerfile.compile"

    %% Styles
    classDef python fill:#D0E6FF,stroke:#0366d6,color:#0366d6
    classDef native fill:#DFFFE0,stroke:#28A745,color:#28A745
    classDef external fill:#FFF0D0,stroke:#FF8C00,color:#FF8C00
    classDef hardware fill:#F0F0F0,stroke:#6C757D,color:#6C757D
    classDef scripts fill:#E6D0FF,stroke:#6f42c1,color:#6f42c1
```
