# linkedin/liger-kernel architecture

> Architecture diagram of the GitHub repository linkedin/liger-kernel, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/linkedin/liger-kernel
- Repository: https://github.com/linkedin/liger-kernel

## Overview

##### Main Components
- Core Library:
– Source code lives in the “src/liger_kernel” directory.
– Split into several major submodules:
▪ Chunked Loss Modules (in src/liger_kernel/chunked_loss): Implements specialized loss functions (CPO, DPO, ORPO, etc.) using chunking for memory efficiency.
▪ Operations (in src/liger_kernel/ops): Contains optimized operators like cross entropy, layer norm, RoPE, etc.
▪ Transformers (in src/liger_kernel/transformers): Provides high-level APIs to automatically patch models—this includes wrappers like AutoLigerKernelForCausalLM and various model-specific patching functions.
▪ Triton Integration (in src/liger_kernel/triton): Contains the Triton kernel specific monkey patches and adjustments.
▪ Utility Components (such as env_report and utils): For environment reporting, logging, and helper functions.
- Benchmarks and Testing:
– “benchmark” and “test” directories show that there is an emphasis on performance benchmarking and convergence testing. Benchmarks validate throughput and memory usage improvements; tests ensure numerical precision and integration with frameworks.
- Examples/Demos:
– “examples” directory contains different use-case examples (Hugging Face Trainer, Lightning Trainer, Medusa, etc.), illustrating how the library integrates into end-to-end training pipelines.
- Documentation and CI/CD:
– “docs” directory holds high- and low-level API explanations, while the “.github” directory with workflows indicates continuous integration pipelines for various GPU architectures.
- External Dependencies:
– The library depends chiefly on PyTorch and Triton; optional integration with Hugging Face Transformers and third-party trainer frameworks is also highlighted.

##### Out Relationships and Interactions
- At the center, represent the Core Library (with its clear submodules: Chunked Loss, Ops, Transformers, Triton integration, etc.).
▪ API layers that communicate with deep learning frameworks like PyTorch.
▪ CI/CD systems (GitHub Actions workflows) in place for automated testing across AMD, NVIDIA, and Intel GPU setups.
▪ Benchmarking modules that feed performance metrics to documentation/dashboards.
- The Utilities, Configuration (setup.py, Makefile and pyproject.toml), and CI scripts show support systems that help in building, testing, and deploying.

##### Architectural Patterns and Design Principles
- Modular design: Clearly separated modules (e.g., chunked loss, ops, transformers) facilitate separation of concerns, making the library highly extensible and testable.
- Patching/Monkey-Patching Pattern: Automatic reconfiguration of external models via high-level patching APIs (e.g., apply_liger_kernel_to_llama) which intercept standard Hugging Face API calls.
- Performance Optimization: Use of kernel fusion, in-place operations, and chunking techniques to reduce memory overhead and boost throughput.
- Multi-Framework Integration: Supports PyTorch FSDP, DeepSpeed, etc.

##### Relevant Technologies and Libraries
- Primary Languages/Frameworks: Python, PyTorch, Triton.
- Dependencies: PyTorch (with specific version requirements for CUDA/ROCm), Triton (for kernel compilation), Hugging Face Transformers (for model patching), and integration frameworks like Lightning and DeepSpeed.

– “Core Library” with internal subdivisions for Chunked Loss, Ops, Transformers, and Triton Integration.
– “High-Level APIs & Patching” showing AutoLigerKernelForCausalLM and model-specific patching functions.
– “Examples & Use Cases” with distinct examples for different trainers (Hugging Face Trainer, Lightning, etc.).
– “Benchmarking & Testing” that covers both performance benchmarks and unit tests.
– “CI/CD and Documentation” supporting automated build pipelines and user guides.
– Clearly mark configuration files and external pipelines (like CI workflows) in a distinct style to illustrate their supportive role.

- Separate blocks for Examples, Testing/Benchmarking, and CI/CD.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top-level external nodes
    EX["Examples & Use Cases"]:::external
    API["High-Level APIs & Patching"]:::api
    ED["External Dependencies"]:::external

    %% Core Library Root node for overall source code
    CL_ROOT["Core Library"]:::core

    %% Subgraph for Core Library Modules
    subgraph "Core Library Modules"
        CL1["Chunked Loss Modules"]:::module
        CL2["Operations"]:::module
        CL3["Transformers"]:::module
        CL4["Triton Integration"]:::module
        CL5["env_report.py"]:::util
        CL6["utils.py"]:::util
    end

    %% Subgraph for Benchmarking & Testing
    subgraph "Benchmarking & Testing"
        BMT1["Benchmarking Modules"]:::external
        BMT2["Testing Modules"]:::external
    end

    %% Subgraph for CI/CD & Documentation
    subgraph "CI/CD & Documentation"
        CI1["CI/CD Pipelines"]:::external
        CI2["Documentation"]:::external
    end

    %% Connections between components
    EX -->|"invokes"| API
    ED -->|"supports"| API
    API -->|"calls"| CL_ROOT
    CL_ROOT -->|"contains"| CL1
    %% Specific high-level API directly patches the Transformers module
    API -->|"patches"| CL3
    ED -->|"enables"| CL4
    %% Benchmark feedback to core operations
    BMT1 -->|"feedback"| CL2
    BMT2 -->|"feedback"| CL2
    %% CI triggers build/test and monitors benchmarks
    CI1 -->|"builds"| CL1
    CI1 -->|"monitors"| BMT1
    CI1 -->|"monitors"| BMT2

    %% Click Events for component mappings
    click CL_ROOT "https://github.com/linkedin/liger-kernel/tree/main/src/liger_kernel"
    click CL1 "https://github.com/linkedin/liger-kernel/tree/main/src/liger_kernel/chunked_loss"
    click CL2 "https://github.com/linkedin/liger-kernel/tree/main/src/liger_kernel/ops"
    click CL3 "https://github.com/linkedin/liger-kernel/tree/main/src/liger_kernel/transformers"
    click CL4 "https://github.com/linkedin/liger-kernel/tree/main/src/liger_kernel/triton"
    click CL5 "https://github.com/linkedin/liger-kernel/blob/main/src/liger_kernel/env_report.py"
    click CL6 "https://github.com/linkedin/liger-kernel/blob/main/src/liger_kernel/utils.py"
    click BMT1 "https://github.com/linkedin/liger-kernel/tree/main/benchmark"
    click BMT2 "https://github.com/linkedin/liger-kernel/tree/main/test"
    click EX "https://github.com/linkedin/liger-kernel/tree/main/examples"
    click CI1 "https://github.com/linkedin/liger-kernel/tree/main/.github/workflows"
    click CI2 "https://github.com/linkedin/liger-kernel/tree/main/docs"

    %% Style Definitions
    classDef core fill:#FFD700,stroke:#B8860B,stroke-width:2px;
    classDef module fill:#ADD8E6,stroke:#00008B,stroke-width:2px;
    classDef util fill:#90EE90,stroke:#006400,stroke-width:2px;
    classDef api fill:#FFB6C1,stroke:#FF69B4,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#696969,stroke-width:2px;
```
