How linkedin/liger-kernel works
Main Components
• Core Library:
– Source code lives in the “src/liger_kernel” directory.
– Split into several major submodules:
▪ Chunked Loss Modules (in src/liger_kernel/chunked_loss): Implements specialized loss functions (CPO, DPO, ORPO, etc.) using chunking for memory efficiency.
▪ Operations (in src/liger_kernel/ops): Contains optimized operators like cross entropy, layer norm, RoPE, etc.
▪ Transformers (in src/liger_kernel/transformers): Provides high-level APIs to automatically patch models—this includes wrappers like AutoLigerKernelForCausalLM and various model-specific patching functions.
▪ Triton Integration (in src/liger_kernel/triton): Contains the Triton kernel specific monkey patches and adjustments.
▪ Utility Components (such as env_report and utils): For environment reporting, logging, and helper functions.
• Benchmarks and Testing:
– “benchmark” and “test” directories show that there is an emphasis on performance benchmarking and convergence testing. Benchmarks validate throughput and memory usage improvements; tests ensure numerical precision and integration with frameworks.
• Examples/Demos:
– “examples” directory contains different use-case examples (Hugging Face Trainer, Lightning Trainer, Medusa, etc.), illustrating how the library integrates into end-to-end training pipelines.
• Documentation and CI/CD:
– “docs” directory holds high- and low-level API explanations, while the “.github” directory with workflows indicates continuous integration pipelines for various GPU architectures.
• External Dependencies:
– The library depends chiefly on PyTorch and Triton; optional integration with Hugging Face Transformers and third-party trainer frameworks is also highlighted.
Out Relationships and Interactions
• At the center, represent the Core Library (with its clear submodules: Chunked Loss, Ops, Transformers, Triton integration, etc.).
▪ API layers that communicate with deep learning frameworks like PyTorch.
▪ CI/CD systems (GitHub Actions workflows) in place for automated testing across AMD, NVIDIA, and Intel GPU setups.
▪ Benchmarking modules that feed performance metrics to documentation/dashboards.
• The Utilities, Configuration (setup.py, Makefile and pyproject.toml), and CI scripts show support systems that help in building, testing, and deploying.
Architectural Patterns and Design Principles
• Modular design: Clearly separated modules (e.g., chunked loss, ops, transformers) facilitate separation of concerns, making the library highly extensible and testable.
• Patching/Monkey-Patching Pattern: Automatic reconfiguration of external models via high-level patching APIs (e.g., apply_liger_kernel_to_llama) which intercept standard Hugging Face API calls.
• Performance Optimization: Use of kernel fusion, in-place operations, and chunking techniques to reduce memory overhead and boost throughput.
• Multi-Framework Integration: Supports PyTorch FSDP, DeepSpeed, etc.
Relevant Technologies and Libraries
• Primary Languages/Frameworks: Python, PyTorch, Triton.
• Dependencies: PyTorch (with specific version requirements for CUDA/ROCm), Triton (for kernel compilation), Hugging Face Transformers (for model patching), and integration frameworks like Lightning and DeepSpeed.
– “Core Library” with internal subdivisions for Chunked Loss, Ops, Transformers, and Triton Integration.
– “High-Level APIs & Patching” showing AutoLigerKernelForCausalLM and model-specific patching functions.
– “Examples & Use Cases” with distinct examples for different trainers (Hugging Face Trainer, Lightning, etc.).
– “Benchmarking & Testing” that covers both performance benchmarks and unit tests.
– “CI/CD and Documentation” supporting automated build pipelines and user guides.
– Clearly mark configuration files and external pipelines (like CI workflows) in a distinct style to illustrate their supportive role.
• Separate blocks for Examples, Testing/Benchmarking, and CI/CD.