# fla-org/flash-linear-attention architecture

> Architecture diagram of the GitHub repository fla-org/flash-linear-attention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fla-org/flash-linear-attention
- Repository: https://github.com/fla-org/flash-linear-attention

## Overview

##### Main Components
- Core Library (fla):
– This forms the heart of the project and is composed of its own subdirectories:
▸ “layers”: Contains the implementations of different attention layers (e.g., MultiScaleRetention, GLA, Based, DeltaNet, etc.) with detailed configurations.
▸ “models”: Holds the model classes (configuration and modeling logic) that are designed to integrate with the Hugging Face Transformers library. These include many implementations like GLA, BitNet, DeltaNet, etc.
▸ “modules”: Provides additional fused activations, normalization routines, cross entropy calculations, etc.
▸ “ops”: Contains the Triton (and CUDA) kernels for various operations. This is the low-level part where performance-critical code lives and is subdivided by attention mechanism variants (e.g., delta_rule, gated_delta_rule, rwkv, etc.).
- Benchmarks, Evals, and Examples:
– “evals”: Provides evaluation harness scripts to integrate with lm-evaluation-harness.
– “examples”: Holds usage examples and guides (including training instructions).
- Legacy and Training Infrastructure:
– “legacy/training”: Older training scripts and configs, which might still be referred to.
– “flame” framework reference (mentioned in the README) integrates with torchtitan for efficient training.
- CI/CD and Infrastructure Files:
– “.github” directory with GitHub workflows and issues templates.
– Other configuration files like flake8, pre-commit-config, and setup scripts.
- Supporting Files:
– “tests”: Automated tests that cover layers, modules, and ops.
– “utils” and “scripts”: Utility functions for conversion and GPU checks.

##### Component Relationships & Interactions
- The Core Library (fla) is the central engine where layers, models, modules, and ops interact. For instance:
– Layers that wrap ops for efficient linear attention kernels.
– Models use these layers to define architectures compatible with Transformers.
- Benchmarks and evaluation harnesses (in “benchmarks” and “evals”) directly call functionality from the core library to perform performance tests and model evaluations.
- The “examples” and “legacy” sections demonstrate usage patterns of the core modules.
- CI/CD pipelines (configured under .github/workflows) make sure that tests, linting, and release tasks run continuously on different hardware targets (e.g., NVIDIA, Intel, Triton).
- Utility scripts are designed to support conversions and dependency management, linking to third-party libraries like Hugging Face Transformers and datasets.

##### Architectural Patterns & Design Principles
- Modular Design: Separation of concerns is clear with different directories dedicated to computation (ops), abstraction (layers and modules), high-level configuration (models), and testing/documentation.
- Extensibility: There is clear support for plugging in new attention mechanisms or modifying existing ones. The design is extensible with separate configurations and integration points (as seen by the hybrid models usage where standard attention modules are interleaved with linear ones).
- Hardware Efficiency: The design leverages Triton-based implementations widely (in ops) to meet specific performance goals.
- Testing & CI Integration: Comprehensive testing (with the “tests” directory) and continuous integration workflows ensure robust development and maintenance.

##### Relevant Technologies, Frameworks, and Libraries
- PyTorch – The core deep learning framework.
- Triton – For hardware-specific kernel optimization.
- Hugging Face Transformers – For integrating pretrained models and providing seamless API usage for inference and generation.
- torchtitan and flame – For training routines (as referenced in the README).
- Additional dependencies such as einops and datasets which assist in tensor operations and data handling.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Top Layer: APIs/Interfaces
    subgraph "APIs/Interfaces"
        BENCH("Benchmarking Suite"):::api
        EVAL("Evaluation Harness"):::api
        EX("Usage Examples"):::api
        LEGACY("Legacy Training"):::api
    end

    %% Middle & Lower Layer: Core Library (fla)
    subgraph "Core Library (fla)"
        MODELS("Models: Configuration & Modeling"):::core
        LAYERS("Layers: Attention Layers"):::core
        MODULES("Modules: Activations, Norm, Loss"):::core
        OPS("Ops: Triton/CUDA Kernels"):::core
    end

    %% Infrastructure and Support
    subgraph "Infrastructure and Support"
        CICD("CI/CD Pipelines"):::infra
        TESTS("Testing Framework"):::infra
        UTILS("Utility Functions"):::infra
        SCRIPTS("Scripts"):::infra
    end

    %% Relationships: Interfaces invoking Core Library
    BENCH -->|"invokes"| MODELS
    EVAL  -->|"invokes"| MODELS
    EX    -->|"invokes"| MODELS
    LEGACY-->|"invokes"| MODELS

    %% Relationships: Core Library internal dependencies
    MODELS -->|"configures"| LAYERS
    LAYERS -->|"calls"| MODULES
    LAYERS -->|"invokes"| OPS

    %% Infrastructure interactions with Core Library
    CICD   -->|"tests"| MODELS
    TESTS  -->|"verifies"| LAYERS
    UTILS  -->|"supports"| MODULES
    SCRIPTS-->|"automates"| OPS

    %% Click Events for APIs/Interfaces
    click BENCH "https://github.com/fla-org/flash-linear-attention/tree/main/benchmarks"
    click EVAL "https://github.com/fla-org/flash-linear-attention/tree/main/evals"
    click EX "https://github.com/fla-org/flash-linear-attention/tree/main/examples"
    click LEGACY "https://github.com/fla-org/flash-linear-attention/tree/main/legacy/training"

    %% Click Events for Core Library components
    click LAYERS "https://github.com/fla-org/flash-linear-attention/tree/main/fla/layers"
    click MODELS "https://github.com/fla-org/flash-linear-attention/tree/main/fla/models"
    click MODULES "https://github.com/fla-org/flash-linear-attention/tree/main/fla/modules"
    click OPS "https://github.com/fla-org/flash-linear-attention/tree/main/fla/ops"

    %% Click Events for Infrastructure and Support
    click CICD "https://github.com/fla-org/flash-linear-attention/blob/main/.github"
    click TESTS "https://github.com/fla-org/flash-linear-attention/tree/main/tests"
    click UTILS "https://github.com/fla-org/flash-linear-attention/tree/main/utils"
    click SCRIPTS "https://github.com/fla-org/flash-linear-attention/tree/main/scripts"

    %% Styles
    classDef api fill:#AED6F1,stroke:#1F618D,stroke-width:2px;
    classDef core fill:#F9E79F,stroke:#B7950B,stroke-width:2px;
    classDef infra fill:#ABEBC6,stroke:#1D8348,stroke-width:2px;
```
