# feifeibear/long-context-attention architecture

> Architecture diagram of the GitHub repository feifeibear/long-context-attention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/feifeibear/long-context-attention
- Repository: https://github.com/feifeibear/long-context-attention

## Overview

##### Project Type and Purpose
- This is a Python library/tool (not a web app) for distributed attention in long-context LLM training and inference.
- It implements a hybrid (“Unified Sequence Parallelism”) approach that combines DeepSpeed-Ulysses and ring-based attention to maximize scalability and performance.

##### Key Components from File Structure
- Top-level:
- pyproject.toml, README.md, LICENSE, CI workflows, scripts/, docs/, benchmark/, test/
- Core Package (yunchang/):
- yunchang/comm/  – process-group setup & tensor-extraction helpers
- yunchang/hybrid/ – high-level attention layers (AsyncLongContextAttention, LongContextAttention)
- yunchang/kernels/ – attention kernel implementations (FlashAttention wrappers, Torch fallback)
- yunchang/ring/  – multiple ring-attention variants (basic, stripe, zigzag, varlen, FlashInfer)
- yunchang/ulysses/ – Ulysses-style all-to-all attention layer
- yunchang/globals.py – global config, enums (AttnType)
- Scripts & Benchmarks:
- scripts/*.sh – quick start and benchmark runners
- benchmark/ – performance comparison scripts
- Tests:
- test/*.py – unit/integration tests exercising each implementation

##### Insights from README
- Supports FlashAttention v2/v3 or pure-Torch fallback
- Offers API set_seq_parallel_pg + EXTRACT_FUNC_DICT for sharding QKV
- Uses strategy pattern: choose ring_impl_type and attn_type at runtime
- Integrates upstream (Megatron-LM, NVIDIA TransformerEngine) via patch

##### Main Components
a. API Layer (User-facing)
- set_seq_parallel_pg() – configures process groups for Ulysses & Ring parallelism
- Extraction functions (EXTRACT_FUNC_DICT) – permute & shard QKV
- LongContextAttention / AsyncLongContextAttention – entry point for attention call
b. Hybrid Controller
- Decision logic: split heads → some to Ulysses module, some to Ring module
c. Ulysses Module
- All-to-all communication for head-parallel attention
- Built with yunchang/ulysses/attn_layer.py
d. Ring Module
- Multiple implementations (basic, stripe, zigzag, varlen, FlashInfer)
- yunchang/ring/* uses yunchang/ring/triton_utils.py + flash-attn or torch
e. Kernel Layer
- FlashAttention wrappers (FA v2, FA v3)
- Torch fallback
f. Communication Layer
- yunchang/comm/all_to_all.py – collective ops for Ulysses
- yunchang/comm/extract_local.py – sharding logic for ring
g. External Dependencies
- PyTorch
- flash-attn (optional)
- SpargeAttn, SageAttention, FlashInfer (optional integrations)

##### Component Relationships and Data Flow
- User code calls set_seq_parallel_pg → PGs created
- Q, K, V tensors → EXTRACT_FUNC_DICT → local shards
- LongContextAttention receives local shards, routes to Ulysses and/or Ring submodules
- Each submodule calls into kernels (FlashAttention or Torch) and comm primitives
- Results from submodules are gathered (all-to-all or ring-collect) → merged and returned

##### Architectural Patterns & Principles
- Strategy/Factory Pattern: pluggable attn_type and ring_impl_type
- Modular layers: clear separation between comm, kernel, control logic
- Extensibility: new ring variants, new kernel backends can be added under well-defined interfaces

##### Essential Elements
- clear component names (LongContextAttention, RingAttention variants, UlyssesAttention)
- process group configuration (Ulysses PG, Ring PG)
- external libraries (torch, flash-attn, Triton)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% API Layer
    subgraph "API Layer"
        style API fill:#AEDFF7
        SPG["set_seq_parallel_pg()"]:::api
        EX["EXTRACT_FUNC_DICT"]:::api
        LCA["LongContextAttention"]:::api
        ACA["AsyncLongContextAttention"]:::api
    end

    %% Hybrid Controller
    subgraph "Hybrid Controller"
        style HC fill:#C8E6C9
        HC["HybridController\n(routing logic)"]:::core
    end

    %% Ulysses Module
    subgraph "Ulysses Module"
        style UM fill:#C8E6C9
        UA["UlyssesAttention"]:::core
    end

    %% Ring Module
    subgraph "Ring Module"
        style RM fill:#C8E6C9
        RPT["ring_pytorch_attn"]:::core
        RFA["ring_flash_attn"]:::core
        RFV["ring_flash_attn_varlen"]:::core
        SFA["stripe_flash_attn"]:::core
        ZFA["zigzag_ring_flash_attn"]:::core
        ZFV["zigzag_ring_flash_attn_varlen"]:::core
        RFI["ring_flashinfer_attn"]:::core
        TRU["triton_utils"]:::core
        RUN["utils"]:::core
    end

    %% Kernel Layer
    subgraph "Kernel Layer"
        style KL fill:#C8E6C9
        KA["FlashAttention / Torch fallback"]:::core
    end

    %% Communication Layer
    subgraph "Communication Layer"
        style CL fill:#C8E6C9
        AA["all_to_all"]:::core
        EL["extract_local"]:::core
    end

    %% External Dependencies
    subgraph "External Dependencies"
        style ED fill:#FFE0B2
        PT["PyTorch"]:::ext
        FA["flash-attn library"]:::ext
        TR["Triton"]:::ext
    end

    %% Peripheral Components
    subgraph "Peripheral Components"
        style PC fill:#FFCDD2
        SC["scripts/"]:::optional
        BM1["benchmark_longctx.py"]:::optional
        BM2["benchmark_longctx_qkvpacked.py"]:::optional
        T1["test_hybrid_attn.py"]:::optional
        T2["test_hybrid_qkvpacked_attn.py"]:::optional
        T3["test_ulysses_attn.py"]:::optional
        T4["test_utils.py"]:::optional
    end

    %% Data Flows
    SPG -->|"config PGs"| AA
    SPG -->|"config PGs"| EL
    EX -->|"shard QKV"| EL
    LCA -->|"call attn"| HC
    ACA -->|"call async attn"| HC
    HC -->|"heads→Ulysses"| UA
    HC -->|"heads→Ring"| RPT
    HC --> RFA
    HC --> RFV
    HC --> SFA
    HC --> ZFA
    HC --> ZFV
    HC --> RFI
    UA -->|"all-to-all"| AA
    RPT -->|"ring collect"| EL
    RFA -->|"ring collect"| EL
    RFV -->|"ring collect"| EL
    SFA -->|"ring collect"| EL
    ZFA -->|"ring collect"| EL
    ZFV -->|"ring collect"| EL
    RFI -->|"ring collect"| EL
    UA --> KA
    RPT --> KA
    RFA --> KA
    RFV --> KA
    SFA --> KA
    ZFA --> KA
    ZFV --> KA
    RFI --> KA
    KA -->|"compute attn"| PT
    KA -->|"compute attn"| FA
    KA -->|"kernels"| TR
    AA --> HC
    EL --> HC
    HC --> LCA
    LCA --> SPG

    %% Peripheral connections
    SC --> SPG
    BM1 --> LCA
    BM2 --> LCA
    T1 --> HC
    T2 --> HC
    T3 --> UA
    T4 --> HC

    %% Click Events
    click SPG "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/comm/__init__.py"
    click EX "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/comm/extract_local.py"
    click LCA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/hybrid/attn_layer.py"
    click ACA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/hybrid/async_attn_layer.py"
    click HC "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/hybrid/utils.py"
    click UA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ulysses/attn_layer.py"
    click RPT "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/ring_pytorch_attn.py"
    click RFA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/ring_flash_attn.py"
    click RFV "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/ring_flash_attn_varlen.py"
    click SFA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/stripe_flash_attn.py"
    click ZFA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/zigzag_ring_flash_attn.py"
    click ZFV "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/zigzag_ring_flash_attn_varlen.py"
    click RFI "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/ring_flashinfer_attn.py"
    click TRU "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/triton_utils.py"
    click RUN "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/ring/utils.py"
    click KA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/kernels/attention.py"
    click AA "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/comm/all_to_all.py"
    click EL "https://github.com/feifeibear/long-context-attention/blob/main/yunchang/comm/extract_local.py"
    click PT "https://github.com/feifeibear/long-context-attention/tree/main/PyTorch"
    click FA "https://github.com/feifeibear/long-context-attention/tree/main/flash-attn"
    click TR "https://github.com/feifeibear/long-context-attention/tree/main/Triton"
    click SC "https://github.com/feifeibear/long-context-attention/tree/main/scripts/"
    click BM1 "https://github.com/feifeibear/long-context-attention/blob/main/benchmark/benchmark_longctx.py"
    click BM2 "https://github.com/feifeibear/long-context-attention/blob/main/benchmark/benchmark_longctx_qkvpacked.py"
    click T1 "https://github.com/feifeibear/long-context-attention/blob/main/test/test_hybrid_attn.py"
    click T2 "https://github.com/feifeibear/long-context-attention/blob/main/test/test_hybrid_qkvpacked_attn.py"
    click T3 "https://github.com/feifeibear/long-context-attention/blob/main/test/test_ulysses_attn.py"
    click T4 "https://github.com/feifeibear/long-context-attention/blob/main/test/test_utils.py"

    %% Styles
    classDef api fill:#AEDFF7,stroke:#333,stroke-width:1px
    classDef core fill:#C8E6C9,stroke:#333,stroke-width:1px
    classDef ext fill:#FFE0B2,stroke:#333,stroke-width:1px
    classDef optional fill:#FFCDD2,stroke:#333,stroke-width:1px
```
