How feifeibear/long-context-attention works
Project Type and Purpose
• This is a Python library/tool (not a web app) for distributed attention in long-context LLM training and inference.
• It implements a hybrid (“Unified Sequence Parallelism”) approach that combines DeepSpeed-Ulysses and ring-based attention to maximize scalability and performance.
Key Components from File Structure
• Top-level:
• pyproject.toml, README.md, LICENSE, CI workflows, scripts/, docs/, benchmark/, test/
• Core Package (yunchang/):
• yunchang/comm/ – process-group setup & tensor-extraction helpers
• yunchang/hybrid/ – high-level attention layers (AsyncLongContextAttention, LongContextAttention)
• yunchang/kernels/ – attention kernel implementations (FlashAttention wrappers, Torch fallback)
• yunchang/ring/ – multiple ring-attention variants (basic, stripe, zigzag, varlen, FlashInfer)
• yunchang/ulysses/ – Ulysses-style all-to-all attention layer
• yunchang/globals.py – global config, enums (AttnType)
• Scripts & Benchmarks:
• scripts/*.sh – quick start and benchmark runners
• benchmark/ – performance comparison scripts
• Tests:
• test/*.py – unit/integration tests exercising each implementation
Insights from README
• Supports FlashAttention v2/v3 or pure-Torch fallback
• Offers API set_seq_parallel_pg + EXTRACT_FUNC_DICT for sharding QKV
• Uses strategy pattern: choose ring_impl_type and attn_type at runtime
• Integrates upstream (Megatron-LM, NVIDIA TransformerEngine) via patch
Main Components
a. API Layer (User-facing)
• set_seq_parallel_pg() – configures process groups for Ulysses & Ring parallelism
• Extraction functions (EXTRACT_FUNC_DICT) – permute & shard QKV
• LongContextAttention / AsyncLongContextAttention – entry point for attention call
b. Hybrid Controller
• Decision logic: split heads → some to Ulysses module, some to Ring module
c. Ulysses Module
• All-to-all communication for head-parallel attention
• Built with yunchang/ulysses/attn_layer.py
d. Ring Module
• Multiple implementations (basic, stripe, zigzag, varlen, FlashInfer)
• yunchang/ring/* uses yunchang/ring/triton_utils.py + flash-attn or torch
e. Kernel Layer
• FlashAttention wrappers (FA v2, FA v3)
• Torch fallback
f. Communication Layer
• yunchang/comm/all_to_all.py – collective ops for Ulysses
• yunchang/comm/extract_local.py – sharding logic for ring
g. External Dependencies
• PyTorch
• flash-attn (optional)
• SpargeAttn, SageAttention, FlashInfer (optional integrations)
Component Relationships and Data Flow
• User code calls set_seq_parallel_pg → PGs created
• Q, K, V tensors → EXTRACT_FUNC_DICT → local shards
• LongContextAttention receives local shards, routes to Ulysses and/or Ring submodules
• Each submodule calls into kernels (FlashAttention or Torch) and comm primitives
• Results from submodules are gathered (all-to-all or ring-collect) → merged and returned
Architectural Patterns & Principles
• Strategy/Factory Pattern: pluggable attn_type and ring_impl_type
• Modular layers: clear separation between comm, kernel, control logic
• Extensibility: new ring variants, new kernel backends can be added under well-defined interfaces
Essential Elements
• clear component names (LongContextAttention, RingAttention variants, UlyssesAttention)
• process group configuration (Ulysses PG, Ring PG)
• external libraries (torch, flash-attn, Triton)