# nvidia/star-attention architecture

> Architecture diagram of the GitHub repository nvidia/star-attention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nvidia/star-attention
- Repository: https://github.com/nvidia/star-attention

## Overview

##### And partition the major system components
- Inference Pipelines:
- RULER: Found under the “ruler” directory. Contains data preparation (synthetic data in ruler/data and subdirectories), evaluation scripts (ruler/eval), result gathering (gather_results_ruler.py), and configuration files (YAML configs).
- BABILong: Found under the “babilong” directory. It has its own scripts (e.g., gather_results_babilong.py, prepare_data.py, prompts.py) for processing and evaluation.
- Core Star Attention Module:
- Star Attention implementations are defined in the “star_attention” directory. There are separate implementations for “ring_flash_attn” and “star_flash_attn” mechanisms, indicating support for multiple attention strategies.
- Inference Launch and Utility Scripts:
- run_ruler.py, run_babilong.py, and run_star_attn_inference.py orchestrate the end-to-end inference flow.
- Scripts in the “scripts” directory (e.g., download_hf_model.py and time_inference.py) are used for model downloading and additional utility purposes.
- Data and Configuration:
- There are supporting files like requirements.txt, launch.sh, and various downloadable data scripts under “ruler/download_data.sh”.
- README.md files in different directories provide guidelines and documentation.

##### Relationships and interactions
- The inference pipelines (RULER and BABILong) feed inputs into the core Star Attention engine. They use separate Python scripts to initiate inference and process results.
- The core attention module is integrated into the model via the PyTorch/HuggingFace Transformers framework. It manipulates the KV cache and manages attention computation in two distinct phases.
- For distributed inference, the system splits the input context across available hosts/GPUs and uses inter-host communication to merge temperature-scaled outputs (local softmax values and attention outputs).
- The project also includes various scripts for data preparation, external model downloads, and evaluation, which interact with the main inference pipelines.

##### Important architectural patterns/design principles
- Modular design: Each major component (inference pipeline for RULER, BABILong, attention implementations) is separated into its own directory and set of scripts.
- Two-phase computation workflow: Clearly distinguish Phase 1 (Context Encoding) and Phase 2 (Query Processing and Token Generation) as depicted in the README. Represent each phase with its subcomponents (e.g., block segmentation, KV cache management, local-to-global aggregation).
- Extensibility: The presence of multiple attention modules (star and ring variants) and separate configuration files indicates that the architecture is designed to plug-and-play with different attention mechanisms and datasets.

- Inside “Star Attention Core Module”, further subdivide into two phases:
- Phase 1: Context Encoding – show block segmentation, anchor block addition, distribution across H hosts, and local KV cache storage.
- Phase 2: Query Processing – show the query host, local attention outputs, softmax denominator (exponent summation), inter-host data gathering, and final global attention output.
- Distinguish between different types of components using:
- Annotations or notes that describe:
- The use of HuggingFace Transformers and PyTorch as foundational libraries.
- The significance of scripts like “launch.sh” for orchestrating distributed execution.
- Points where configuration files (e.g., synthetic_inference_config.yaml) adjust system behavior.

##### Final checklist
- Every major component is represented and labeled.
- Provide annotations regarding key configuration variables (e.g., “-np”, “-nn”, block size, etc.) to indicate where engineers may need to adjust system parameters.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% General Configuration
    A["General Config & Dependencies"]:::config
    K["requirements.txt & Others"]:::config
    A --> K

    %% Utility & Data Preparation Tools
    B["Utility & Data Preparation Tools"]:::utility
    I["Utility Scripts (scripts/)"]:::utility
    J["launch.sh"]:::launch
    A --> B
    B --> I
    B --> J

    %% Inference Launch Scripts
    subgraph "Inference Launch Scripts"
        C1["run_ruler.py"]:::launch
        C2["run_babilong.py"]:::launch
        C3["run_star_attn_inference.py"]:::launch
    end
    B --> C1
    B --> C2
    B --> C3

    %% Inference Pipelines
    subgraph "Inference Pipelines"
        D["RULER Inference Pipeline"]:::pipeline
        E["BABILong Inference Pipeline"]:::pipeline
    end
    C1 --> D
    C2 --> E

    %% Core Star Attention Module
    F["Core Star Attention Module"]:::star
    D --> F
    E --> F

    %% Phase 1: Context Encoding
    subgraph "Phase 1: Context Encoding"
        F1a["Block Segmentation"]:::phase
        F1b["Anchor Block Addition"]:::phase
        F1c["Distributed Data (across GPU Hosts)"]:::phase
        F1d["Local KV Cache Storage"]:::phase
    end

    %% Phase 2: Query Processing
    subgraph "Phase 2: Query Processing"
        F2a["Query Host Processing"]:::phase
        F2b["Local Attention Outputs"]:::phase
        F2c["SoftmaxDenom_Calc"]:::phase
        F2d["Inter-host Data Gathering"]:::phase
        F2e["Final Global Attention Output"]:::phase
    end

    %% Connect phases within Core Star Attention Module
    F --> F1a
    F1a --> F1b
    F1b --> F1c
    F1c --> F1d
    F1d --> F2a
    F2a --> F2b
    F2b --> F2c
    F2c --> F2d
    F2d --> F2e

    %% Attention Variants (alternative implementations)
    subgraph "Attention Variants"
        H1["Ring Variant"]:::variant
        H2["Star Variant"]:::variant
    end
    %% Dotted connection indicating that variant implementations can plug into the module
    F1d -.-> H1
    F1d -.-> H2
    H1 -.-> F2a
    H2 -.-> F2a

    %% Model & HuggingFace Integration
    G["Model & HuggingFace Integration"]:::model
    F2e --> G

    %% Click Events
    click K "https://github.com/nvidia/star-attention/blob/main/requirements.txt"
    click I "https://github.com/nvidia/star-attention/tree/main/scripts/"
    click J "https://github.com/nvidia/star-attention/blob/main/launch.sh"
    click C1 "https://github.com/nvidia/star-attention/blob/main/run_ruler.py"
    click C2 "https://github.com/nvidia/star-attention/blob/main/run_babilong.py"
    click C3 "https://github.com/nvidia/star-attention/blob/main/run_star_attn_inference.py"
    click D "https://github.com/nvidia/star-attention/tree/main/ruler/"
    click E "https://github.com/nvidia/star-attention/tree/main/babilong/"
    click H1 "https://github.com/nvidia/star-attention/tree/main/star_attention/ring_flash_attn/"
    click H2 "https://github.com/nvidia/star-attention/tree/main/star_attention/star_flash_attn/"
    click G "https://github.com/nvidia/star-attention/blob/main/model.py"

    %% Styles
    classDef config fill:#f9e79f,stroke:#000,stroke-width:2px;
    classDef utility fill:#aed6f1,stroke:#000,stroke-width:2px;
    classDef launch fill:#fad7a0,stroke:#000,stroke-width:2px;
    classDef pipeline fill:#d7bde2,stroke:#000,stroke-width:2px;
    classDef star fill:#a3e4d7,stroke:#000,stroke-width:2px;
    classDef phase fill:#d5f5e3,stroke:#000,stroke-width:2px;
    classDef model fill:#f5b7b1,stroke:#000,stroke-width:2px;
    classDef variant fill:#fcf3cf,stroke:#000,stroke-width:2px,stroke-dasharray: 5, 5
```
