# poptropicasahil/flashattention-from-scratch architecture

> Architecture diagram of the GitHub repository poptropicasahil/flashattention-from-scratch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/poptropicasahil/flashattention-from-scratch
- Repository: https://github.com/poptropicasahil/flashattention-from-scratch

## Overview

##### Your Main Components
- High‑level Python interface
– The “flash-attn” directory (program.py, triton_attn.py) provides the core attention computation logic.
– Integration with PyTorch via a custom autograd Function (visible in the explanation of forward and backward passes).
- GPU Kernel Implementations:
– The “cuda” directory includes raw CUDA files for vector and matrix addition. These are likely used as building blocks or proofs of concept for low‑level GPU operations.
– The “triton” directory (with the vector_add.ipynb notebook) shows usage of the Triton language for writing GPU kernels at a higher abstraction level.
- Core Flash Attention Algorithm:
– The detailed README explains the online softmax trick, block matrix multiplication, CUDA memory hierarchy (HBM, shared memory), pipelining, and more.
- Autograd and Gradient Computation:
– The explanation of forward and backward passes (including derivations for gradients, softmax Jacobians, chain rule) shows that the system has a custom backward pass that manually computes gradients for Q, K, V.
- Supporting Materials:

##### Their Interactions and Flow
- Input data (Q, K, V tensors) is fed by a higher‑level network (or stand‑alone demo program).
- The forward pass is segmented into “blocks” – each block of Queries and Keys is handled by kernels running on the GPU.
- The kernels perform:
– Block matrix multiplication (computing QKᵀ).
– Safe softmax (using online or correction‑factor approaches) on a per‑row or per‑block basis.
– A computed multiplication of the softmax output with V.
- The system is designed to work in parallel (batch, head, query‑block levels) and uses GPU memory optimizations (such as moving data from global memory to shared memory, reusing data).
- The custom autograd function’s backward pass leverages saved intermediates (or possibly recomputed ones) to derive gradients flow via chain rule and efficient sparse Jacobian operations.
- The Triton kernels (as evidenced in the notebook and in the flash-attn/triton_attn.py file) provide another method to define the GPU work at block level without manually managing thread ids.

##### Key Architectural Patterns and Design Principles
- Separation of concerns: Low‑level GPU kernels (CUDA/Triton) are isolated from the high-level Python interface.
- Parallelism: Architectural emphasis on fully exploiting parallelism at several levels (batch, head, query block, and low‑level kernel threading).
- Memory optimization: Emphasis on using shared memory and asynchronous transfers, plus software pipelining.
- Numerical robustness: Special care in computing softmax in a numerically stable way using the log‑sum‑exp trick and online corrections.
- Autograd integration: The system wraps GPU operations in custom autograd functions allowing correct backward propagation while fusing multiple operations.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Input Tensors (Q, K, V)"]:::input

    subgraph "High-Level API / PyTorch Integration"
        HL1["flash-attn/program.py"]:::highLevel
        HL2["flash-attn/triton_attn.py"]:::highLevel
    end
    A -->|"feeds"| HL1
    A --> HL2

    subgraph "Flash Attention Core Engine - Forward Pass"
        F1["QKᵀ Computation"]:::proc
        F2["Safe Softmax Correction"]:::proc
        F3["Block Matrix Multiplication with V"]:::proc
    end
    HL1 -->|"triggers"| F1
    HL2 --> F1
    F1 --> F2
    F2 --> F3

    subgraph "GPU Kernel Layer"
        subgraph "CUDA Kernels"
            C1["CUDA: vector_add"]:::kernel
            C2["CUDA: vector_add_simple"]:::kernel
            C3["CUDA: matrix_add"]:::kernel
        end
        subgraph "Triton Kernels"
            T1["Triton: vector_add.ipynb"]:::kernel
        end
    end
    F3 -->|"kernel ops"| C1
    F3 --> C2
    F3 --> C3
    F3 --> T1

    subgraph "Flash Attention Core Engine - Backward Pass"
        B1["Gradient (∂loss/∂O)"]:::proc
        B2["Autograd & Jacobian Computation"]:::proc
    end
    T1 --> B1
    C1 --> B1
    C2 --> B1
    C3 --> B1
    B1 --> B2
    HL1 --> B2
    HL2 --> B2

    subgraph "Data & Memory Management"
        MM1["Global Memory (HBM)"]:::memory
        MM2["Shared Memory"]:::memory
        MM3["Software Pipelining"]:::memory
    end
    C1 --- MM1
    C2 --- MM1
    C3 --- MM1
    T1 --- MM2
    MM3 --- MM1

    subgraph "Mathematical / Algorithmic Details"
        MA1["Online Softmax (log-sum-exp trick)"]:::algo
        MA2["Block Accumulation & Differentiation"]:::algo
    end
    F2 --- MA1
    B2 --- MA2

    S["Supporting Materials (Diagrams & Explanations)"]:::support
    B2 --- S

    D["Dependency & Configuration (requirements.txt)"]:::dep
    HL1 --- D
    HL2 --- D

    %% Click Events
    click HL1 "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/flash-attn/program.py"
    click HL2 "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/flash-attn/triton_attn.py"
    click C1 "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/cuda/vector_add.cu"
    click C2 "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/cuda/vector_add_simple.cu"
    click C3 "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/cuda/matrix_add.cu"
    click T1 "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/triton/vector_add.ipynb"
    click S "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/readme-images/autograd.drawio"
    click S "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/readme-images/grad_softmax.drawio"
    click D "https://github.com/poptropicasahil/flashattention-from-scratch/blob/main/requirements.txt"

    %% Styles
    classDef input fill:#d1e8ff,stroke:#000,stroke-width:1px
    classDef highLevel fill:#fce5cd,stroke:#000,stroke-width:1px
    classDef kernel fill:#e2efda,stroke:#000,stroke-width:1px
    classDef proc fill:#ead1dc,stroke:#000,stroke-width:1px
    classDef memory fill:#fff2cc,stroke:#000,stroke-width:1px
    classDef algo fill:#d9ead3,stroke:#000,stroke-width:1px
    classDef support fill:#cfe2f3,stroke:#000,stroke-width:1px
    classDef dep fill:#f4cccc,stroke:#000,stroke-width:1px
```
