# ze33m/flashattention architecture

> Architecture diagram of the GitHub repository ze33m/flashattention, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ze33m/flashattention
- Repository: https://github.com/ze33m/flashattention

## Overview

This repository presents FlashAttention as an algorithm for exact, memory-efficient attention; the tree contains a notebook and a README, but no source excerpts were provided. The documented workflow tiles Q, K, and V, computes each score block, updates online-softmax statistics, and accumulates the output without materializing the full attention matrix. The graph reflects those documented algorithmic stages, not verified notebook internals.

Main entry points: [`README.md`](https://github.com/ze33m/flashattention/blob/main/README.md) (QKV tiles)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_compute["Attention computation"]
  node_qkv["Q, K, V inputs<br/>[README.md]"]
  node_tiles["QKV tiles<br/>[README.md]"]
  node_scores["Block scores<br/>[README.md]"]
  node_softmax["Online softmax<br/>[README.md]"]
  node_output["Output accumulator<br/>[README.md]"]
  node_result["Attention output<br/>[README.md]"]
end

subgraph group_memory["Memory hierarchy"]
  node_hbm[("GPU HBM<br/>[README.md]")]
  node_sram[("GPU SRAM<br/>[README.md]")]
end

node_qkv -->|"tiles into"| node_tiles
node_hbm -->|"loads blocks"| node_tiles
node_tiles -->|"computes scores"| node_scores
node_scores -->|"updates statistics"| node_softmax
node_softmax -->|"normalizes accumulation"| node_output
node_tiles -->|"fits blocks"| node_sram
node_scores -->|"computes in"| node_sram
node_softmax -->|"updates in"| node_sram
node_output -->|"produces"| node_result
node_output -.->|"writes output"| node_hbm

click node_qkv "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_tiles "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_scores "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_softmax "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_output "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_result "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_hbm "https://github.com/ze33m/flashattention/blob/main/README.md"
click node_sram "https://github.com/ze33m/flashattention/blob/main/README.md"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_qkv,node_tiles,node_scores,node_softmax,node_output,node_result toneBlue
class node_hbm,node_sram toneAmber
```

## Components

### Attention computation

- [Q, K, V inputs](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`
- [QKV tiles](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`
- [Block scores](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`
- [Online softmax](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`
- [Output accumulator](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`
- [Attention output](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`

### Memory hierarchy

- [GPU HBM](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`
- [GPU SRAM](https://github.com/ze33m/flashattention/blob/main/README.md): `README.md`

## Connections

- Q, K, V inputs → QKV tiles: tiles into (evidence: `README.md`)
- GPU HBM → QKV tiles: loads blocks (evidence: `README.md`)
- QKV tiles → Block scores: computes scores (evidence: `README.md`)
- Block scores → Online softmax: updates statistics (evidence: `README.md`)
- Online softmax → Output accumulator: normalizes accumulation (evidence: `README.md`)
- QKV tiles → GPU SRAM: fits blocks (evidence: `README.md`)
- Block scores → GPU SRAM: computes in (evidence: `README.md`)
- Online softmax → GPU SRAM: updates in (evidence: `README.md`)
- Output accumulator → Attention output: produces (evidence: `README.md`)
- Output accumulator → GPU HBM: writes output
