How ze33m/flashattention works
This repository presents FlashAttention as an algorithm for exact, memory-efficient attention; the tree contains a notebook and a README, but no source excerpts were provided. The documented workflow tiles Q, K, and V, computes each score block, updates online-softmax statistics, and accumulates the output without materializing the full attention matrix. The graph reflects those documented algorithmic stages, not verified notebook internals.
Main entry points: README.md QKV tiles
Components 8
Attention computation
- Q, K, V inputs
README.md - QKV tiles
README.md - Block scores
README.md - Online softmax
README.md - Output accumulator
README.md - Attention output
README.md
Connections 10
- Q, K, V inputs → QKV tiles: tiles into
README.md - GPU HBM → QKV tiles: loads blocks
README.md - QKV tiles → Block scores: computes scores
README.md - Block scores → Online softmax: updates statistics
README.md - Online softmax → Output accumulator: normalizes accumulation
README.md - QKV tiles → GPU SRAM: fits blocks
README.md - Block scores → GPU SRAM: computes in
README.md - Online softmax → GPU SRAM: updates in
README.md - Output accumulator → Attention output: produces
README.md - Output accumulator → GPU HBM: writes output