FlashAttention: Revolutionizing Transformers by Overcoming Hardware Performance Bottlenecks

In recent years, attention models have become the go-to architecture for various applications, including natural language processing and image classification. However, as these models have grown larger and more complex, they have encountered significant challenges in terms of runtime and memory consumption, especially when dealing with long sequences. The main culprit behind these bottlenecks is the self-attention mechanism, which exhibits quadratic time and memory complexity with respect to sequence length. To address this issue, researchers have explored various approximate attention methods. However, these methods often fail to achieve substantial speedup in real-world scenarios, primarily due to the neglect of an essential factor: input/output (IO) operations. This article introduces FlashAttention, a novel attention algorithm that not only significantly accelerates attention computations but also minimizes memory overheads by incorporating IO-awareness.

LEFT: FlashAttention use tilling to prevent materialisation of the large N x N attention matrix (dotted box) on (relatively) slow GPU HBM. In the outer loop (red arrows), FlashAttention loops through blocks of the K and V matrices and loads them to fast on-chip SRAM. In each block, FlashAttention loops over blocks of Q matrix (blue arrows), loading them to SRAM, and writing the output of the attention computation back to HBM.
RIGHT: Speedup over the PyTorch implementation of attention on GPT-2. FlashAttention does not read and write the large N X N attention matrix to HBM, resulting in a 7.6x speedup on the attention computation.

The Problem with Standard Attention Models

The standard attention model suffers from two major challenges when dealing with long sequences: computational complexity and memory requirements. The quadratic time and memory complexity of self-attention make it slow and memory-hungry for large inputs. While approximate attention methods have attempted to mitigate these challenges, they often prioritize reducing floating-point operations (FLOPs) without considering the IO overheads. As a result, they fail to achieve significant wall-clock speedup.

Furthermore, the computation speed of GPU processors has improved more rapidly relative to memory speed, operations are increasingly bottlenecked by high bandwidth memory (HBM) accesses.

Introducing FlashAttention: Tiling, Recomputation, and Kernel Fusion

FlashAttention addresses the limitations of standard attention by introducing IO-awareness and incorporating three key techniques: tiling, recomputation, and kernel fusion, to exploit fast SRAM.

Tiling: FlashAttention employs tiling, which involves splitting the input into smaller blocks and performing attention computations incrementally. By doing so, FlashAttention reduces the need to materialize the large attention matrix on relatively slow GPU high bandwidth memory (HBM). Tiling also allows FlashAttention to exploit the fast GPU on-chip SRAM for storing and processing data, resulting in fewer memory reads and writes between different levels of memory.

Recomputation: To further reduce memory access overheads, FlashAttention leverages recomputation. Instead of storing the intermediate attention matrix in memory, FlashAttention recalculates attention values on-the-fly during the backward pass. This approach eliminates the need to read the attention matrix from HBM, leading to faster computations and reduced memory usage.

Kernel Fusion: FlashAttention combines all attention operations into a single GPU kernel, leveraging fine-grained control over memory access. By fusing these operations, FlashAttention minimizes the overhead associated with launching multiple kernels, resulting in improved efficiency.

Benefits and Impact of FlashAttention

FlashAttention brings several significant benefits and advancements to attention models:

Faster Model Training: FlashAttention enables faster training of Transformer models. Experimental results demonstrate a 15% end-to-end wall-clock speedup on BERT-large (sequence length 512) compared to the MLPerf 1.1 training speed record. It also achieves impressive speedups of 3x on GPT-2 (sequence length 1K) and 2.4x on long-range arena (sequence length 1K-4K) compared to existing baselines.

Higher Quality Models: By enabling longer context in Transformers, FlashAttention leads to higher quality models. It achieves a 0.7 improvement in perplexity on GPT-2 and 6.4 points of lift on long-document classification. Notably, FlashAttention allows Transformers to achieve better-than-chance performance on challenging tasks such as the Path-X challenge (sequence length 16K, 61.4% accuracy) and Path-256 (sequence length 64K, 63.1% accuracy).

Benchmarking Attention: FlashAttention outperforms standard attention implementations, with speedups of up to 3x across common sequence lengths ranging from 128 to 2K. It scales up to a sequence length of 64K, demonstrating its superior performance and efficiency.


FlashAttention represents a significant breakthrough in addressing the hardware performance bottlenecks faced by attention models. By incorporating IO-awareness and leveraging techniques such as tiling, recomputation, and kernel fusion, FlashAttention achieves remarkable improvements in both computation speed and memory efficiency. The experimental results validate its effectiveness in training high-quality Transformer models and overcoming the limitations of long sequences. As attention models continue to play a crucial role in various domains, FlashAttention paves the way for more efficient and scalable training of transformer, opening up new possibilities for research and application development in the field of deep learning.

This article is drafted with the assistance of A.I. and referencing from the sources below:

The work described in this article was supported by InnoHK initiative, The Government of the HKSAR, and Laboratory for AI-Powered Financial Technologies.
(AIFT strives but cannot guarantee the accuracy and reliability of the content, and will not be responsible for any loss or damage caused by any inaccuracy or omission.)

Share this content


Units 1101-1102 & 1121-1123,
Building 19W Science Park West Avenue,
Hong Kong Science Park,
Shatin, Hong Kong

Products & Solutions


About Us


Copyright © 2024 Laboratory for AI-Powered Financial Technologies Ltd. All Rights Reserved.