keyword
High-Bandwidth Memory
High-Bandwidth Memory is a high-speed computer memory architecture that vertically stacks multiple dynamic random-access memory dies to provide substantially faster data transfer rates and higher energy efficiency than traditional memory. By connecting these vertically integrated layers using through-silicon vias and mounting them on a wide interface close to processing units via a silicon interposer, it significantly reduces physical distance and communication latency while maximizing throughput. This architecture is widely utilized in data-intensive hardware environments, including graphics processing units, artificial intelligence accelerators, and high-performance computing platforms, where standard memory interfaces create data bottlenecks.
2 items
FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning
Tri Dao
Why you should read this
Presents FlashAttention-2, an exact attention algorithm that doubles the execution speed of FlashAttention and reaches up to 73% of theoretical peak GPU utilization through improved work partitioning and thread-block parallelism.
Scaling Transformers to longer sequence lengths has been a major problem in the last several years, promising to improve performance in language modeling and high-resolution image understanding, as well as to unlock new applications in code, audio, and video generation. The attention layer is the main bottleneck in scaling to longer sequences, as its runtime and memory increase quadratically in the sequence length. FlashAttention exploits the asymmetric GPU memory hierarchy to bring significant memory saving (linear instead of quadratic) and runtime speedup (2-4 compared to optimized baselines), with no approximation. However, FlashAttention is still not nearly as fast as optimized matrix-multiply (GEMM) operations, reaching only 25-40\% of the theoretical maximum FLOPs/s. We observe that the inefficiency is due to suboptimal work partitioning between different thread blocks and warps on the GPU, causing either low-occupancy or unnecessary shared memory reads/writes. We propose FlashAttention-2, with better work partitioning to address these issues. In particular, we (1) tweak the algorithm to reduce the number of non-matmul FLOPs (2) parallelize the attention computation, even for a single head, across different thread blocks to increase occupancy, and (3) within each thread block, distribute the work between warps to reduce communication through shared memory. These yield around 2 speedup compared to FlashAttention, reaching 50-73\% of the theoretical maximum FLOPs/s on A100 and getting close to the efficiency of GEMM operations. We empirically validate that when used end-to-end to train GPT-style models, FlashAttention-2 reaches training speed of up to 225 TFLOPs/s per A100 GPU (72\% model FLOPs utilization).
Added
2026-09-12
Ring Attention with Blockwise Transformers for Near-Infinite Context
Hao Liu, Matei Zaharia, Pieter Abbeel
Why you should read this
Overlaps the communication of key-value blocks with local computation over a mesh ring, theoretically allowing context lengths bounded only by the total cluster memory.
Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities in complex environments. We present a novel approach, Ring Attention with Blockwise Transformers (Ring Attention), which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention. Our approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of our approach in allowing millions of tokens context size and improving performance.
Added
2026-03-13