Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Blockwise Transformers

Blockwise Transformers are deep learning architectures that process long sequences of data by partitioning inputs into smaller, discrete blocks to perform attention and feedforward operations incrementally. Rather than materializing and storing the entire attention matrix or all intermediate activations simultaneously, this approach computes self-attention and subsequent layer activations block by block. By organizing computations into these modular segments, Blockwise Transformers dramatically reduce memory overhead and bypass standard sequence length limitations while preserving exact transformer mathematics. This chunked structure also enables efficient sequence-level distributed computing, allowing the communication of key and value states between hardware accelerators to overlap with local computation and scale context capacities to millions of tokens.

1 item

Ring Attention with Blockwise Transformers for Near-Infinite Context

Ring Attention with Blockwise Transformers for Near-Infinite Context

Hao Liu, Matei Zaharia, Pieter Abbeel

OrganizationsUniversity of California Berkeley

Why you should read this

Overlaps the communication of key-value blocks with local computation over a mesh ring, theoretically allowing context lengths bounded only by the total cluster memory.

Part of:Prep reading for the model training infrastructure interview

Transformers have emerged as the architecture of choice for many state-of-the-art AI models, showcasing exceptional performance across a wide range of AI applications. However, the memory demands imposed by Transformers limit their ability to handle long sequences, thereby posing challenges in utilizing videos, actions, and other long-form sequences and modalities in complex environments. We present a novel approach, Ring Attention with Blockwise Transformers (Ring Attention), which leverages blockwise computation of self-attention and feedforward to distribute long sequences across multiple devices while fully overlapping the communication of key-value blocks with the computation of blockwise attention. Our approach enables training and inference of sequences that are up to device count times longer than those achievable by prior memory-efficient Transformers, without resorting to approximations or incurring additional communication and computation overheads. Extensive experiments on language modeling and reinforcement learning tasks demonstrate the effectiveness of our approach in allowing millions of tokens context size and improving performance.

Added

2026-03-13