keyword

Blockwise RingAttention

Blockwise RingAttention is a distributed computing technique that enables transformer-based artificial intelligence models to process extremely long sequences across multiple hardware accelerators. The method divides an input sequence into smaller blocks and distributes them across a ring network of processing devices, where each device computes attention locally for its assigned query block. While computing attention, devices concurrently pass key and value blocks around the ring in a circular pipeline, fully overlapping communication time with computation. By avoiding the need to load entire sequences or large intermediate attention matrices onto a single device, this approach eliminates memory bottlenecks and allows models to scale context lengths linearly with the number of available devices without relying on approximations.

1 item

World Model on Million-Length Video And Language With Blockwise RingAttention

Hao Liu, Wilson Yan, Matei Zaharia, Pieter Abbeel

OrganizationsUniversity of California Berkeley

Why you should read this

Presents an open-source training method and 7B parameter model family that scales multimodal context to one million tokens using Blockwise RingAttention, enabling the processing of hour-long videos and massive text sequences.

Enabling long-context understanding remains a key challenge in scaling existing sequence models -- a crucial component in developing generally intelligent models that can process and operate over long temporal horizons that potentially consist of millions of tokens. In this paper, we aim to address these challenges by providing a comprehensive exploration of the full development process for producing 1M context language models and video-language models, setting new benchmarks in language retrieval and new capabilities in long video understanding. We detail our long context data curation process, progressive context extension from 4K to 1M tokens, and present an efficient open-source implementation for scalable training on long sequences. Additionally, we open-source a family of 7B parameter models capable of processing long text documents and videos exceeding 1M tokens.

Source

https://arxiv.org/pdf/2402.08268.pdf

Added

2026-09-26