Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

Semiseparable Matrix Transformations

Semiseparable matrix transformations are linear mappings governed by structured matrices whose off-diagonal submatrices satisfy low-rank constraints. In applied linear algebra and sequence modeling, these transformations act on input vectors or sequential data by exploiting the fact that the triangular parts of the transformation matrix can be factored into lower-dimensional components. This structural property allows matrix-vector multiplication and related operations to be evaluated in linear time and memory relative to sequence length, bypassing the quadratic complexity typical of dense matrix operations. Consequently, semiseparable matrix transformations can be computed through multiple equivalent mathematical algorithms, including recursive state-space evaluations, parallel scans, and structured attention mechanisms, serving as an efficient framework for signal processing, numerical computation, and deep learning.

1 item

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

Tri Dao, Albert Gu

OrganizationsCarnegie Mellon UniversityPrinceton University

Why you should read this

Establishes a theoretical connection between Transformers and state-space models via structured semiseparable matrices, introducing the Mamba-2 architecture that processes sequences two to eight times faster than Mamba while matching Transformer language modeling performance.

While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

Added

2026-09-18