Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

expressive hidden states

Expressive hidden states refer to internal memory representations in sequence modeling architectures that possess higher representational capacity than traditional fixed-dimensional vectors, enabling them to compress and retain complex contextual information across long sequences. In recurrent sequence models, an expressive hidden state can take the form of an embedded machine learning model, such as a linear layer or a multilayer neural network, whose internal parameters function as the memory state. Instead of relying on static vector activations updated by fixed transition rules, models with expressive hidden states dynamically update these state parameters during sequence processing through optimization steps, such as self-supervised learning updates. This design allows architectures to maintain linear computational complexity with respect to sequence length while significantly reducing the information bottleneck and memory degradation common in conventional recurrent networks.

1 item

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

Learning to (Learn at Test Time): RNNs with Expressive Hidden States

Yu Sun, Xinhao Li, Karan Dalal, Jiarui Xu, Arjun Vikram, Genghan Zhang, Yann Dubois, Xinlei Chen, Xiaolong Wang, Sanmi Koyejo, Tatsunori Hashimoto, Carlos Guestrin

OrganizationsMetaStanford UniversityUniversity of California BerkeleyUniversity of California, San Diego

Why you should read this

Introduces Test-Time Training layers that treat RNN hidden states as internal machine learning models updated via self-supervised gradient steps, achieving linear-time sequence modeling that continues to improve across long contexts where existing architectures plateau.

Self-attention performs well in long context but has quadratic complexity. Existing RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden states. We present a practical framework for instantiating sequence modeling layers with linear complexity and expressive hidden states. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training even on test sequences, our layers are called Test-Time Training (TTT) layers. We consider two instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a strong Transformer and Mamba, a modern RNN. Similar to Transformer, TTT-Linear and TTT-MLP can keep reducing perplexity by conditioning on more tokens, while Mamba cannot after 16k context. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research.

Added

2026-09-26