Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

memory coefficients

Memory coefficients are numerical weighting values in neural networks that indicate how strongly an input representation matches specific stored memory patterns or keys. In the context of transformer feed-forward layers interpreted as associative key-value memory networks, memory coefficients are computed by taking the product of an incoming representation with learned key vectors and passing the results through a non-linear activation function. Each coefficient quantifies the activation level of an individual memory slot, serving as a scalar weight that dictates how much its corresponding value vector contributes to the final layer output. Larger memory coefficients reflect stronger pattern matches in the input, allowing the network to dynamically retrieve and compose relevant stored knowledge or vocabulary distributions for downstream predictions.

1 item

Transformer Feed-Forward Layers Are Key-Value Memories

Transformer Feed-Forward Layers Are Key-Value Memories

Mor Geva, Roei Schuster, Jonathan Berant, Omer Levy

OrganizationsAllen Institute for AICornell UniversityTel Aviv University

Why you should read this

Shows that transformer feed-forward layers function as key-value memories, revealing how the majority of network parameters store human-interpretable textual patterns and directly predict next-token distributions.

Feed-forward layers constitute two-thirds of a transformer model's parameters, yet their role in the network remains under-explored. We show that feed-forward layers in transformer-based language models operate as key-value memories, where each key correlates with textual patterns in the training examples, and each value induces a distribution over the output vocabulary. Our experiments show that the learned patterns are human-interpretable, and that lower layers tend to capture shallow patterns, while upper layers learn more semantic ones. The values complement the keys' input patterns by inducing output distributions that concentrate probability mass on tokens likely to appear immediately after each pattern, particularly in the upper layers. Finally, we demonstrate that the output of a feed-forward layer is a composition of its memories, which is subsequently refined throughout the model's layers via residual connections to produce the final output distribution.

Added

2026-09-24