Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

MLP value vectors

MLP value vectors are parameter vectors comprising the output projection matrix of feedforward or multilayer perceptron layers within transformer neural networks. In mechanistic interpretability and key-value memory formulations of transformers, feedforward layers are conceptualized as associative memories where the initial projection acts as keys that detect specific features or context patterns, and the second down-projection consists of value vectors. When an input pattern activates a given key, the corresponding activation coefficient scales its associated value vector, which is then added directly into the model residual stream. These vectors typically encode and promote specific semantic concepts, factual information, or shifts in vocabulary probability distributions, enabling the network to store, retrieve, and inject learned representations during inference.

1 item

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, Rada Mihalcea

OrganizationsHarvard UniversityUniversity of MichiganUniversity of Sydney

Why you should read this

Reveals that Direct Preference Optimization merely bypasses rather than eliminates toxic capabilities in language models, providing a mechanistic explanation for safety jailbreaks and enabling a simple method to reverse alignment.

While alignment algorithms are commonly used to tune pre-trained language models towards user preferences, we lack explanations for the underlying mechanisms in which models become “aligned”, thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in pre-trained language models (GPT2-medium, Llama2-7b). We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting models avert toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the models, reverting them back to their toxic behavior.

Added

2026-09-30