Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

linear shift

A linear shift refers to a systematic geometric translation or additive vector offset within the internal representation space of a neural network, such as the residual stream of a transformer architecture. In mechanistic interpretability and model alignment, a linear shift modifies intermediate representations by displacing activation vectors along consistent directional paths rather than removing or re-architecting the underlying pre-trained feature representations. This translation effectively steers hidden states away from specific activation regions, bypassing the elicitation of particular behaviors or undesirable outputs during inference while leaving the model's core latent capabilities intact. Because the underlying mechanisms are avoided rather than eliminated, these offsets can potentially be counteracted or navigated back toward the original behavior through targeted representation steering or adversarial prompts.

1 item

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, Rada Mihalcea

OrganizationsHarvard UniversityUniversity of MichiganUniversity of Sydney

Why you should read this

Reveals that Direct Preference Optimization merely bypasses rather than eliminates toxic capabilities in language models, providing a mechanistic explanation for safety jailbreaks and enabling a simple method to reverse alignment.

While alignment algorithms are commonly used to tune pre-trained language models towards user preferences, we lack explanations for the underlying mechanisms in which models become “aligned”, thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in pre-trained language models (GPT2-medium, Llama2-7b). We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting models avert toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the models, reverting them back to their toxic behavior.

Added

2026-09-30