Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

toxic vectors

Toxic vectors refer to directional representations within the latent activation or parameter spaces of neural language models that correspond to and promote the generation of harmful, abusive, or offensive content. In machine learning and mechanistic interpretability, these vectors capture how undesirable concepts are structurally encoded across network layers, feed-forward submodules, or individual neurons. By locating and isolating these mathematical directions, researchers can analyze the internal mechanisms of safety alignment algorithms, determine whether safety training permanently eliminates or merely bypasses hazardous behaviors, and perform targeted interventions such as activation steering or pruning to suppress toxic outputs.

1 item

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicity

Andrew Lee, Xiaoyan Bai, Itamar Pres, Martin Wattenberg, Jonathan K. Kummerfeld, Rada Mihalcea

OrganizationsHarvard UniversityUniversity of MichiganUniversity of Sydney

Why you should read this

Reveals that Direct Preference Optimization merely bypasses rather than eliminates toxic capabilities in language models, providing a mechanistic explanation for safety jailbreaks and enabling a simple method to reverse alignment.

While alignment algorithms are commonly used to tune pre-trained language models towards user preferences, we lack explanations for the underlying mechanisms in which models become “aligned”, thus making it difficult to explain phenomena like jailbreaks. In this work we study a popular algorithm, direct preference optimization (DPO), and the mechanisms by which it reduces toxicity. Namely, we first study how toxicity is represented and elicited in pre-trained language models (GPT2-medium, Llama2-7b). We then apply DPO with a carefully crafted pairwise dataset to reduce toxicity. We examine how the resulting models avert toxic outputs, and find that capabilities learned from pre-training are not removed, but rather bypassed. We use this insight to demonstrate a simple method to un-align the models, reverting them back to their toxic behavior.

Added

2026-09-30