Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

behavior expectation bounds

Behavior expectation bounds is a theoretical framework in artificial intelligence safety used to mathematically analyze the characteristics and fundamental limits of aligning large language models with intended behavioral standards. The framework formalizes alignment by evaluating the expected score of a model's responses across different output distributions and prompts. Within this structure, it proves that if a model retains any non-zero probability of exhibiting an undesirable behavior, there exist adversarial prompts whose length enables the elicitation of that behavior with high probability. As a result, the framework demonstrates that alignment methods that merely suppress or lower the likelihood of unsafe outputs, rather than completely removing the underlying behavioral modes, remain inherently vulnerable to adversarial jailbreaks and targeted prompting attacks.

1 item

Fundamental Limitations of Alignment in Large Language Models

Fundamental Limitations of Alignment in Large Language Models

Yotam Wolf, Noam Wies, Oshri Avnery, Yoav Levine, Amnon Shashua

OrganizationsAI21 LabsThe Hebrew University of Jerusalem

Why you should read this

Establishes a theoretical framework called Behavior Expectation Bounds to mathematically prove that standard alignment methods like RLHF cannot prevent adversarial jailbreaks as long as harmful behaviors retain a non-zero probability of being generated.

An important aspect in developing language models that interact with humans is aligning their behavior to be useful and unharmful for their human users. This is usually achieved by tuning the model in a way that enhances desired behaviors and inhibits undesired ones, a process referred to as alignment. In this paper, we propose a theoretical approach called Behavior Expectation Bounds (BEB) which allows us to formally investigate several inherent characteristics and limitations of alignment in large language models. Importantly, we prove that within the limits of this framework, for any behavior that has a finite probability of being exhibited by the model, there exist prompts that can trigger the model into outputting this behavior, with probability that increases with the length of the prompt. This implies that any alignment process that attenuates an undesired behavior but does not remove it altogether, is not safe against adversarial prompting attacks. Furthermore, our framework hints at the mechanism by which leading alignment approaches such as reinforcement learning from human feedback make the LLM prone to being prompted into the undesired behaviors. This theoretical result is being experimentally demonstrated in large scale by the so called contemporary “chatGPT jailbreaks”, where adversarial users trick the LLM into breaking its alignment guardrails by triggering it into acting as a malicious persona. Our results expose fundamental limitations in alignment of LLMs and bring to the forefront the need to devise reliable mechanisms for ensuring AI safety.

Added

2026-09-30