Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

toxic prompts

Toxic prompts are text inputs, queries, or sentence prefixes provided to artificial intelligence language models that contain offensive, hateful, or harmful language, or are designed to elicit toxic responses from the system. In artificial intelligence safety research and evaluation, these prompts are utilized as benchmarking tools to measure the propensity of a model to generate inappropriate content, such as profanity, insults, or biased remarks, and to test the robustness of safety guardrails and moderation filters. Such prompts can range from overtly abusive statements to subtly provocative or incomplete phrases that inadvertently trigger harmful text generation learned from uncurated pretraining data.

1 item

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models

Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, Noah A. Smith

OrganizationsAllen Institute for AIUniversity of Washington

Why you should read this

Introduces RealToxicityPrompts, a 100,000-prompt benchmark to evaluate toxic degeneration in language models, demonstrating that benign prompts can trigger severe toxicity and that current mitigation methods remain inadequate.

Pretrained neural language models (LMs) are prone to generating racist, sexist, or otherwise toxic language which hinders their safe deployment. We investigate the extent to which pretrained LMs can be prompted to generate toxic language, and the effectiveness of controllable text generation algorithms at preventing such toxic degeneration. We create and release RealToxicityPrompts, a dataset of 100K naturally occurring, sentence-level prompts derived from a large corpus of English web text, paired with toxicity scores from a widely-used toxicity classifier. Using RealToxicityPrompts, we find that pretrained LMs can degenerate into toxic text even from seemingly innocuous prompts. We empirically assess several controllable generation methods, and find that while data- or compute-intensive methods (e.g., adaptive pretraining on non-toxic data) are more effective at steering away from toxicity than simpler solutions (e.g., banning "bad" words), no current method is failsafe against neural toxic degeneration. To pinpoint the potential cause of such persistent toxic degeneration, we analyze two web text corpora used to pretrain several LMs (including GPT-2; Radford et. al, 2019), and find a significant amount of offensive, factually unreliable, and otherwise toxic content. Our work provides a test bed for evaluating toxic generations by LMs and stresses the need for better data selection processes for pretraining.

Added

2026-09-18