Built independently by an author, for readers. Read the story and support ChapterPal
ChapterPal Logo
Log inSign up

keyword

toxicity classification

Toxicity classification is a natural language processing task that uses machine learning algorithms to automatically identify and categorize harmful, offensive, or abusive content within text. This process involves analyzing digital text to distinguish benign communication from various forms of toxic language, which can include explicit hate speech, harassment, profanity, and threats, as well as subtle, context-dependent implicit toxicity such as microaggressions or veiled insults. Widely applied in content moderation platforms, online discussion forums, and artificial intelligence safety auditing, toxicity classification helps filter inappropriate material, enforce community standards, and monitor outputs from generative language models to maintain safe and respectful communication environments.

1 item

Unveiling the Implicit Toxicity in Large Language Models

Unveiling the Implicit Toxicity in Large Language Models

Jiaxin Wen, Pei Ke, Hao Sun, Zhexin Zhang, Chengfei Li, Jinfeng Bai, Minlie Huang

Why you should read this

Reveals that large language models can generate subtle, implicit toxicity that evades standard safety filters, and introduces a reinforcement learning attack framework that exposes these safety blind spots while providing training data to improve classifier defenses.

The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. While recent studies primarily focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, we show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. Moreover, we propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs. Specifically, we optimize the language model with a reward that prefers implicit toxic outputs to explicit toxic and non-toxic ones. Experiments on five widely-adopted toxicity classifiers demonstrate that the attack success rate can be significantly improved through RL fine-tuning. For instance, the RL-finetuned LLaMA-13B model achieves an attack success rate of 90.04% on BAD and 62.85% on Davinci003. Our findings suggest that LLMs pose a significant threat in generating undetectable implicit toxic outputs. We further show that fine-tuning toxicity classifiers on the annotated examples from our attacking method can effectively enhance their ability to detect LLM-generated implicit toxic language. The code is publicly available at https://github.com/thu-coai/Implicit-Toxicity.

Added

2026-10-03