Unveiling the Implicit Toxicity in Large Language Models

Jiaxin WenPei KeHao SunZhexin ZhangChengfei LiJinfeng BaiMinlie Huang

article2023EMNLP73 citations

Reveals that large language models can generate subtle, implicit toxicity that evades standard safety filters, and introduces a reinforcement learning attack framework that exposes these safety blind spots while providing training data to improve classifier defenses.

Listen

As large language models become widely deployed across consumer and enterprise applications, automated safety filters are essential to prevent harmful or abusive outputs. While existing safety mechanisms reliably catch explicit toxicity containing profanity or overt hate speech, bad actors can exploit generative models to convey toxicity through subtle, indirect language. Addressing implicit toxicity has become an urgent priority, as deployed systems risk spreading harmful content undetected if existing moderation tools fail to recognize nuanced harmful rhetoric.

The article demonstrates that large language models possess an innate ability to generate implicit toxic content that consistently bypasses current state-of-the-art moderation systems. It also establishes a reinforcement learning framework that further optimizes language models to produce highly evasive implicit toxicity, while evaluating whether retraining safety classifiers on this generated data can close the resulting security gap.

The researchers evaluated their methods on a standard benchmark dialogue dataset using a three-stage machine learning pipeline. First, they warm-started a base model using automated zero-shot examples generated by an instruction-tuned model. Second, they trained a reward model to score and prefer implicitly toxic responses over overtly toxic or benign ones, while incorporating penalties from an existing moderation filter. Third, they fine-tuned open-source language models—scaling from 1.3 billion to 13 billion parameters—using reinforcement learning to optimize against this reward function. The evasiveness of the generated text was then evaluated across five widely used industry and research toxicity classifiers, with ground truth verified by independent human annotators.

The findings reveal that standard commercial and open-source toxicity classifiers are highly vulnerable to large language model outputs. Zero-shot prompting alone generated implicit toxicity that bypassed classifiers 58% to nearly 97% of the time. Reinforcement learning fine-tuning exacerbated these vulnerabilities significantly: on a fine-tuned 13-billion-parameter model, the rate of undetected toxic responses reached 90% against a standard dialogue classifier and roughly 63% against an advanced language-model-based evaluator. Additionally, larger language models proved substantially more effective at generating evasive content because they combine multiple complex rhetorical devices, such as sarcasm, euphemism, and circumlocution. Despite these vulnerabilities, fine-tuning moderation classifiers on just 4,000 annotated examples of these evasive responses improved their detection rates against implicit toxicity from under 38% to roughly 79% without compromising general accuracy.

These results demonstrate a critical operational and compliance risk: current automated moderation systems provide a false sense of security, allowing subtly harmful outputs to slip through undetected. Organizations deploying generative AI risk reputational damage and policy violations if they rely solely on conventional toxicity filters. However, the study also proves that the same adversarial techniques can be used proactively to red-team models and strengthen automated moderation layers prior to deployment.

Organizations developing and deploying generative AI systems should implement red-teaming pipelines that specifically test for implicit toxicity rather than relying solely on explicit keyword detection. Moderation classifiers should be retrained and augmented using datasets composed of subtle, multi-feature toxic examples. Furthermore, because detecting implicit toxicity often requires contextual knowledge and linguistic reasoning, organizations should explore multi-layered evaluation systems that prompt evaluator models to explicitly analyze semantic nuances before classifying text as safe.

The primary limitation of this work involves noise in the automatically annotated comparison data used during reward modeling, which occasionally mislabeled subtle toxicity as benign. Additionally, the experimental scope was limited to models up to 13 billion parameters due to compute constraints, meaning the evasive potential of frontier-scale models remains an area requiring ongoing investigation. Nevertheless, the findings provide high confidence that current moderation classifiers suffer from systemic vulnerabilities against implicit toxicity and demonstrate a viable pathway to remediate these weaknesses.

classifiers large language models reinforcement learning implicit toxic language large language models implicit hate speech

arXiv: 2311.17391 thu-coai/Implicit-Toxicity

Paper: ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection, Thomas Hartvigsen et al. (2022). Read ToxiGen first to understand the machine-generated implicit-hate benchmark and detection weaknesses that this study’s evasive-toxicity experiments build on.
Paper: Red Teaming Language Models with Language Models, Ethan Perez et al. (2022). Its language-model red-teaming framework provides the foundation for understanding how this study uses generated adversarial outputs to probe safety failures.
Paper: RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models, Samuel Gehman et al. (2020). RealToxicityPrompts establishes the earlier benchmark tradition for measuring toxic language generation that frames this study’s evaluation.

Paper: Disarming Strategic Text: Span-Aware Counterfactuals for Robust Content Moderation, Hardik Meisheri et al. (2025). This work carries the study’s concern with evasive language into moderation training, using span-aware counterfactuals to make classifiers more robust to strategic rephrasing.

Cover for Unveiling the Implicit Toxicity in Large Language Models

Abstract

The open-endedness of large language models (LLMs) combined with their impressive capabilities may lead to new safety issues when being exploited for malicious use. While recent studies primarily focus on probing toxic outputs that can be easily detected with existing toxicity classifiers, we show that LLMs can generate diverse implicit toxic outputs that are exceptionally difficult to detect via simply zero-shot prompting. Moreover, we propose a reinforcement learning (RL) based attacking method to further induce the implicit toxicity in LLMs. Specifically, we optimize the language model with a reward that prefers implicit toxic outputs to explicit toxic and non-toxic ones. Experiments on five widely-adopted toxicity classifiers demonstrate that the attack success rate can be significantly improved through RL fine-tuning. For instance, the RL-finetuned LLaMA-13B model achieves an attack success rate of 90.04% on BAD and 62.85% on Davinci003. Our findings suggest that LLMs pose a significant threat in generating undetectable implicit toxic outputs. We further show that fine-tuning toxicity classifiers on the annotated examples from our attacking method can effectively enhance their ability to detect LLM-generated implicit toxic language. The code is publicly available at https://github.com/thu-coai/Implicit-Toxicity.

Topics

classifiers large language models reinforcement learning statistical language models n-gram language models statistical language modeling

Keywords

implicit toxic language large language models implicit hate speech toxicity detection toxicity classification dialogue generation figurative language

1 Introduction
2 Preliminary Experiments on Implicit Toxicity in Large Language Models
3 Methodology
3.1 Supervised Learning
3.2 Reward Model Training
3.3 Reinforcement Learning
4 Experiments
4.1 Settings
4.2 Baselines
4.3 Attacked Toxicity Classifiers
4.4 Metrics
4.5 Main Results
4.6 Analysis
4.7 Analysis of Implicit Toxicity in LLMs
4.8 Improving Toxicity Classifiers
5 Related Work
5.1 Safety Issues of Language Models
5.2 Toxicity Detection
6 Conclusion
Limitations
Ethics Statement
Acknowledgements
References
A Proximal Policy Optimization
B Implementation Details
B.1 Data Preprocessing
B.2 Training Details
B.3 Example Prompts
B.3.1 Supervised Learning Data Collection
B.3.2 Comparison Data Collection
B.3.3 Toxicity Classification
C Generated Examples
D Instructions for Human Annotation
E Scaling Properties of Linguistic Features

Knowls

Knowl 1 — A three-stage reinforcement-learning attack induces implicit toxicity
model/method
The method trains a language model to produce toxic responses that imply harmful content without relying on overtly offensive wording. It has three stages. First, an instruction-tuned model such as GPT-3.5-turbo generates responses to queries; the prompts request non-toxic, explicitly toxic, and implicitly toxic responses. A trainable policy is warm-started on the generated query-response pairs with maximum-likelihood training. Second, the current policy generates candidate responses for each query. GPT-3.5-turbo labels each response as implicit toxic, explicit toxic, or non-toxic; implicit-toxic responses are preferred to the other two classes, which are treated as equally preferred. The resulting comparisons train a reward model with the pairwise loss $-\log\sigma(R_\theta(x,y^w)-R_\theta(x,y^l))$ , where $x$ is a query, $y^w$ and $y^l$ are the preferred and less-preferred responses, $R_\theta$ is a scalar reward model, and $\sigma$ is the logistic function. Third, proximal policy optimization (PPO) updates the policy using a reward that combines the learned preference signal, a toxicity-classifier penalty, and a KL penalty against the supervised policy:

$\hat R_{\theta,\phi}(x,y)=R_\theta(x,y)-\alpha P(\mathrm{toxic}\mid x,y)-\beta D_{\mathrm{KL}}(\pi_\phi\|\pi_0).$

Here $y$ is a generated response, $P(\mathrm{toxic}\mid x,y)$ is the toxicity probability assigned to an existing classifier, $\pi_\phi$ is the policy being optimized, and $\pi_0$ is the supervised policy; the KL term penalizes divergence between their response distributions. The classifier penalty discourages responses that the classifier readily recognizes as toxic, while the learned reward preserves a signal favoring toxicity rather than merely steering generation toward non-toxicity. In the main setup, the policy and reward model use LLaMA-13B, the classifier penalty uses a BAD-trained RoBERTa classifier, and $\alpha=5$ and $\beta=0.1$ . PPO maximizes expected reward over queries and policy-generated responses.

Knowl 2 — RL fine-tuning raises attack success across five toxicity classifiers

empirical result

The authors compared toxic benchmark data, zero-shot GPT-3.5-turbo responses, and three LLaMA-13B generation methods on five toxicity classifiers. Attack success rate (ASR) is the percentage of toxic responses classified as non-toxic; annotated toxic probability is the share judged toxic by human annotators. Reward is the learned reward-model score, and Distinct-4 is the reported four-gram diversity score. Classifier ASRs and annotated toxic probabilities are reported as percentages; reward and Distinct-4 are shown at the paper's reported precision. Sources distinguish crawled data, crowdsourced data, and language-model-generated data. SL is supervised fine-tuning; SL-R continues training on the top-ranked of five SL responses per query; RL is the proposed PPO method.

Test data	Source	Reward	Annotated toxic probability	P-API ASR	Moderation ASR	TOXIGEN ASR	BAD ASR	Davinci003 ASR	Distinct-4
Offensive Twitter	Crawl	-5.91	N/A	14.10	73.20	14.40	1.90	6.20	0.99
TOXIGEN	LM	-3.96	N/A	72.28	67.93	33.97	20.92	9.24	0.94
Latent Hatred	Crawl + CS	-3.86	N/A	72.92	74.64	42.14	16.09	11.37	0.98
BAD	CS + LM	-3.36	N/A	76.77	82.11	55.28	24.85	26.25	0.95
GPT-3.5-turbo	LM	0.78	56.91	96.69	96.69	75.14	64.09	58.47	0.93
SL LLaMA-13B	LM	0.35	54.02	97.03	94.64	69.05	64.29	58.34	0.91
SL-R LLaMA-13B	LM	1.01	55.23	99.41	95.27	75.15	68.64	56.80	0.87
RL LLaMA-13B	LM	2.47	58.84	99.55	97.81	82.51	90.16	62.85	0.85

The RL model achieved higher ASR than the SL model on all five classifiers, including an increase from 64.29% to 90.16% on BAD and from 58.34% to 62.85% on Davinci003. The results also show that classifier evasion is not limited to the BAD classifier used in the reward: ASR increased on the other four evaluators as well. At the same time, human-annotated toxicity rose from 54.02% for SL to 58.84% for RL.

Knowl 3 — Human-labeled LLM outputs improve toxicity detection
empirical result
The authors collected 4,000 human-labeled toxic responses generated by language models—2,000 from GPT-3.5-turbo and 2,000 from RL-finetuned LLaMA-13B—and fine-tuned a RoBERTa-base classifier on these examples augmented with the BAD dataset. Toxic recall on three test sources was compared with classifiers trained on TOXIGEN, BAD, and Davinci003 data. The values below are percentages; columns identify the source of the toxic test responses.

Classifier BAD test GPT-3.5-turbo test RL LLaMA-13B test
TOXIGEN 44.72 24.86 17.49
BAD 75.15 35.91 9.84
Davinci003 70.62 41.53 37.16
Ours 78.16 82.32 78.69
Ours without RL data 76.31 80.11 71.58

The model trained on the authors' data achieved 82.32% recall on GPT-3.5-turbo responses and 78.69% on RL LLaMA-13B responses, compared with 35.91% and 9.84%, respectively, for the BAD-trained classifier. It also achieved 78.16% recall on BAD test data, above the BAD classifier's 75.15%. Excluding RL-generated examples reduced recall on RL LLaMA-13B responses to 71.58%.
Knowl 4 — Implicit toxicity is conveyed without overtly offensive wording
definition
In this work, an implicitly toxic response implies toxicity or harmfulness without using overtly toxic or offensive words. Its meaning may depend on figurative or indirect language—such as euphemism, circumlocution, sarcasm, metaphor, antithesis, or rhetorical questions—or on extralinguistic knowledge and reasoning. This distinguishes the target from explicit abuse and explains why detecting it can require interpreting the response in its query context rather than matching offensive terms.
Knowl 5 — Evaluation uses BAD-derived queries, human judgments, and five classifiers
experimental setup
Queries were extracted from the human utterances in the BAD chatbot-adversarial dataset, which contains nearly 6,000 dialogues; nonsense greetings were filtered. The reported query counts and mean lengths were 14,712 and 15.42 tokens for supervised learning, 22,441 and 15.25 for reward-model training, 22,441 and 15.25 for reinforcement learning, and 311 and 15.21 for testing. LLaMA-13B was the main policy and reward-model backbone. The toxicity classifier included in the RL reward was a 125M-parameter RoBERTa-base model fine-tuned on BAD.

The five attack evaluators were Perspective-API, OpenAI Moderation, RoBERTa-base classifiers fine-tuned on TOXIGEN and BAD, and a zero-shot Davinci003 toxicity classifier. Three annotators—college students majoring in English, recruited through a professional annotation company—provided toxic/non-toxic judgments used as gold labels. The authors report moderate-to-substantial inter-annotator agreement by Fleiss' kappa. They measured learned reward, Distinct-4 diversity, human-annotated toxic probability, classifier ASR on toxic responses, and continuous toxic confidence.

For LLaMA-13B training, supervised fine-tuning used batch size 16, learning rate $2\times10^{-7}$ , and at most 10 epochs. Reward-model training used batch size 32, learning rate $10^{-5}$ , and at most 5 epochs; the model was initialized from the supervised policy and its first 70% of hidden layers were frozen. RL used batch size 384 and learning rate $5\times10^{-6}$ , froze the first 80% of hidden layers, and shared parameters between policy and value functions. Experiments ran on eight 80GB A100 GPUs.

Classifier	BAD test	GPT-3.5-turbo test	RL LLaMA-13B test
TOXIGEN	44.72	24.86	17.49
BAD	75.15	35.91	9.84
Davinci003	70.62	41.53	37.16
Ours	78.16	82.32	78.69
Ours without RL data	76.31	80.11	71.58

Knowl 6 — The learned preference signal and classifier penalty have complementary roles

empirical result

Reward ablations were evaluated by learned reward, human-annotated toxic probability, and average ASR across the five toxicity classifiers. Removing the reward model mainly reduced the production of toxic responses, whereas removing the toxicity classifier penalty reduced classifier evasion.

Variant	Reward	Annotated toxic probability (%)	Average ASR (%)
SL LLaMA-13B	0.35	54.02	76.67
RL LLaMA-13B	2.47	58.84	86.58
RL without classifier penalty $P$	1.89	54.61	81.75
RL without learned reward $R_\theta$	0.42	20.90	86.34

Thus, removing $R_\theta$ left ASR high but lowered annotated toxic probability from 58.84% to 20.90%, indicating that the classifier penalty alone tends to favor non-toxic output. Removing $P$ lowered average ASR from 86.58% to 81.75%, showing that the classifier penalty contributes to evasion. In coefficient analyses, increasing the KL coefficient $\beta$ reduced reward and toxic probability; BAD ASR first rose and then fell, and the authors selected $\beta=0.1$ to avoid excessive reward-model optimization. Raising classifier coefficient $\alpha$ over a moderate range improved ASR while preserving comparable toxic probability, but too large a value substantially reduced toxic probability.

Knowl 7 — Larger models use more linguistic features per implicit-toxic response
empirical result
To examine scaling, the authors manually labeled linguistic features in responses from RL-finetuned LLaMA-1.3B and LLaMA-13B on 50 randomly sampled queries per model. With larger model size, the average number of features per response rose from 0.89 to 1.40. The reported percentages below describe the occurrence of each feature in the sampled responses.

Model size Average features per response Sarcasm Circumlocution Euphemism Antithesis Metaphor Rhetorical question
1.3B 0.89 65.0% 22.5% 10.0% 0.0% 2.5% 0.0%
13B 1.40 34.9% 27.0% 19.0% 12.7% 3.2% 3.2%

Using the same supervised and RL data, the authors also observed that ASR increased across the tested model scales from 1.3B to 13B, with the 13B model attaining the highest toxic probability and ASR. They interpret the feature-labeling results as evidence that larger models combine a broader range of linguistic devices; the proposed explanation that larger models absorb more relevant knowledge during pretraining is presented as a conjecture.
Knowl 8 — Generated implicit toxicity spans multiple toxic categories
empirical result
Human evaluation grouped generated toxic responses into four categories: offending the user, unfairness and discrimination, toxic agreement with harmful input, and sensitive-topic content. The distributions differed between GPT-3.5-turbo and RL-finetuned LLaMA-13B.

Toxic type GPT-3.5-turbo RL LLaMA-13B
Offending user 33.33% 39.20%
Unfairness and discrimination 33.90% 29.55%
Toxic agreement 22.03% 23.30%
Sensitive topic 10.74% 7.95%

The results show that the generated toxicity was not confined to one target or category. The authors caution that the proportions depend strongly on the input context; for example, queries designed to elicit an insulting response can increase the share of offending-user cases.
Knowl 9 — Some successful attacks require contextual knowledge and reasoning to detect
empirical result
The authors manually inspected toxic responses from GPT-3.5-turbo and RL LLaMA-13B that all five evaluated classifiers had misclassified. In the examples they analyzed, identifying the harmful implication required combining the query with contextual or world knowledge and interpreting indirect linguistic cues. In a diagnostic test, adding a human-written explanation of the relevant implication to the Davinci003 classifier prompt enabled it to identify toxicity that it had missed when given only the query and response. This case study indicates that the observed misses can involve reasoning demands, not just the absence of obvious toxic keywords; it does not establish that explanations generally solve the detection problem.
Knowl 10 — Automatic preference labels and model scale limit the demonstrated attack
limitation
Reward-model comparisons were labeled automatically by GPT-3.5-turbo to reduce annotation cost. The authors report a toxic recall of 68.8% for this labeler in preliminary experiments, and manual review found that nearly 30% of responses assigned to the non-toxic subset were actually implicitly toxic. This noise and bias limit the quality of the reward signal, so the RL policy cannot be expected to find all possible implicit-toxic language. The experiments also did not cover extra-large policy models such as LLaMA-65B or GPT-3.5-turbo, owing to computational-resource or access constraints.

Model size	Average features per response	Sarcasm	Circumlocution	Euphemism	Antithesis	Metaphor	Rhetorical question
1.3B	0.89	65.0%	22.5%	10.0%	0.0%	2.5%	0.0%
13B	1.40	34.9%	27.0%	19.0%	12.7%	3.2%	3.2%

Toxic type	GPT-3.5-turbo	RL LLaMA-13B
Offending user	33.33%	39.20%
Unfairness and discrimination	33.90%	29.55%
Toxic agreement	22.03%	23.30%
Sensitive topic	10.74%	7.95%

Coverage note — No substantial contributed material was omitted; individual prompt templates and example responses are illustrative instances rather than independent findings.

References

1.Abubakar Abid, Maheen Farooqi, and James Zou. 2021. Persistent anti-muslim bias in large language models. In Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society, pages 298–306.
2.Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
3.Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom B Brown, Dawn Song, Ulfar Erlingsson, et al. 2021. Extracting training data from large language models. In USENIX Security Symposium, volume 6.
4.Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311.
5.Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416.
6.Thomas Davidson, Dana Warmsley, Michael W. Macy, and Ingmar Weber. 2017. Automated hate speech detection and the problem of offensive language. In Proceedings of the Eleventh International Conference on Web and Social Media, pages 512–515.
7.Ameet Deshpande, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. 2023. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint arXiv:2304.05335.
8.Emily Dinan, Samuel Humeau, Bharath Chintagunta, and Jason Weston. 2019. Build it break it fix it for dialogue safety: Robustness from adversarial human attack. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 4536–4545.
9.Mai ElSherief, Caleb Ziems, David Muchlinski, Vaishnavi Anupindi, Jordyn Seybolt, Munmun De Choudhury, and Diyi Yang. 2021. Latent hatred: A benchmark for understanding implicit hate speech. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 345–363.
10.Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin, 76(5):378.
11.Jane Frank. 1990. You call that a rhetorical question?: Forms and functions of rhetorical questions in conversation. Journal of Pragmatics, 14(5):723–738.
12.Simona Frenda, Alessandra Teresa Cignarella, Valerio Basile, Cristina Bosco, Viviana Patti, and Paolo Rosso. 2022. The unbearable hurtfulness of sarcasm. Expert Systems with Applications, 193:116398.
13.Deep Ganguli, Danny Hernandez, Liane Lovitt, Amanda Askell, Yuntao Bai, Anna Chen, Tom Conerly, Nova Dassarma, Dawn Drain, Nelson Elhage, et al. 2022. Predictability and surprise in large generative models. In 2022 ACM Conference on Fairness, Accountability, and Transparency, pages 1747–1764.
14.Lei Gao and Ruihong Huang. 2017. Detecting online hate speech using context aware models. In Proceedings of the International Conference Recent Advances in Natural Language Processing, pages 260–266.
15.Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Realtoxicityprompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369.
16.Google. 2023. perspective.
17.Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3309–3326.
18.Borja Ibarz, Jan Leike, Tobias Pohlen, Geoffrey Irving, Shane Legg, and Dario Amodei. 2018. Reward learning from human preferences and demonstrations in atari. Advances in neural information processing systems, 31.
19.Liwei Jiang, Jena D Hwang, Chandra Bhagavatula, Ronan Le Bras, Jenny Liang, Jesse Dodge, Keisuke Sakaguchi, Maxwell Forbes, Jon Borchardt, Saadia Gabriel, et al. 2021. Can machines learn morality? the delphi experiment. arXiv e-prints, pages arXiv–2110.
20.Jens Lemmens, Ilia Markov, and Walter Daelemans. 2021. Improving hate speech type and target detection with hateful metaphor features. In Proceedings of the Fourth Workshop on NLP for Internet Freedom: Censorship, Disinformation, and Propaganda, pages 7–16.
21.Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. A diversity-promoting objective function for neural conversation models. In The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 110–119.
22.Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint arXiv:2303.16634.
23.Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
24.Rijul Magu and Jiebo Luo. 2018. Determining code words in euphemistic hate speech using word embedding networks. In Proceedings of the 2nd workshop on abusive language online (ALW2), pages 93–100.
25.Nicolas Ocampo, Ekaterina Sviridova, Elena Cabrio, and Serena Villata. 2023. An in-depth analysis of implicit and subtle hate speech messages. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1989–2005.
26.OpenAI. 2022. Introducing chatgpt.
27.OpenAI. 2023a. GPT-4 technical report. arXiv preprint arXiv:2303.08774.
28.OpenAI. 2023b. moderation.
29.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744.
30.Ethan Perez, Saffron Huang, H. Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448.
31.Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
32.Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick S. H. Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander H. Miller. 2019. Language models as knowledge bases? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pages 2463–2473.
33.Stephen Roller, Emily Dinan, Naman Goyal, Da Ju, Mary Williamson, Yinhan Liu, Jing Xu, Myle Ott, Eric Michael Smith, Y-Lan Boureau, and Jason Weston. 2021. Recipes for building an open-domain chatbot. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 300–325.
34.Nigora Ruzibaeva. 2019. Peculiarities of the antithesis in the literary text. European Journal of Research and Reflection in Educational Sciences Vol, 7(11).
35.Maarten Sap, Saadia Gabriel, Lianhui Qin, Dan Jurafsky, Noah A. Smith, and Yejin Choi. 2020. Social bias frames: Reasoning about social and power implications of language. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5477–5490.
36.John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
37.Rohit Sridhar and Diyi Yang. 2022. Explaining toxic text via knowledge enhanced text generation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 811–826.
38.Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008–3021.
39.Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436.
40.Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.
41.Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is chatgpt a good nlg evaluator? a preliminary study. arXiv preprint arXiv:2303.04048.
42.Jing Xu, Da Ju, Margaret Li, Y-Lan Boureau, Jason Weston, and Emily Dinan. 2020. Recipes for safety in open-domain chatbots. arXiv preprint arXiv:2010.07079.
43.Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020. DIALOGPT : Large-scale generative pre-training for conversational response generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 270–278.
44.Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023a. Safety-bench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045.
45.Zhexin Zhang, Jiaxin Wen, and Minlie Huang. 2023b. ETHICIST: Targeted training data extraction through loss smoothed soft prompting and calibrated confidence estimation. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12674–12687, Toronto, Canada. Association for Computational Linguistics.

Citation

MLA

Wen, J., et al. “Unveiling the Implicit Toxicity in Large Language Models”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 2023, pp. 1322–38, https://doi.org/10.18653/v1/2023.emnlp-main.84.

APA

Wen, J., Ke, P., Sun, H., Zhang, Z., Li, C., Bai, J., & Huang, M. (2023). Unveiling the Implicit Toxicity in Large Language Models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 1322–1338. https://doi.org/10.18653/v1/2023.emnlp-main.84

Chicago

Wen, J., P. Ke, H. Sun, et al. 2023. “Unveiling the Implicit Toxicity in Large Language Models”. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 1322–38. https://doi.org/10.18653/v1/2023.emnlp-main.84.

Harvard

Wen, J. et al. (2023) “Unveiling the Implicit Toxicity in Large Language Models”, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp. 1322–1338. Available at: https://doi.org/10.18653/v1/2023.emnlp-main.84.

Vancouver

1. Wen J, Ke P, Sun H, Zhang Z, Li C, Bai J, Huang M (2023) Unveiling the Implicit Toxicity in Large Language Models. In: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, pp 1322–1338

BibTeX

@inproceedings{wen-etal-2023-unveiling,
    title = "Unveiling the Implicit Toxicity in Large Language Models",
    author = "Wen, Jiaxin  and
      Ke, Pei  and
      Sun, Hao  and
      Zhang, Zhexin  and
      Li, Chengfei  and
      Bai, Jinfeng  and
      Huang, Minlie",
    editor = "Bouamor, Houda  and
      Pino, Juan  and
      Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.84/",
    doi = "10.18653/v1/2023.emnlp-main.84",
    pages = "1322--1338"
}

Metadata:ACL Anthology

Source Code

This paper has an official code repository available. Click below to access the source code.

View Repository

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF

License: https://creativecommons.org/licenses/by/4.0/

Unveiling the Implicit Toxicity in Large Language Models

Abstract

Topics

Keywords

Table of Contents

Knowls

Knowl 1 — A three-stage reinforcement-learning attack induces implicit toxicity

Knowl 2 — RL fine-tuning raises attack success across five toxicity classifiers

Knowl 3 — Human-labeled LLM outputs improve toxicity detection

Knowl 4 — Implicit toxicity is conveyed without overtly offensive wording

Knowl 5 — Evaluation uses BAD-derived queries, human judgments, and five classifiers

Knowl 6 — The learned preference signal and classifier penalty have complementary roles

Knowl 7 — Larger models use more linguistic features per implicit-toxic response

Knowl 8 — Generated implicit toxicity spans multiple toxic categories

Knowl 9 — Some successful attacks require contextual knowledge and reasoning to detect

Knowl 10 — Automatic preference labels and model scale limit the demonstrated attack

References

Citation

BibTeX

Source Code

Access the Paper