keyword
harmful queries
Harmful queries are user inputs or prompts submitted to an artificial intelligence system that seek to elicit unsafe, unethical, toxic, or dangerous outputs. In the context of large language models and conversational AI, these queries encompass requests involving illegal acts, cyberattacks, hate speech, self-harm, fraud, or the creation of hazardous materials. While some harmful queries are straightforward policy violations, others are sophisticated adversarial attacks or jailbreak attempts designed to manipulate the system and circumvent its safety filters and alignment guardrails. Identifying and neutralizing harmful queries is a core focus of AI safety research, which employs safety benchmarks, detoxification methods, and robust alignment training to prevent automated systems from producing damaging information.
2 items
Detoxifying Large Language Models via Knowledge Editing
Mengru Wang, Ningyu Zhang, Ziwen Xu, Zekun Xi, Shumin Deng, Yunzhi Yao, Qishen Zhang, Linyi Yang, Jindong Wang, Huajun Chen
Why you should read this
Presents the SafeEdit benchmark and a single-instance knowledge editing method, DINM, to directly modify toxic model parameters rather than merely suppressing their activations, effectively detoxifying large language models without degrading their general performance.
This paper investigates using knowledge editing techniques to detoxify Large Language Models (LLMs). We construct a benchmark, SafeEdit, which covers nine unsafe categories with various powerful attack prompts and equips comprehensive metrics for systematic evaluation. We conduct experiments with several knowledge editing approaches, indicating that knowledge editing has the potential to detoxify LLMs with a limited impact on general performance efficiently. Then, we propose a simple yet effective baseline, dubbed Detoxifying with Intraoperative Neural Monitoring (DINM), to diminish the toxicity of LLMs within a few tuning steps via only one instance. We further provide an in-depth analysis of the internal mechanism for various detoxifying approaches, demonstrating that previous methods like SFT and DPO may merely suppress the activations of toxic parameters, while DINM mitigates the toxicity of the toxic parameters to a certain extent, making permanent adjustments. We hope that these insights could shed light on future work of developing detoxifying approaches and the underlying knowledge mechanisms of LLMs¹.
Added
2026-10-02
How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs
Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, Weiyan Shi
Why you should read this
Establishes a social-science-derived persuasion taxonomy to generate human-readable persuasive adversarial prompts that achieve over a 92% jailbreak success rate against top aligned models like GPT-4 and Llama-2 without requiring complex optimization algorithms.
Added
2026-09-26