Back

Open Problems in Machine Unlearning for AI Safety

Fazl BarezTingchen FuAmeya PrabhuStephen CasperAmartya SanyalAdel BibiAidan O'GaraRobert KirkBen BucknallTim Fist

article2025arXiv62 citations

Exposes critical limitations of using machine unlearning for AI safety, showing why attempting to erase hazardous dual-use knowledge in domains like cybersecurity and biosecurity degrades beneficial capabilities and conflicts with existing safety mechanisms.

Listen

As artificial intelligence systems grow increasingly capable and autonomous across sensitive fields such as cybersecurity, biology, and healthcare, controlling hazardous behaviors has become an urgent safety priority. The article evaluates machine unlearning—the process of selectively removing or suppressing specific knowledge or capabilities from models—to determine whether it can serve as a dependable mechanism for safety control and risk mitigation.

The authors analyze current unlearning techniques, including gradient ascent, task vectors, model editing, representation misdirection, curated fine-tuning, adversarial training, and in-context methods. Rather than treating unlearning merely as a tool for discrete data removal, the analysis synthesizes empirical evidence and theoretical limits across capability control domains, such as chemical, biological, radiological, and nuclear hazards, as well as jailbreak prevention, value alignment, and regulatory compliance.

The findings demonstrate that machine unlearning cannot serve as a standalone solution for controlling dangerous AI capabilities. First, complex harmful capabilities are deeply entangled and distributed across parameters; even if specific hazardous data is removed, models can easily reconstruct dangerous outputs by recombining retained benign concepts. Second, current techniques typically mask rather than eliminate underlying knowledge, leaving models highly vulnerable to rapid relearning through lightweight, few-shot fine-tuning or subtle prompt alterations. Third, attempts at deep or iterative knowledge removal frequently suffer from severe performance trade-offs, leading to cumulative degradation of benign capabilities or creating safety blind spots in related tasks. Finally, existing evaluation benchmarks provide a false sense of security by measuring only immediate surface-level forgetting rather than long-term, adversarial robustness.

These limitations mean that relying on unlearning for high-stakes capability control introduces significant operational and safety risks. While unlearning remains viable for privacy compliance tasks, such as removing discrete personal data under data protection laws, it cannot reliably enforce complex behavioral boundaries or prevent dual-use misuse. Consequently, unlearning must be viewed as an auxiliary tool rather than a comprehensive safety safeguard.

Decision-makers should avoid relying on machine unlearning as a single point of failure for capability restriction. Instead, technical teams must pursue a defense-in-depth framework that combines rigorous adversarial evaluation, multi-layered access controls, monitoring, and non-unlearning safety architectures. Future research must prioritize causal intervention tools, predictive impact models, and standardized stress-testing benchmarks before unlearning can be trusted in safety-critical deployments.

Leverhulme Centre for the Future of IntelligenceMassachusetts Institute of TechnologyMila – Québec Artificial Intelligence InstituteNanyang Technological UniversitySingapore AI Safety InstituteTangenticTel Aviv UniversityUK AI Safety InstituteUniversitat Politècnica de ValènciaUniversity of CopenhagenUniversity of OxfordUniversity of Tübingenartificial intelligencecybersecurityprivacymachine unlearningAI safetyharmful fine-tuning attacks
arXiv: 2501.04952
Cover for Open Problems in Machine Unlearning for AI Safety

Abstract

As AI systems become more capable, widely deployed, and increasingly autonomous in critical areas such as cybersecurity, biological research, and healthcare, ensuring their safety and alignment with human values is paramount. Machine unlearning -- the ability to selectively forget or suppress specific types of knowledge -- has shown promise for privacy and data removal tasks, which has been the primary focus of existing research. More recently, its potential application to AI safety has gained attention. In this paper, we identify key limitations that prevent unlearning from serving as a comprehensive solution for AI safety, particularly in managing dual-use knowledge in sensitive domains like cybersecurity and chemical, biological, radiological, and nuclear (CBRN) safety. In these contexts, information can be both beneficial and harmful, and models may combine seemingly harmless information for harmful purposes -- unlearning this information could strongly affect beneficial uses. We provide an overview of inherent constraints and open problems, including the broader side effects of unlearning dangerous knowledge, as well as previously unexplored tensions between unlearning and existing safety mechanisms. Finally, we investigate challenges related to evaluation, robustness, and the preservation of safety features during unlearning. By mapping these limitations and open challenges, we aim to guide future research toward realistic applications of unlearning within a broader AI safety framework, acknowledging its limitations and highlighting areas where alternative approaches may be required.

Topics

artificial intelligencecybersecurityprivacycontext information

Keywords

machine unlearningAI safetyharmful fine-tuning attacksknowledge sanitizationadversarial evaluationssafety-critical knowledgedual-use knowledgeinference attacksmodel robustness

Table of Contents

  • 1 Introduction
  • 2 Applications of Unlearning for AI Safety: An Overview
  • 2.1 Unlearning for Safety-Critical Knowledge Management
  • 2.2 Correcting Value Alignment and Improving Corrigibility via Unlearning
  • 2.3 Verification and Governance Aspects of Unlearning
  • 2.4 Unlearning for Privacy, Data Protection and Legal Compliance
  • 3 Unlearning Practices and Evaluation Techniques for AI safety
  • 3.1 Evaluation Methods for Unlearning: What are the promising directions for AI Safety?
  • 3.1.1 What Metrics Would be Most Suitable for AI Safety Applications?
  • 3.2 Current Unlearning Methods
  • 4 Open Problems in Machine Unlearning for AI Safety
  • 4.1 Evaluation of Unlearning
  • 4.2 Robustness to Relearning
  • 4.3 Dual-use Capabilities Can Emerge From Beneficial Elements
  • 4.4 Context-Dependent Challenges
  • 4.5 Neural and Representational Level Interventions
  • 4.6 Continual and Iterative Unlearning
  • 4.7 Capability Interactions and Dependencies
  • 5 Conclusion
  • References

Knowls

  1. Knowl 1 — Capability removal is harder than removing identifiable knowledge

    theoretical result

    The paper’s central limitation is that a harmful capability need not be stored as one separable fact or training example. It can arise from combining distributed pieces of otherwise benign knowledge, potentially across domains. Removing a targeted corpus or representation therefore does not guarantee that the model can no longer reconstruct the capability through retained knowledge or alternative pathways. Capability control must contend with the tension between suppressing harmful combinations and preserving beneficial uses of their component knowledge.

  2. Knowl 2 — Unlearning requires forgetting while preserving unrelated utility

    definition

    Machine unlearning for AI safety modifies a system using a designated forget corpus: after the intervention, the system should no longer express or retain the knowledge or behaviors demonstrated in that corpus, while performance on tasks unrelated to it should remain unaffected. This objective is stronger than merely making private information harder to infer: it seeks to undo the effect of learning the targeted knowledge or behavior without broadly degrading system utility.

  3. Knowl 3 — Unlearning is a better fit for data removal than broad capability control

    empirical result

    The paper’s analysis finds that unlearning’s suitability varies by safety application. Removing identifiable data points for privacy and legal compliance is a comparatively tractable target, although technical verification remains probabilistic rather than a guarantee of binary deletion. By contrast, targeted removal alone is not a reliable way to change value alignment, which depends on broader model behavior and capabilities. Unlearning can address some specific vulnerabilities, such as effects of poisoned data, but cannot ensure that broader jailbreak or exploit behaviors disappear when they arise from functionality the system still needs.

  4. Knowl 4 — Safety evaluation must test reconstruction, not just direct reproduction

    limitation

    A model’s failure to reproduce a targeted example after unlearning does not establish that the underlying knowledge or capability is inaccessible. Evaluation for safety-critical unlearning must test whether a model can reconstruct the target indirectly by combining retained knowledge, express it in new forms or contexts, recover it during extended interactions, or reveal it under adversarial elicitation. Where relevant, tests should also examine internal mechanisms, not only input-output behavior. The paper identifies standardized, comprehensive, and computationally feasible verification of these properties as an open challenge.

  5. Knowl 5 — Unlearned knowledge can be rapidly relearned

    empirical result

    The literature surveyed in the paper reports that knowledge suppressed by unlearning can often be restored through few-shot fine-tuning; some studies also find recovery after fine-tuning on small amounts of benign, unrelated data. This vulnerability matters particularly for open-weight models and systems whose users can fine-tune them. Approaches intended to resist post-deployment modification are being explored, but the paper notes trade-offs with efficiency, stability, and performance on benign tasks; tamper-resistant unlearning remains an open problem.

  6. Knowl 6 — Dual-use knowledge makes removal context-dependent and can weaken safeguards

    limitation

    Whether information is dangerous can depend on the user and context. For example, detailed cybersecurity vulnerability knowledge may be useful to trusted professionals yet risky in other settings, making blanket removal a poor substitute for context-sensitive access control. Removing that knowledge may also impair a model’s ability to recognize related vulnerabilities and avoid them in new situations. Thus, preserving safety mechanisms can require retaining some knowledge that an unlearning intervention is intended to suppress.

  7. Knowl 7 — Unlearning evaluation has five distinct dimensions

    model/method

    The paper organizes evaluation of unlearning around five dimensions: (1) generalizability—whether forgetting extends to other expressions of the target; (2) locality—whether benign model capabilities remain intact; (3) efficiency and resource use—whether the intervention is feasible in time and computation; (4) robustness to adversarial attacks—including whether membership-inference and model-inversion attacks can reveal information; and (5) robustness to relearning—whether the target remains difficult to restore. These dimensions provide complementary measures; the paper cautions that passing standard task-specific tests does not, by itself, establish safety-critical capability prevention.

  8. Knowl 8 — Current unlearning methods intervene at different levels and have different failure modes

    model/method

    The paper surveys seven families of unlearning methods. Gradient ascent increases loss on the forget corpus, but unrestricted updates can damage unrelated abilities; methods may constrain updates with KL divergence or remaining-data losses, or restrict updates to selected parameters. Task vectors subtract a parameter-space difference between the original model and a model fine-tuned on the target, but can be sensitive to hyperparameters. Model editing changes selected activations, neurons, or output logits, requiring relevant representations to be located. Representation misdirection steers target-related hidden states toward noise; it may be vulnerable to adversarial fine-tuning and need not remove the underlying knowledge. Fine-tuning on curated data teaches refusal or uses anonymized versions of target data, but requires constructing suitable data. Adversarial training trains against input-, latent-, or weight-space modifications to improve robustness, at greater computational cost. In-context unlearning uses instructions or demonstrations at inference time and can be API-friendly, but may incur high computation and leaves parametric knowledge unchanged.

  9. Knowl 9 — Neural-level edits can have unintended and interacting effects

    limitation

    The mapping between weights, activations, semantic representations, and higher-level capabilities is not sufficiently understood to assume that a local intervention will have only local effects. Editing harmful-content representations can also impair contextual understanding in related benign domains, while distributed features and dependencies can cause modifications to propagate across capabilities. The paper identifies causal interventions, predictive models of how edits propagate, formal checks of safety bounds, and ongoing validation and monitoring as needed research directions. It also warns that repeated unlearning and retraining, compensatory behaviors, and combinations of safety techniques can produce emergent risks or compounded performance degradation.

  10. Knowl 10 — Repeated unlearning requires adaptive targets and protection against cumulative utility loss

    limitation

    Real systems may receive many unlearning requests over time, while current methods often trade forgetting effectiveness against preservation of utility. Processing requests sequentially can compound degradation because each intervention starts from an already modified model, potentially destabilizing it or erasing valuable knowledge. The paper identifies as open problems methods that scale to many requests, dynamically balance multiple objectives, refine targets as new harmful manifestations are discovered, and determine when requests should be combined or unlearning should be interleaved with continued learning.

Coverage note — Detailed examples involving reward hacking, situational awareness, and governance or formal-audit mechanisms are omitted because the paper presents them as illustrative application areas rather than independent results.

References

  1. 1.Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
  2. 2.Dario Amodei, Christopher Olah, Jacob Steinhardt, Paul Francis Christiano, John Schulman, and Dandelion Mané. Concrete problems in ai safety. ArXiv, abs/1606.06565, 2016. URL https://api.semanticscholar.org/CorpusID:10242377.
  3. 3.Arditi Andy. Unlearning via rmu is mostly shallow. In Less Wrong.
  4. 4.Anonymous. CASE-bench: Context-aware safety evaluation benchmark for large language models. In Submitted to The Thirteenth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=y9tQNJ2n1y. under review.
  5. 5.George-Octavian Barbulescu and Peter Triantafillou. To each (textual sequence) its own: Improving memorized-data unlearning in large language models. arXiv preprint arXiv:2405.03097, 2024.
  6. 6.Nora Belrose, Zach Furman, Logan Smith, Danny Halawi, Igor Ostrovsky, Lev McKinney, Stella Biderman, and Jacob Steinhardt. Eliciting latent predictions from transformers with the tuned lens. arXiv preprint arXiv:2303.08112, 2023.
  7. 7.Rishabh Bhardwaj and Soujanya Poria. Language model unalignment: Parametric red-teaming to expose hidden harms and biases. arXiv preprint arXiv:2310.14303, 2023.
  8. 8.Alberto Blanco-Justicia, Najeeb Jebreel, Benet Manzanares, David Sánchez, Josep Domingo-Ferrer, Guillem Collell, and Kuan Eeik Tan. Digital forgetting in large language models: A survey of unlearning methods. arXiv preprint arXiv:2404.02062, 2024.
  9. 9.Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE, 2021.
  10. 10.Trenton Bricken, Adly Templeton, Joshua Batson, Brian Chen, Adam Jermyn, Tom Conerly, Nick Turner, Cem Anil, Carson Denison, Amanda Askell, Robert Lasenby, Yifan Wu, Shauna Kravec, Nicholas Schiefer, Tim Maxwell, Nicholas Joseph, Zac Hatfield-Dodds, Alex Tamkin, Karina Nguyen, Brayden McLean, Josiah E Burke, Tristan Hume, Shan Carter, Tom Henighan, and Christopher Olah. Towards monosemanticity: Decomposing language models with dictionary learning. Transformer Circuits Thread, 2023. https://transformer-circuits.pub/2023/monosemantic-features/index.html.
  11. 11.California State Legislature. California Consumer Privacy Act of 2018, 2018. URL https://leginfo.legislature.ca.gov/faces/codes_displayText.xhtml?division=3.&part=4.&lawCode=CIV&title=1.81.5. Cal. Civ. Code § 1798.100 et seq.
  12. 12.Yinzhi Cao and Junfeng Yang. Towards making systems forget with machine unlearning. In 2015 IEEE symposium on security and privacy, pages 463–480. IEEE, 2015.
  13. 13.Joe Carlsmith. Scheming ais: Will ais fake alignment during training in order to get power? arXiv preprint arXiv:2311.08379, 2023.
  14. 14.Stephen Casper, Yuxiao Li, Jiawei Li, Tong Bu, Kevin Zhang, Kaivalya Hariharan, and Dylan Hadfield-Menell. Red teaming deep neural networks with feature synthesis tools. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA, 2024a. Curran Associates Inc.
  15. 15.Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030, 2024b.
  16. 16.Zachary Charles, Arun Ganesh, Ryan McKenna, H Brendan McMahan, Nicole Mitchell, Krishna Pillutla, and Keith Rush. Fine-tuning large language models with user-level differential privacy. arXiv preprint arXiv:2407.07737, 2024.
  17. 17.Jiaao Chen and Diyi Yang. Unlearn what you want to forget: Efficient unlearning for LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12041–12052, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.738. URL https://aclanthology.org/2023.emnlp-main.738.
  18. 18.Lichang Chen, Chen Zhu, Jiuhai Chen, Davit Soselia, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, and Bryan Catanzaro. ODIN: Disentangled reward mitigates hacking in RLHF. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=zcIV8OQFVF.
  19. 19.Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, pages 896–911, 2021.
  20. 20.Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Can bad teaching induce forgetting? unlearning in deep networks using an incompetent teacher. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-Fifth Conference on Innovative Applications of Artificial Intelligence and Thirteenth Symposium on Educational Advances in Artificial Intelligence, AAAI’23/IAAI’23/EAAI’23. AAAI Press, 2023a. ISBN 978-1-57735-880-0. doi: 10.1609/aaai.v37i6.25879. URL https://doi.org/10.1609/aaai.v37i6.25879.
  21. 21.Vikram S Chundawat, Ayush K Tarun, Murari Mandal, and Mohan Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 2023b.
  22. 22.Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. Or-bench: An over-refusal benchmark for large language models. arXiv preprint arXiv:2405.20947, 2024.
  23. 23.Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, Baobao Chang, and Furu Wei. Knowledge neurons in pretrained transformers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8493–8502, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.acl-long.581. URL https://aclanthology.org/2022.acl-long.581.
  24. 24.Patricia Gomes Rêgo de Almeida, Carlos Denner dos Santos, and Josivania Silva Farias. Artificial intelligence regulation: a framework for governance. Ethics and Information Technology, 23(3):505–525, 2021.
  25. 25.Aghyad Deeb and Fabien Roger. Do unlearning methods remove information from language model weights? arXiv preprint arXiv:2410.08827, 2024.
  26. 26.Jiangyi Deng, Shengyuan Pang, Yanjiao Chen, Liangming Xia, Yijie Bai, Haiqin Weng, and Wenyuan Xu. Sophon: Non-fine-tunable learning to restrain task transferability for pre-trained models. arXiv preprint arXiv:2404.12699, 2024.
  27. 27.Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? In First Conference on Language Modeling, 2024a. URL https://openreview.net/forum?id=av0D19pSkU.
  28. 28.Michael Duan, Anshuman Suri, Niloofar Mireshghallah, Sewon Min, Weijia Shi, Luke Zettlemoyer, Yulia Tsvetkov, Yejin Choi, David Evans, and Hannaneh Hajishirzi. Do membership inference attacks work on large language models? In Conference on Language Modeling (COLM), 2024b.
  29. 29.Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. In A. Oh, T. Naumann, K. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, pages 30039–30069. Curran Associates, Inc., 2023. URL https://proceedings.neurips.cc/paper_files/paper/2023/file/5fc47800ee5b30b8777fdd30abcaaf3b-Paper-Conference.pdf.
  30. 30.Ronen Eldan and Mark Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023.
  31. 31.European Parliament and Council of the European Union. General Data Protection Regulation (GDPR), 2016. URL https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32016R0679. Regulation (EU) 2016/679.
  32. 32.Eoin Farrell, Yeu-Tong Lau, and Arthur Conmy. Applying sparse autoencoders to unlearn knowledge in language models. arXiv preprint arXiv:2410.19278, 2024.
  33. 33.Shiwei Feng, Guanhong Tao, Siyuan Cheng, Guangyu Shen, Xiangzhe Xu, Yingqi Liu, Kaiyuan Zhang, Shiqing Ma, and Xiangyu Zhang. Detecting backdoors in pre-trained encoders. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16352–16362, 2023.
  34. 34.Alex Foote, Neel Nanda, Esben Kran, Ioannis Konstas, and Fazl Barez. N2g: A SCALABLE APPROACH FOR QUANTIFYING INTERPRETABLE NEURON REPRESENTATION IN LLMS. In ICLR 2023 Workshop on Trustworthy and Reliable Large-Scale Machine Learning Models, 2023. URL https://openreview.net/forum?id=ZB6bK6MTYq.
  35. 35.Jasmina Gajcin, James McCarthy, Rahul Nair, Radu Marinescu, Elizabeth Daly, and Ivana Dusparic. Iterative reward shaping using human feedback for correcting reward misspecification. arXiv preprint arXiv:2308.15969, 2023.
  36. 36.Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. Real-ToxicityPrompts: Evaluating neural toxic degeneration in language models. In Trevor Cohn, Yulan He, and Yang Liu, editors, Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/v1/2020.findings-emnlp.301. URL https://aclanthology.org/2020.findings-emnlp.301.
  37. 37.Shashwat Goel, Ameya Prabhu, Amartya Sanyal, Ser-Nam Lim, Philip Torr, and Ponnurangam Kumaraguru. Towards adversarial evaluations for inexact machine unlearning. arXiv preprint arXiv:2201.06640, 2022.
  38. 38.Shashwat Goel, Ameya Prabhu, Philip Torr, Ponnurangam Kumaraguru, and Amartya Sanyal. Corrective machine unlearning. arXiv preprint arXiv:2402.14015, 2024.
  39. 39.Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, and Evan Hubinger. Alignment faking in large language models, 2024. URL https://arxiv.org/abs/2412.14093.
  40. 40.Kang Gu, Md Rafi Ur Rashid, Najrin Sultana, and Shagufta Mehnaz. Second-order information matters: Revisiting machine unlearning for large language models. arXiv preprint arXiv:2403.10557, 2024.
  41. 41.Phillip Guo, Aaquib Syed, Abhay Sheshadri, Aidan Ewart, and Gintare Karolina Dziugaite. Mechanistic unlearning: Robust knowledge unlearning and editing via mechanistic localization. arXiv preprint arXiv:2410.12949, 2024.
  42. 42.Thomas Hartvigsen, Swami Sankaranarayanan, Hamid Palangi, Yoon Kim, and Marzyeh Ghassemi. Aging with GRACE: Lifelong model editing with discrete key-value adaptors. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=Oc1SIKxwdV.
  43. 43.Jamie Hayes, Ilia Shumailov, Eleni Triantafillou, Amr Khalifa, and Nicolas Papernot. Inexact unlearning needs more careful evaluations to avoid a false sense of privacy. arXiv preprint arXiv:2403.01218, 2024.
  44. 44.Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pages 287–296, 2023.
  45. 45.Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021. URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  46. 46.Dan Hendrycks, Mantas Mazeika, and Thomas Woodside. An overview of catastrophic ai risks. arXiv preprint arXiv:2306.12001, 2023.
  47. 47.Yihuai Hong, Lei Yu, Shauli Ravfogel, Haiqin Yang, and Mor Geva. Intrinsic evaluation of unlearning using parametric knowledge traces. arXiv preprint arXiv:2406.11614, 2024.
  48. 48.Chris Jay Hoofnagle, Bart Van Der Sloot, and Frederik Zuiderveen Borgesius. The european union general data protection regulation: what it is and what it means. Information & Communications Technology Law, 28(1):65–98, 2019.
  49. 49.Shengyuan Hu, Yiwei Fu, Zhiwei Steven Wu, and Virginia Smith. Jogging the memory of unlearned model through targeted relearning attack. arXiv preprint arXiv:2406.13356, 2024.
  50. 50.James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. Offset unlearning for large language models. ArXiv, abs/2404.11045, 2024a. URL https://api.semanticscholar.org/CorpusID:269187650.
  51. 51.Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  52. 52.Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18169, 2024b.
  53. 53.Tiansheng Huang, Sihao Hu, and Ling Liu. Vaccine: Perturbation-aware alignment for large language model. arXiv preprint arXiv:2402.01109, 2024c.
  54. 54.Robert Huben, Hoagy Cunningham, Logan Riggs Smith, Aidan Ewart, and Lee Sharkey. Sparse autoencoders find highly interpretable features in language models. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=F76bwRSLeK.
  55. 55.Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
  56. 56.Dang Huu-Tien, Trung-Tin Pham, Hoang Thanh-Tung, and Naoya Inoue. On effects of steering latent representation for large language model unlearning. arXiv preprint arXiv:2408.06223, 2024.
  57. 57.Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj.
  58. 58.Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674, 2023.
  59. 59.Yoichi Ishibashi and Hidetoshi Shimodaira. Knowledge sanitization of large language models. arXiv preprint arXiv:2309.11852, 2023.
  60. 60.Masaru Isonuma and Ivan Titov. Unlearning traces the influential training data of language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6312–6325, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.343.
  61. 61.Samyak Jain, Robert Kirk, Ekdeep Singh Lubana, Robert P Dick, Hidenori Tanaka, Edward Grefenstette, Tim Rocktäschel, and David Scott Krueger. Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks. arXiv preprint arXiv:2311.12786, 2023.
  62. 62.Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. Knowledge unlearning for mitigating privacy risks in language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14389–14408, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.805. URL https://aclanthology.org/2023.acl-long.805.
  63. 63.Hyejun Jeong, Shiqing Ma, and Amir Houmansadr. Sok: Challenges and opportunities in federated unlearning. arXiv preprint arXiv:2403.02437, 2024.
  64. 64.Jiaming Ji, Kaile Wang, Tianyi Qiu, Boyuan Chen, Jiayi Zhou, Changye Li, Hantao Lou, and Yaodong Yang. Language models resist alignment. arXiv preprint arXiv:2406.06144, 2024.
  65. 65.Jinghan Jia, Jiancheng Liu, Yihua Zhang, Parikshit Ram, Nathalie Baracaldo, and Sijia Liu. WAGLE: Strategic weight attribution for effective and modular unlearning in large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum?id=VzOgnDJMgh.
  66. 66.Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239, 2024b.
  67. 67.Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. Rwku: Benchmarking real-world knowledge unlearning for large language models, 2024.
  68. 68.Aly Kassem, Omar Mahmoud, and Sherif Saad. Preserving privacy through dememorization: An unlearning technique for mitigating memorization risks in language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 4360–4379, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.265. URL https://aclanthology.org/2023.emnlp-main.265.
  69. 69.Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML’17, page 1885–1894. JMLR.org, 2017.
  70. 70.Victoria Krakovna, Jonathan Uesato, Vladimir Mikulik, Matthew Rahtz, Tom Everitt, Ramana Kumar, Zac Kenton, Jan Leike, and Shane Legg. Specification gaming: the flip side of AI ingenuity. 4 2020. URL https://www.deepmind.com/blog/specification-gaming-the-flip-side-of-ai-ingenuity.
  71. 71.Dmitrii Krasheninnikov, Egor Krasheninnikov, Bruno Kacper Mlodozeniec, Tegan Maharaj, and David Krueger. Implicit meta-learning may lead language models to trust more reliable sources. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=Fzp1DRzCIN.
  72. 72.Rudolf Laine, Bilal Chughtai, Jan Betley, Kaivalya Hariharan, Jeremy Scheurer, Mikita Balesni, Marius Hobbhahn, Alexander Meinke, and Owain Evans. Me, myself, and ai: The situational awareness dataset (sad) for llms. arXiv preprint arXiv:2407.04694, 2024.
  73. 73.Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew Bo Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, Bhrugu Bharathi, Ariel Herbert-Voss, Cort B Breuer, Andy Zou, Mantas Mazeika, Zifan Wang, Palash Oswal, Weiran Lin, Adam Alfred Hunt, Justin Tienken-Harder, Kevin Y. Shih, Kemper Talley, John Guan, Ian Steneker, David Campbell, Brad Jokubaitis, Steven Basart, Stephen Fitz, Ponnurangam Kumaraguru, Kallol Krishna Karmakar, Uday Tupakula, Vijay Varadharajan, Yan Shoshitaishvili, Jimmy Ba, Kevin M. Esvelt, Alexandr Wang, and Dan Hendrycks. The WMDP benchmark: Measuring and reducing malicious use with unlearning. In Forty-first International Conference on Machine Learning, 2024a. URL https://openreview.net/forum?id=xlr6AUDuJz.
  74. 74.Wenjie Li, Jiawei Li, Christian Schroeder de Witt, Ameya Prabhu, and Amartya Sanyal. Delta-influence: Unlearning poisons via influence functions. arXiv preprint arXiv:2411.13731, 2024b.
  75. 75.Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation, 2023.
  76. 76.Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950–1965, 2022a.
  77. 77.Hong Liu, Zhiyuan Li, David Hall, Percy Liang, and Tengyu Ma. Sophia: A scalable stochastic second-order optimizer for language model pre-training. arXiv preprint arXiv:2305.14342, 2023.
  78. 78.Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787, 2024a.
  79. 79.Yang Liu, Mingyuan Fan, Cen Chen, Ximeng Liu, Zhuo Ma, Li Wang, and Jianfeng Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE conference on computer communications, pages 280–289. IEEE, 2022b.
  80. 80.Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Complex backdoor detection by symmetric feature differencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15003–15013, 2022c.
  81. 81.Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. Towards safer large language models through machine unlearning. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics ACL 2024, pages 1817–1829, Bangkok, Thailand and virtual meeting, August 2024b. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-acl.107.
  82. 82.Ziyao Liu, Huanyi Ye, Chen Chen, and Kwok-Yan Lam. Threats, attacks, and defenses in machine unlearning: A survey. arXiv preprint arXiv:2403.13682, 2024c.
  83. 83.Michelle Lo, Shay Cohen, and Fazl Barez. Large language models relearn removed concepts. In Findings of the Association for Computational Linguistics ACL 2024, pages 8306–8323, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.findings-acl.492.
  84. 84.Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. A pretrainer’s guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint arXiv:2305.13169, 2023.
  85. 85.Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409.18025, 2024.
  86. 86.Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835, 2024.
  87. 87.Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C. Lipton, and J. Zico Kolter. Tofu: A task of fictitious unlearning for llms, 2024.
  88. 88.Alessandro Mantelero. The eu proposal for a general data protection regulation and the roots of the ‘right to be forgotten’. Computer Law & Security Review, 29(3):229–235, 2013.
  89. 89.Justus Mattern, Fatemehsadat Mireshghallah, Zhijing Jin, Bernhard Schoelkopf, Mrinmaya Sachan, and Taylor Berg-Kirkpatrick. Membership inference attacks against language models via neighbourhood comparison. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 11330–11343, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.719. URL https://aclanthology.org/2023.findings-acl.719.
  90. 90.Alexander Meinke, Bronson Schoen, Jérémy Scheurer, Mikita Balesni, Rusheb Shah, and Marius Hobbhahn. Frontier models are capable of in-context scheming. arXiv preprint arXiv:2412.04984, 2024.
  91. 91.Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual associations in GPT. Advances in Neural Information Processing Systems, 35, 2022.
  92. 92.Kevin Meng, Arnab Sen Sharma, Alex Andonian, Yonatan Belinkov, and David Bau. Mass editing memory in a transformer. The Eleventh International Conference on Learning Representations (ICLR), 2023.
  93. 93.Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference-free reward. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
  94. 94.Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843, 2016.
  95. 95.Yuchun Miao, Sen Zhang, Liang Ding, Rong Bao, Lefei Zhang, and Dacheng Tao. InfoRM: Mitigating reward hacking in RLHF via information-theoretic reward modeling. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https://openreview.net/forum?id=3XnBVK9sD6.
  96. 96.Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15817–15831. PMLR, 17–23 Jul 2022. URL https://proceedings.mlr.press/v162/mitchell22a.html.
  97. 97.John Xavier Morris, Wenting Zhao, Justin T Chiu, Vitaly Shmatikov, and Alexander M Rush. Language model inversion. In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/forum?id=t9dWHpGkPj.
  98. 98.Andrei Muresanu, Anvith Thudi, Michael R Zhang, and Nicolas Papernot. Unlearnable algorithms for in-context learning. arXiv preprint arXiv:2402.00751, 2024.
  99. 99.Neel Nanda, Senthooran Rajamanoharan, J’anos Kram’ar, and Rohin Shah. Fact finding: Attempting to reverse-engineer factual recall on the neuron level, Dec 2023. URL https://www.alignmentforum.org/posts/iGuwZTHWb6DFY3sKB/fact-finding-attempting-to-reverse-engineer-factual-recall.
  100. 100.Richard Ngo, Lawrence Chan, and Sören Mindermann. The alignment problem from a deep learning perspective, 2024. URL https://arxiv.org/abs/2209.00626.
  101. 101.Ngoc-Bao Nguyen, Keshigeyan Chandrasegaran, Milad Abdollahzadeh, and Ngai-Man Cheung. Rethinking model inversion attacks against deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16384–16393, 2023.
  102. 102.Shiwen Ni, Dingwei Chen, Chengming Li, Xiping Hu, Ruifeng Xu, and Min Yang. Forgetting before learning: Utilizing parametric arithmetic for knowledge updating in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5716–5731, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.310.
  103. 103.Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., 2022. URL https://proceedings.neurips.cc/paper_files/paper/2022/file/b1efde53be364a73914f58805a001731-Paper-Conference.pdf.
  104. 104.Ryan Park, Rafael Rafailov, Stefano Ermon, and Chelsea Finn. Disentangling length from quality in direct preference optimization. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Findings of the Association for Computational Linguistics: ACL 2024, pages 4998–5017, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/2024.findings-acl.297. URL https://aclanthology.org/2024.findings-acl.297.
  105. 105.Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. BBQ: A hand-built bias benchmark for question answering. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio, editors, Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.findings-acl.165. URL https://aclanthology.org/2022.findings-acl.165.
  106. 106.Vaidehi Patil, Peter Hase, and Mohit Bansal. Can sensitive information be deleted from llms? objectives for defending against extraction attacks. arXiv preprint arXiv:2309.17410, 2023.
  107. 107.Martin Pawelczyk, Seth Neel, and Himabindu Lakkaraju. In-context unlearning: Language models as few shot unlearners, 2024. URL https://openreview.net/forum?id=5LhYYajlqV.
  108. 108.Ethan Perez, Sam Ringer, Kamile Lukosiute, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Benjamin Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemi Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, and Jared Kaplan. Discovering language model behaviors with model-written evaluations. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 13387–13434, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.847. URL https://aclanthology.org/2023.findings-acl.847.
  109. 109.Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693, 2023.
  110. 110.Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946, 2024.
  111. 111.Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, Jan Batzner, Hassan Sajjad, and Frank Rudzicz. Immunization against harmful fine-tuning attacks. arXiv preprint arXiv:2402.16382, 2024a.
  112. 112.Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, carsten maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. Representation noising: A defence mechanism against harmful finetuning. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024b. URL https://openreview.net/forum?id=eP9auEJqFg.
  113. 113.Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. Towards understanding sycophancy in language models, 2023. URL https://arxiv.org/abs/2310.13548.
  114. 114.Erfan Shayegani, Md Abdullah Al Mamun, Yu Fu, Pedram Zaree, Yue Dong, and Nael Abu-Ghazaleh. Survey of vulnerabilities in large language models revealed by adversarial attacks. arXiv preprint arXiv:2310.10844, 2023.
  115. 115.Wei Shen, Rui Zheng, Wenyu Zhan, Jun Zhao, Shihan Dou, Tao Gui, Qi Zhang, and Xuanjing Huang. Loose lips sink ships: Mitigating length bias in reinforcement learning from human feedback. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2859–2873, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-emnlp.188. URL https://aclanthology.org/2023.findings-emnlp.188.
  116. 116.Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Casper Stephen. Latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2407.15549, 2024.
  117. 117.Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL https://openreview.net/forum?id=zWqr3MQuNs.
  118. 118.Weijia Shi, Jaechan Lee, Yangsibo Huang, Sadhika Malladi, Jieyu Zhao, Ari Holtzman, Daogao Liu, Luke Zettlemoyer, Noah A Smith, and Chiyuan Zhang. Muse: Machine unlearning six-way evaluation for language models. arXiv preprint arXiv:2407.06460, 2024b.
  119. 119.Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jimenez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative ai. arXiv preprint arXiv:2407.00106, 2024.
  120. 120.Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. A long way to go: Investigating length correlations in rlhf. arXiv preprint arXiv:2310.03716, 2023.
  121. 121.Joar Max Viktor Skalse, Nikolaus H. R. Howe, Dmitrii Krasheninnikov, and David Krueger. Defining and characterizing reward gaming. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https://openreview.net/forum?id=yb3HOXO3lX2.
  122. 122.Nexhi Sula, Abhinav Kumar, Jie Hou, Han Wang, and Reza Tourani. Silver linings in the shadows: Harnessing membership inference for machine unlearning. arXiv preprint arXiv:2407.00866, 2024.
  123. 123.Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In International conference on machine learning, pages 3319–3328. PMLR, 2017.
  124. 124.Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, Zou Andy, Song Dawn, Li Bo, Hendrycks Dan, and Mazeika Mantas. Tamper-resistant safeguards for open-weight llms. arXiv preprint arXiv:2408.00761, 2024.
  125. 125.Ayush K Tarun, Vikram S Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
  126. 126.Ayush K. Tarun, Vikram S. Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 35(9):13046–13055, 2024. doi: 10.1109/TNNLS.2023.3266233.
  127. 127.Pratiksha Thaker, Yash Maurya, and Virginia Smith. I’m not familiar with the name harry potter: Prompting baselines for unlearning in LLMs. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models, 2024. URL https://openreview.net/forum?id=eBcVsC4h6A.
  128. 128.UK AI Safety Institute. Advanced ai evaluations at aisi: May update, 2024. URL https://www.aisi.gov.uk/work/advanced-ai-evaluations-may-update.
  129. 129.UKAISI. UK AI Safety Institute. https://www.aisi.gov.uk. Accessed: 2024-09-01.
  130. 130.USAISI. US AI Safety Institute. https://www.nist.gov/aisi. Accessed: 2024-09-01.
  131. 131.Àlex Pujol Vidal, Anders S Johansen, Mohammad NS Jahromi, Sergio Escalera, Kamal Nasrollahi, and Thomas B Moeslund. Verifying machine unlearning with explainable ai. arXiv preprint arXiv:2411.13332, 2024.
  132. 132.Paul Voigt and Axel Von dem Bussche. The eu general data protection regulation (gdpr). A Practical Guide, 1st Ed., Cham: Springer International Publishing, 10(3152676):10–5555, 2017.
  133. 133.Lingzhi Wang, Tong Chen, Wei Yuan, Xingshan Zeng, Kam-Fai Wong, and Hongzhi Yin. KGA: A general machine unlearning framework based on knowledge gap alignment. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13264–13276, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.740. URL https://aclanthology.org/2023.acl-long.740.
  134. 134.Peng Wang, Zexi Li, Ningyu Zhang, Ziwen Xu, Yunzhi Yao, Yong Jiang, Pengjun Xie, Fei Huang, and Huajun Chen. WISE: Rethinking the knowledge memory for lifelong model editing of large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024a. URL https://openreview.net/forum?id=VJMYOfJVC2.
  135. 135.Yu Wang, Ruihan Wu, Zexue He, Xiusi Chen, and Julian McAuley. Large scale knowledge washing. arXiv preprint arXiv:2405.16720, 2024b.
  136. 136.Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Li Tianle, Ku Max, Wang Kai, Zhuang Alex, Fan Rongqi, Yue Xiang, and Chen Wenhu. Mmlu-pro: A more robust and challenging multi-task language understanding benchmark. arXiv preprint arXiv:2406.01574, 2024c.
  137. 137.Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966, 2023b.
  138. 138.Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. Jailbroken: How does LLM safety training fail? In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=jA235JGM09.
  139. 139.Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162, 2024.
  140. 140.Jiaxin Wen, Ruiqi Zhong, Akbir Khan, Ethan Perez, Jacob Steinhardt, Minlie Huang, Sam Boman, He He, and Shi Feng. Language models learn to mislead humans via rlhf. arXiv:2409.12822, 2024a.
  141. 141.Yuxin Wen, Yuchen Liu, Chen Chen, and Lingjuan Lyu. Detecting, explaining, and mitigating memorization in diffusion models. In The Twelfth International Conference on Learning Representations, 2024b. URL https://openreview.net/forum?id=84n3UwkH7b.
  142. 142.Marcus Williams, Micah Carroll, Adhyyan Narang, Constantin Weisser, Brendan Murphy, and Anca Dragan. On targeted manipulation and deception when optimizing llms for user feedback, 2024. URL https://arxiv.org/abs/2411.02306.
  143. 143.Xinwei Wu, Junzhuo Li, Minghui Xu, Weilong Dong, Shuangzhi Wu, Chao Bian, and Deyi Xiong. DEPN: Detecting and editing privacy neurons in pretrained language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2875–2886, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.emnlp-main.174. URL https://aclanthology.org/2023.emnlp-main.174.
  144. 144.Sophie Xhonneux, David Dobre, Jian Tang, Gauthier Gidel, and Dhanya Sridhar. In-context learning can re-learn forbidden tasks. arXiv preprint arXiv:2402.05723, 2024.
  145. 145.Chunpu Xu, Steffi Chern, Ethan Chern, Ge Zhang, Zekun Wang, Ruibo Liu, Jing Li, Jie Fu, and Pengfei Liu. Align on the fly: Adapting chatbot behavior to established norms. arXiv preprint arXiv:2312.15907, 2023.
  146. 146.Heng Xu, Tianqing Zhu, Lefeng Zhang, and Wanlei Zhou. Really unlearned? verifying machine unlearning via influential sample pairs. arXiv preprint arXiv:2406.10953, 2024a.
  147. 147.Weiwei Xu, Kai Gao, Hao He, and Minghui Zhou. A first look at license compliance capability of llms in code generation. arXiv e-prints, pages arXiv–2408, 2024b.
  148. 148.Jin Yao, Eli Chien, Minxin Du, Xinyao Niu, Tianhao Wang, Zezhou Cheng, and Xiang Yue. Machine unlearning of pre-trained large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8403–8419, Bangkok, Thailand, August 2024a. Association for Computational Linguistics. URL https://aclanthology.org/2024.acl-long.457.
  149. 149.Yifan Yao, Jinhao Duan, Kaidi Xu, Yuanfang Cai, Zhibo Sun, and Yue Zhang. A survey on large language model (llm) security and privacy: The good, the bad, and the ugly. High-Confidence Computing, page 100211, 2024b.
  150. 150.Yuanshun Yao, Xiaojun Xu, and Yang Liu. Large language model unlearning. In Socially Responsible Language Modelling Research, 2023. URL https://openreview.net/forum?id=wKe6jE065x.
  151. 151.Charles Yu, Sullam Jeoung, Anish Kasi, Pengfei Yu, and Heng Ji. Unlearning bias in language models by partitioning gradients. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational Linguistics: ACL 2023, pages 6032–6048, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.findings-acl.375. URL https://aclanthology.org/2023.findings-acl.375.
  152. 152.Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, Yekhanin Sergey, and Zhang Huishuai. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500, 2021.
  153. 153.Hongbang Yuan, Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, and Jun Zhao. Towards robust knowledge unlearning: An adversarial framework for assessing and improving unlearning robustness in large language models. arXiv preprint arXiv:2408.10682, 2024.
  154. 154.Yi Zeng, Weiyu Sun, Tran Ngoc Huynh, Dawn Song, Bo Li, and Ruoxi Jia. Beear: Embedding-based adversarial removal of safety backdoors in instruction-tuned language models. arXiv preprint arXiv:2406.17092, 2024.
  155. 155.Binchi Zhang, Zihan Chen, Cong Shen, and Jundong Li. Verification of machine unlearning is fragile. In Forty-first International Conference on Machine Learning, 2024. URL https://openreview.net/forum?id=OkChMnjF6s.
  156. 156.Jinghan Zhang, Shiqi Chen, Junteng Liu, and Junxian He. Composing parameter-efficient modules with arithmetic operations. In Advances in Neural Information Processing Systems, 2023a.
  157. 157.Yue Zhang, Leyang Cui, Wei Bi, and Shuming Shi. Alleviating hallucinations of large language models through induced hallucinations. ArXiv, abs/2312.15710, 2023b. URL https://api.semanticscholar.org/CorpusID:266551298.
  158. 158.Chenxu Zhao, Wei Qian, Yangyi Li, Aobo Chen, and Mengdi Huai. Rethinking adversarial robustness in the context of the right to be forgotten. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp, editors, Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 60927–60939. PMLR, 21–27 Jul 2024. URL https://proceedings.mlr.press/v235/zhao24k.html.
  159. 159.Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https://openreview.net/forum?id=uccHPGDlao.
  160. 160.Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Rose doesn’t do that: Boosting the safety of instruction-tuned large language models with reverse prompt contrastive decoding. In Annual Meeting of the Association for Computational Linguistics, 2024. URL https://api.semanticscholar.org/CorpusID:267751264.
  161. 161.Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-following evaluation for large language models. arXiv preprint arXiv:2311.07911, 2023.
  162. 162.Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.

Citation

MLA
Barez, F., et al. “Open Problems in Machine Unlearning for AI Safety”. arXiv, 2025, http://arxiv.org/abs/2501.04952v1.
APA
Barez, F., Fu, T., Prabhu, A., Casper, S., Sanyal, A., Bibi, A., O'Gara, A., Kirk, R., Bucknall, B., Fist, T., Ong, L., Torr, P., Lam, K.-Y., Trager, R., Krueger, D., Mindermann, S., Hernandez-Orallo, J., Geva, M., & Gal, Y. (2025). Open Problems in Machine Unlearning for AI Safety. arXiv. http://arxiv.org/abs/2501.04952v1
Chicago
Barez, F., T. Fu, A. Prabhu, et al. 2025. “Open Problems in Machine Unlearning for AI Safety”. arXiv. http://arxiv.org/abs/2501.04952v1.
Harvard
Barez, F. et al. (2025) “Open Problems in Machine Unlearning for AI Safety”, arXiv [Preprint]. Available at: http://arxiv.org/abs/2501.04952v1.
Vancouver
1. Barez F, Fu T, Prabhu A, et al (2025) Open Problems in Machine Unlearning for AI Safety. arXiv

BibTeX

@article{barez2025open,
  title = {Open Problems in Machine Unlearning for AI Safety},
  author = {Barez, Fazl and Fu, Tingchen and Prabhu, Ameya and Casper, Stephen and Sanyal, Amartya and Bibi, Adel and O'Gara, Aidan and Kirk, Robert and Bucknall, Ben and Fist, Tim and Ong, Luke and Torr, Philip and Lam, Kwok-Yan and Trager, Robert and Krueger, David and Mindermann, Sören and Hernandez-Orallo, José and Geva, Mor and Gal, Yarin},
  year = {2025},
  journal = {arXiv},
  url = {http://arxiv.org/abs/2501.04952v1},
  eprint = {2501.04952}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: https://creativecommons.org/licenses/by/4.0/