Elamur Research⌕
☰
home / papers / 2025 / Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models

Paper · 2025 · Anmol Mekala, Vineeth Dorna, Shreya Dubey et al.

Summary

Approximate unlearning methods for LLMs, such as gradient ascent, GradDiff and NPO, typically penalize the original answers in the forget set. They give the model no guidance on what to say instead. As a result, unlearned models often produce gibberish or off-topic text when asked about the "forgotten" entity, which hurts utility and may reveal that unlearning took place. The authors propose Alternate Preference Optimization (AltPO), which works as follows:

  1. The LLM is prompted to generate several plausible but factually different answers to each forget-set question.
  2. A DPO-style loss treats these alternatives as preferred responses and the true answer as the dispreferred one.
  3. A standard retain-set NLL term is added.

The authors also argue that the existing TOFU metrics miss these failure modes and propose new metrics for the quality of responses on forget-set prompts. On TOFU with Llama2, AltPO achieves strong Forget Quality while keeping Model Utility high and producing coherent, if untrue, answers. This makes it a practical recipe for removing factual knowledge without breaking the model.

Key points

  • Benchmark: TOFU contains 200 fictitious authors, and the forget splits cover 1%, 5% or 10% of them. Forget Quality is the p-value of a Kolmogorov–Smirnov test on the Truth Ratio, and values above 0.05 count as successful forgetting.
  • Main result: On the 10% split with Llama2 and M = 5 alternate answers (the default), AltPO reaches an FQ of about 0.74. The fine-tuned model before unlearning scores about 2.2e-20.
  • Ablations:
    • Variants that use only positive feedback (AltPO-pos, AltNLL-pos) fail to forget, with FQ near 1e-12 to 1e-15. Negative feedback is therefore required.
    • The DPO-style loss beats an NLL-based contrast (AltNLL).
    • Prompt-specific alternatives work better and train more stably than generic "I don't know" targets (IdkPO).
  • Number of alternatives: Raising M to 10 lowers FQ to 0.25, which is still above the 0.05 significance threshold. It also reduces model self-confidence on forget prompts.
  • Alternate-answer source: Alternatives generated by a base Llama2-7b that never saw TOFU work about as well (FQ about 0.71). Code is available at github.com/molereddy/Alternate-Preference-Optimization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢MicrosoftRedmond · United States325 works · 142,931 citations🎓University of Massachusetts AmherstAmherst Center · United States106 works · 66,692 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan and Elita Lobo. 2025. Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mekala2025alternate,
  title = {{Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models}},
  author = {Anmol Mekala, Vineeth Dorna, Shreya Dubey, Abhishek Lalwani, David Koleczek, Mukund Rungta, Sadid Hasan and Elita Lobo},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.252/}
}

References

35 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Karuna Bhaila, Minh-Hao Van, and Xintao Wu. 2024. Soft prompting for unlearning in large language models. arXiv preprint arXiv:2406.12038.
  2. Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. IEEE. find
  3. Jiaao Chen and Diyi Yang. 2023. Unlearn what you want to forget: Efficient unlearning for llms. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  4. Min Chen, Zhikun Zhang, Tianhao Wang, Michael Backes, Mathias Humbert, and Yang Zhang. 2021. When machine unlearning jeopardizes privacy. In Proceedings of the 2021 ACM SIGSAC conference on computer and communications security, pages 896–911. find
  5. Our study focuses on enabling LLMs to forget specific knowledge and does not address broader Cheng-Han Chiang and Hung-yi Lee. 2023. Can large questions about the ideal behavior of an unlearned language models be an alternative to human evalua-model. For instance, should the model respond tions? In… — [arXiv:2402.10052](https://arxiv.org/abs/2402.10052) find
  6. Ronen Eldan and Mark Russinovich. 2023. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238. find
  7. Rohit Gandikota, Sheridan Feucht, Samuel Marks, and David Bau. 2024. Erasing conceptual knowledge from language models. arXiv preprint arXiv:2410.02760. — [arXiv:2410.02760](https://arxiv.org/abs/2410.02760) find
  8. Chongyang Gao, Lixu Wang, Chenkai Weng, Xiao Wang, and Qi Zhu. 2024. Practical unlearning for large language models. arXiv preprint arXiv:2407.10223. — [arXiv:2407.10223](https://arxiv.org/abs/2407.10223) find
  9. Laura Graves, Vineel Nagisetty, and Vijay Ganesh. 2021. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 13, pages 11516–11524. find
  10. Meghdad Kurmanji, Peter Triantafillou, Jamie Hayes, and Eleni Triantafillou. 2024. Towards unbounded machine unlearning. Advances in neural information processing systems, 36. find
  11. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D Li, Ann-Kathrin Dombrowski, Shashwat Goel, Long Phan, et al. 2024. The wmdp benchmark: Measuring and reducing malicious use with unlearning. arXiv preprint arXiv:2403.03218. find
  12. Chris Yuhao Liu, Yaxuan Wang, Jeffrey Flanigan, and Yang Liu. 2024a. Large language model unlearning via embedding-corrupted prompts. arXiv preprint arXiv:2406.07933. — [arXiv:2406.07933](https://arxiv.org/abs/2406.07933) find
  13. Ken Ziyu Liu. 2024. Machine unlearning in 2024. find
  14. James Y. Huang, Wenxuan Zhou, Fei Wang, Fred Morstatter, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2024. Offset unlearning for large language models. Preprint, arXiv:2404.11045. find
  15. Sijia Liu, Yuanshun Yao, Jinghan Jia, Stephen Casper, Nathalie Baracaldo, Peter Hase, Xiaojun Xu, Yuguang Yao, Hang Li, Kush R Varshney, et al. 2024b. Rethinking machine unlearning for large language models. arXiv preprint arXiv:2402.08787. find
  16. Jiabao Ji, Yujian Liu, Yang Zhang, Gaowen Liu, Ramana Rao Kompella, Sijia Liu, and Shiyu Chang. 2024. Reversing the forget-retain objectives: An efficient llm unlearning framework from logit difference. arXiv preprint arXiv:2406.08607. — [arXiv:2406.08607](https://arxiv.org/abs/2406.08607) find
  17. Zheyuan Liu, Guangyao Dou, Zhaoxuan Tan, Yijun Tian, and Meng Jiang. 2024c. Towards safer large language models through machine unlearning. arXiv preprint arXiv:2402.10058.
  18. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38.
  19. Jinghan Jia, Yihua Zhang, Yimeng Zhang, Jiancheng Liu, Bharat Runwal, James Diffenderfer, Bhavya Kailkhura, and Sijia Liu. 2024. Soul: Unlocking the power of second-order optimization for llm unlearning. arXiv preprint arXiv:2404.18239.
  20. Zhuoran Jin, Pengfei Cao, Chenhao Wang, Zhitao He, Hongbang Yuan, Jiachun Li, Yubo Chen, Kang Liu, and Jun Zhao. 2024. Rwku: Benchmarking real-world knowledge unlearning for large language models. arXiv preprint arXiv:2406.10890. — [arXiv:2406.10890](https://arxiv.org/abs/2406.10890) find
  21. Yoonhwa Jung, Ikhyun Cho, Shun-Hsiang Hsu, and Julia Hockenmaier. 2024. Attack and reset for unlearning: Exploiting adversarial noise toward machine unlearning through parameter re-initialization. arXiv preprint arXiv:2401.08998. — [arXiv:2401.08998](https://arxiv.org/abs/2401.08998) find
  22. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, et al. 2022. Language models (mostly) know what they know. arXiv preprint arXiv:2207.05221. find
  23. Antonia Karamolegkou, Jiaang Li, Li Zhou, and Anders Søgaard. 2023. Copyright violations and large language models. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  24. Zhuo Ma, Yang Liu, Ximeng Liu, Jian Liu, Jianfeng Ma, and Kui Ren. 2022. Learn to forget: Machine unlearning via neuron masking. IEEE Transactions on Dependable and Secure Computing, 20(4):3194– 3207. find
  25. Pratyush Maini, Zhili Feng, Avi Schwarzschild, Zachary C Lipton, and J Zico Kolter. 2024. Tofu: A task of fictitious unlearning for llms. First Conference On Language Modeling.
  26. Thanh Tam Nguyen, Thanh Trung Huynh, Phi Le Nguyen, Alan Wee-Chung Liew, Hongzhi Yin, and Quoc Viet Hung Nguyen. 2022. A survey of machine unlearning. arXiv preprint arXiv:2209.02299. — [arXiv:2209.02299](https://arxiv.org/abs/2209.02299) find
  27. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36.
  28. Yan Scholten, Stephan Günnemann, and Leo Schwinn. 2024. A probabilistic perspective on unlearning and alignment for large language models. arXiv preprint arXiv:2410.03523. — [arXiv:2410.03523](https://arxiv.org/abs/2410.03523) find
  29. Weijia Shi, Anirudh Ajith, Mengzhou Xia, Yangsibo Huang, Daogao Liu, Terra Blevins, Danqi Chen, and Luke Zettlemoyer. 2024. Detecting pretraining data from large language models. In The Twelfth International Conference on Learning Representations. find
  30. Pratiksha Thaker, Yash Maurya, and Virginia Smith. 2024. Guardrail baselines for unlearning in llms. arXiv preprint arXiv:2403.03329. — [arXiv:2403.03329](https://arxiv.org/abs/2403.03329) find
  31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  32. Eleni Triantafillou, Peter Kairouz, Fabian Pedregosa, Jamie Hayes, Meghdad Kurmanji, Kairan Zhao, Vincent Dumoulin, Julio Jacques Junior, Ioannis Mitliagkas, Jun Wan, et al. 2024. Are we making progress in unlearning? findings from the first neurips unlearning competition. arXiv preprint arXiv:2406.… — [arXiv:2406.09073](https://arxiv.org/abs/2406.09073) find
  33. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2023. Large language model unlearning. In Socially Responsible Language Modelling Research.
  34. Dawen Zhang, Pamela Finckenberg-Broman, Thong Hoang, Shidong Pan, Zhenchang Xing, Mark Staples, and Xiwei Xu. 2023. Right to be forgotten in the era of large language models: Implications, challenges, and solutions. arXiv preprint arXiv:2307.03941. — [arXiv:2307.03941](https://arxiv.org/abs/2307.03941) find
  35. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. First Conference on Language Modelling.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mekala-2025-alternate-preference-optimization-unlearning-factual/