Elamur Research
☰
home / papers / 2023 / Large Language Model Unlearning

Large Language Model Unlearning

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2310.10683

Summary

The paper treats unlearning as a cheap alignment tool for LLMs. The idea is to make a model drop unwanted behaviour using only negative examples, such as harmful replies found through red teaming or user reports, without the human-written positive answers that RLHF needs. The authors define the setting, goals and evaluation criteria. Given a forget set of bad prompt–output pairs, the unlearned model should move its outputs away from those responses while keeping its usefulness on ordinary prompts. The method builds on gradient ascent (GA) over the forget set. A variant, GA+Mismatch, also trains toward random outputs, and both use weighted loss terms. They test three scenarios: removing harmful responses, erasing copyrighted material and forgetting hallucinated facts. In an ablation on OPT-1.3B with PKU-SafeRLHF, unlearning reaches harmful rates comparable to or below full RLHF while using about 2% of its compute. This makes it attractive for teams with tight budgets whose main concern is preventing damaging outputs.

Key points

  • Advantages claimed: only negative samples are required, collection of those samples can be largely automated, and the compute cost is close to light finetuning. The method also fits well when the offending training data is already known.
  • Harmful rate on OPT-1.3B / PKU-SafeRLHF (seen prompts):
    • original model: 47%
    • SFT: 34%
    • full RLHF: 4%
    • GA: 1%
    • GA+Mismatch: 6%
  • Harmful rate on unseen prompts:
    • original model: 53%
    • full RLHF: 7.5%
    • GA: 3%
    • GA+Mismatch: 7%
  • Trade-offs between the GA variants: plain GA tends to produce degenerate text (diversity 0.118, fluency rated not meaningful). GA+Mismatch stays fluent and scores the best utility reward on normal prompts (−2.982, compared with −3.212 for RLHF).
  • Templated outputs and code: replacing random targets with a fixed refusal ("I can't assist it.") gives similar results, with a 1% harmful rate on seen prompts. The code is available at github.com/kevinyaobytedance/llm_unlearn.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

120 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 118 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.10683 · arXiv 2310.10683

@misc{yao2023large,
  title = {{Large Language Model Unlearning}},
  author = {Yao, Yuanshun and Xu, Xiaojun and Liu, Yang},
  year = {2023},
  doi = {10.48550/arxiv.2310.10683},
  eprint = {2310.10683},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.10683}
}

References

46 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Y. Bai, S. Kadavath, S. Kundu, A. Askell, J. Kernion, A. Jones, A. Chen, A. Goldie, A. Mirhoseini, C. McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073, 2022. find our card
  2. L. Bourtoule, V. Chandrasekaran, C. A. Choquette-Choo, H. Jia, A. Travers, B. Zhang, D. Lie, and N. Papernot. Machine unlearning. In 2021 IEEE Symposium on Security and Privacy (SP), pages 141–159. find
  3. Y. Cao and J. Yang. Towards making systems forget with machine unlearning. In 2015 IEEE Symposium on Security and Privacy, pages 463–480. find
  4. N. Carlini, F. Tramer, E. Wallace, M. Jagielski, A. Herbert-Voss, K. Lee, A. Roberts, T. B. Brown, D. Song, U. Erlingsson, et al. Extracting training data from large language models. In USENIX Security Symposium, volume 6, 2021. find our card
  5. N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramer, and C. Zhang. Quantifying memorization across neural language models. arXiv preprint arXiv:2202.07646, 2022. find
  6. P. F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, and D. Amodei. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30, 2017. find
  7. V. S. Chundawat, A. K. Tarun, M. Mandal, and M. Kankanhalli. Zero-shot machine unlearning. IEEE Transactions on Information Forensics and Security, 2023. find
  8. G. Copilot. Github copilot lawsuit. https://www.courthousenews.com/microsoft-and-github-ask-court-to-scrap-lawsuit-over-ai-powered-copilot/, 2023. ↗
  9. R. Eldan and M. Russinovich. Who’s harry potter? approximate unlearning in llms. arXiv preprint arXiv:2310.02238, 2023. find
  10. Facebook. Facebook community standards. https://www.facebook.com/communitystandards/, 2023. ↗
  11. M. Grynbaum and R. Mac. The times sues openai and microsoft over a.i. use of copyrighted work. The New York Times, 2023. URL https://www.nytimes.com/2023/12/27/business/media/ new-york-times-open-ai-microsoft-lawsuit.html. Accessed: 2024-01-16. ↗
  12. C. Guo, T. Goldstein, A. Hannun, and L. Van Der Maaten. Certified data removal from machine learning models. arXiv preprint arXiv:1911.03030, 2019. — [arXiv:1911.03030](https://arxiv.org/abs/1911.03030) find
  13. A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi. The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751, 2019. find
  14. Z. Izzo, M. A. Smart, K. Chaudhuri, and J. Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pages 2008–2016. find
  15. J. Ji, M. Liu, J. Dai, X. Pan, C. Zhang, C. Bian, R. Sun, Y. Wang, and Y. Yang. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657, 2023. find our card
  16. J. Jia, J. Liu, P. Ram, Y. Yao, G. Liu, Y. Liu, P. Sharma, and S. Liu. Model sparsification can simplify machine unlearning. arXiv preprint arXiv:2304.04934, 2023. — [arXiv:2304.04934](https://arxiv.org/abs/2304.04934) find
  17. J. Lee, T. Le, J. Chen, and D. Lee. Do language models plagiarize? In Proceedings of the ACM Web Conference 2023, pages 3637–3647, 2023. find
  18. J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen. Halueval: A large-scale hallucination evaluation benchmark for large language models. arXiv e-prints, pages arXiv–2305, 2023. find our card
  19. S. Lin, J. Hilton, and O. Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958, 2021. find our card
  20. Y. Liu, M. Fan, C. Chen, X. Liu, Z. Ma, L. Wang, and J. Ma. Backdoor defense with machine unlearning. In IEEE INFOCOM 2022-IEEE Conference on Computer Communications, pages 280–289. find
  21. Y. Liu, Y. Yao, J.-F. Ton, X. Zhang, R. G. H. Cheng, Y. Klochkov, M. F. Taufiq, and H. Li. Trustworthy llms: a survey and guideline for evaluating large language models’ alignment. arXiv preprint arXiv:2308.05374, 2023. find our card
  22. X. Lu, S. Welleck, J. Hessel, L. Jiang, L. Qin, P. West, P. Ammanabrolu, and Y. Choi. Quark: Controllable text generation with reinforced unlearning. Advances in neural information processing systems, 35:27591–27609, 2022. find our card
  23. R. Müller, S. Kornblith, and G. E. Hinton. When does label smoothing help? Advances in neural information processing systems, 32, 2019. find
  24. K. P. Murphy. Probabilistic Machine Learning: An introduction. MIT Press, 2022. URL probml.ai. find
  25. S. Neel, A. Roth, and S. Sharifi-Malvajerdi. Descent-to-delete: Gradient-based methods for machine unlearning. In Algorithmic Learning Theory, pages 931–962. PMLR, 2021. find
  26. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022. find
  27. K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318, 2002. find our card
  28. M. Pawelczyk, S. Neel, and H. Lakkaraju. In-context unlearning: Language models as few shot unlearners. arXiv preprint arXiv:2310.07579, 2023. find
  29. J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017. find
  30. T. Sellam, D. Das, and A. P. Parikh. Bleurt: Learning robust metrics for text generation. arXiv preprint arXiv:2004.04696, 2020. find
  31. R. Shokri, M. Stronati, C. Song, and V. Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. find our card
  32. Z. Small. Sarah silverman sues openai and meta over copyright infringement. The New York Times, 2023. URL https://www.nytimes.com/2023/07/10/arts/ sarah-silverman-lawsuit-openai-meta.html. Accessed: 2024-01-16. ↗
  33. A. K. Tarun, V. S. Chundawat, M. Mandal, and M. Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023. find
  34. A. Thudi, G. Deza, V. Chandrasekaran, and N. Papernot. Unrolling sgd: Understanding factors influencing machine unlearning. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pages 303–319. find
  35. TikTok. Tiktok community guidelines. https://www.tiktok.com/community-guidelines?lang= en, 2023. ↗
  36. H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023. find
  37. Twitter. Twitter rules and policies. twitter-rules, 2023. https://help.twitter.com/en/rules-and-policies/ ↗
  38. J. P. Wahle, T. Ruas, F. Kirstein, and B. Gipp. How large language models are transforming machine-paraphrased plagiarism. arXiv preprint arXiv:2210.03568, 2022. — [arXiv:2210.03568](https://arxiv.org/abs/2210.03568) find
  39. A. Warnecke, L. Pirch, C. Wressnegger, and K. Rieck. Machine unlearning of features and labels. arXiv preprint arXiv:2108.11577, 2021. — [arXiv:2108.11577](https://arxiv.org/abs/2108.11577) find
  40. H. Xu, T. Zhu, L. Zhang, W. Zhou, and P. S. Yu. Machine unlearning: A survey. ACM Computing Surveys, 56(1):1–36, 2023. find
  41. Z. Yuan, H. Yuan, C. Tan, W. Wang, S. Huang, and F. Huang. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302, 2023. find our card
  42. S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022. find
  43. T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019. find
  44. R. Zheng, S. Dou, S. Gao, W. Shen, B. Wang, Y. Liu, S. Jin, Q. Liu, L. Xiong, L. Chen, et al. Secrets of rlhf in large language models part i: Ppo. arXiv preprint arXiv:2307.04964, 2023. find our card
  45. Y. Zhu, R. Kiros, R. Zemel, R. Salakhutdinov, R. Urtasun, A. Torralba, and S. Fidler. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In The IEEE International Conference on Computer Vision (ICCV), December 2015. find our card
  46. T. Y. Zhuo, Z. Li, Y. Huang, Y.-F. Li, W. Wang, G. Haffari, and F. Shiri. On robustness of prompt-based semantic parsing with large pre-trained language model: An empirical study on codex. arXiv preprint arXiv:2301.12868, 2023. find