Elamur Research⌕
☰
home / papers / 2026 / From Narrow Unlearning to Emergent Misalignment in LLMs

From Narrow Unlearning to Emergent Misalignment in LLMs

Paper · 2026 · Erum Mushtaq, Anil Ramakrishna, Satyapriya Krishna et al.

Summary

Earlier work found that fine-tuning an aligned LLM on a narrow task, such as writing insecure code, can cause emergent misalignment (EMA): harmful behaviour that spreads to unrelated domains. This paper asks whether the same happens with narrow refusal unlearning. That approach needs only the model's own refusals, not harmful labeled completions. The authors collect refusal responses from Mistral-7B-0.3v and Qwen-7B-2.5 on Cybersecurity and Safety prompts. They then apply negative preference optimization (NPO) to make the models comply on those concepts, and compare this with a fine-tuning baseline. Refusal rates are tracked across seven responsible AI domains, alongside MMLU and XSTest over-deflection.

Unlearning cuts refusals on the targeted concept sharply. It also lowers refusals elsewhere, and the Safety intervention spreads more widely than the Cybersecurity one. Adding a cross-entropy loss on a small retain set from the affected domains recovers much of the lost alignment, though not all of it. A representation analysis links EMA to concept overlap in early layers. The practical lesson is that any targeted relaxation of refusals needs safety evaluation across many domains.

Key points

  • Data:
    • Cybersecurity data comes from CyberSecEval (MITRE and Interpreter Abuse), with 1,500 prompts split 1,400/100 for train/test. This yields 620 refusal prompts for Mistral and 700 for Qwen.
    • The Safety set has 637 prompts drawn from StrongReject, SorryBench, MSTS and SGBench. A GPT-3.5-turbo judge sorted the prompts into the seven concepts.
  • Training setup:
    • Training uses LoRA with grid-searched hyperparameters. Qwen Safety runs required full-rank training, because LoRA produced code-style refusals instead of compliance.
    • Checkpoints were kept only if the MMLU score dropped by less than 3%.
  • Effect on the targeted concept:
    • Cybersecurity refusals fell from 67.00 to 6.00 on Mistral and from 93.00 to 10.00 on Qwen. Here unlearning slightly beat fine-tuning.
    • Safety refusals fell from 94.00 to 36.55 on Mistral and from 98.00 to 18.27 on Qwen. Here fine-tuning did somewhat better.
  • Mitigation: Retain-augmented training using forget:retain ratios of 1:1, 1:2 and 1:3 substantially restored refusals in the affected domains. A gap relative to the aligned models remained.
  • Representation analysis: Concept vectors were built with PCA over last-token hidden states at layer 5, using N=200 prompts per concept. The Cybersecurity vector overlaps mostly with Safety, while the Safety vector overlaps broadly with all other concepts. This pattern mirrors where EMA appeared.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Southern CaliforniaLos Angeles · United States162 works · 35,836 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Erum Mushtaq, Anil Ramakrishna, Satyapriya Krishna, Sattvik Sahai, Prasoon Goyal, Kai-Wei Chang, Tao Zhang and Rahul Gupta. 2026. From Narrow Unlearning to Emergent Misalignment in LLMs. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-short.32

@inproceedings{mushtaq2026narrow,
  title = {{From Narrow Unlearning to Emergent Misalignment in LLMs}},
  author = {Erum Mushtaq, Anil Ramakrishna, Satyapriya Krishna, Sattvik Sahai, Prasoon Goyal, Kai-Wei Chang, Tao Zhang and Rahul Gupta},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-short.32},
  url = {https://doi.org/10.18653/v1/2026.acl-short.32}
}

References

33 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loubna Ben Allal, Anton Lozhkov, Elie Bakouch, Leandro von Werra, and Thomas Wolf. 2024. Smollm-blazingly fast and remarkably powerful. Hugging Face Blog, 16. find
  2. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. Advances in Neural Information Processing Systems, 37:136037–136083.
  3. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  4. Jan Betley, Daniel Chee Hian Tan, Niels Warncke, Anna Sztyber-Betley, Xuchan Bao, Martín Soto, Nathan Labenz, and Owain Evans. 2025. Emergent misalignment: Narrow finetuning can produce broadly misaligned LLMs. In Forty-second International Conference on Machine Learning. find
  5. Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, and 1 others. 2024. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models. arXiv preprint arXiv:2404.13161. — [arXiv:2404.13161](https://arxiv.org/abs/2404.13161) find
  6. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. Toxigen: A large-scale machine-generated dataset for implicit and adversarial hate speech detection. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics. find
  7. Zihao He, Siyi Guo, Ashwin Rao, and Kristina Lerman. 2024. Whose emotions and moral sentiments do language models reflect? In Findings of the Association for Computational Linguistics: ACL 2024, pages 6611–6631, Bangkok, Thailand. Association for Computational Linguistics. find
  8. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  9. Zachary Kenton, Tom Everitt, Laura Weidinger, Iason Gabriel, Vladimir Mikulik, and Geoffrey Irving. 2021. Alignment of language agents. CoRR, abs/2103.14659. find
  10. Tharindu Kumarage, Ninareh Mehrabi, Anil Ramakrishna, Xinyan Zhao, Richard Zemel, Kai-Wei Chang, Aram Galstyan, Rahul Gupta, and Charith Peris. 2025. Towards safety reasoning in llms: Ai-agentic deliberation for policy-embedded cot data creation. arXiv preprint arXiv:2505.21784. find
  11. Bruce W. Lee, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Erik Miehling, Pierre Dognin, Manish Nagireddy, and Amit Dhurandhar. 2025a. Programming refusal with conditional activation steering. In The Thirteenth International Conference on Learning Representations. find
  12. Daniel Lee, Eric Breck, and Andy Arditi. 2025b. Finding features causally upstream of refusal. Accessed: 2025-08-20. find
  13. Michael Matthews, Samuel Matthews, and Thomas Kelemen. 2022. The alignment problem: Machine learning and human values. Personnel Psychology, 75(1). find
  14. Yutao Mou, Shikun Zhang, and Wei Ye. 2024. Sgbench: Evaluating llm safety generalization across diverse tasks and prompt types. Advances in Neural Information Processing Systems, 37:123032–123054. find
  15. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso…
  16. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety alignment should be made more than just a few tokens deep. In The Thirteenth International Conference on Learning Representations. find
  17. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  18. Paul Röttger, Giuseppe Attanasio, Felix Friedrich, Janis Goldzycher, Alicia Parrish, Rishabh Bhardwaj, Chiara Di Bonaventura, Roman Eng, Gaia El Khoury Geagea, Sujata Goswami, and 1 others. 2025. Msts: A multimodal safety test suite for vision-language models. arXiv preprint arXiv:2501.10057. — [arXiv:2501.10057](https://arxiv.org/abs/2501.10057) find
  19. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
  20. Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang, Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, and 1 others. 2025. Spurious rewards: Rethinking training signals in rlvr. arXiv preprint arXiv:2506.10947. find
  21. Vincent Siu, Nicholas Crispino, David Park, Nathan W Henry, Zhun Wang, Yang Liu, Dawn Song, and Chenguang Wang. 2025. Steeringcontrol: Holistic evaluation of alignment steering in llms. arXiv preprint arXiv:2509.13450. — [arXiv:2509.13450](https://arxiv.org/abs/2509.13450) find
  22. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and 1 others. 2024. A strongreject for empty jailbreaks. Advances in Neural Information Processing Systems, 37:125416–125440. find
  23. Daniel Tan, David Chanin, Aengus Lynch, Brooks Paige, Dimitrios Kanoulas, Adrià Garriga-Alonso, and Robert Kirk. 2024. Analysing the generalisation and reliability of steering vectors. Advances in Neural Information Processing Systems, 37:139179– 139212. find
  24. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu-lab/stanford_alpaca. ↗
  25. Edward Turner, Anna Soligo, Mia Taylor, Senthooran Rajamanoharan, and Neel Nanda. 2025. Model organisms for emergent misalignment. In ICML 2025 Workshop on Reliable and Responsible Foundation Models. find
  26. Miles Wang, Tom Dupré la Tour, Olivia Watkins, Alex Makelov, Ryan A Chi, Samuel Miserendino, Johannes Heidecke, Tejal Patwardhan, and Dan Mossing. 2025. Persona features control emergent misalignment. arXiv preprint arXiv:2506.19823. — [arXiv:2506.19823](https://arxiv.org/abs/2506.19823) find
  27. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024. Do-not-answer: Evaluating safeguards in LLMs. In Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911, St. Julian’s, Malta. Association for Computational Linguistics.
  28. Zhengxuan Wu, Aryaman Arora, Atticus Geiger, Zheng Wang, Jing Huang, Dan Jurafsky, Christopher D Manning, and Christopher Potts. 2025. Axbench: Steering LLMs? even simple baselines outperform sparse autoencoders. In Forty-second International Conference on Machine Learning. find
  29. Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Keloth, and 1 others. 2024. Me-llama: Foundation large language models for medical applications. Research square, pages rs–3.
  30. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. 2025. SORRY-bench: Systematically evaluating large language model safety refusal. In The Thi… find
  31. Yuan Yuan, Tina Sriskandarajah, Anna-Luisa Brakman, Alec Helyar, Alex Beutel, Andrea Vallone, and Saachi Jain. 2025. From hard refusals to safe-completions: Toward output-centric safety training. arXiv preprint arXiv:2508.09224. — [arXiv:2508.09224](https://arxiv.org/abs/2508.09224) find
  32. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024. Negative preference optimization: From catastrophic collapse to effective unlearning. In First Conference on Language Modeling.
  33. Andy Zou, Long Phan, Sarah Chen, James Campbell, Phillip Guo, Richard Ren, Alexander Pan, Xuwang Yin, Mantas Mazeika, Ann-Kathrin Dombrowski, Shashwat Goel, Nathaniel Li, Michael J. Byun, Zifan Wang, Alex Mallen, Steven Basart, Sanmi Koyejo, Dawn Song, Matt Fredrikson, and 2 others. 2023. Representa…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mushtaq-2026-narrow-unlearning-emergent-misalignment-llms/