Elamur Research⌕
☰
home / papers / 2026 / The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training

The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training

Paper · 2026 · Rui Zhang, Hongwei Li, Yun Shen et al.

Summary

Safety alignment tools cut both ways. The fine-tuning that teaches an LLM to refuse harmful requests can also be used to remove those refusals, and the compromised models can then be shared on open hubs. The authors frame this as an attacker–defender game. They ask which fine-tuning methods most effectively misalign a safety-tuned model, and how well a defender can restore safety afterward through "realignment".

They build MisQA, which pairs harmful questions with both unsafe and refusal answers. For realignment they use balanced subsets of hh-rlhf and safe-rlhf. Six methods are compared on Llama3.1, GLM4, Gemma2 and Mistral: four SFT adapters (LoRA, QLoRA, AdaLoRA, IA3) and two preference methods (DPO, ORPO). Unsafety is scored by a majority vote of three LLM judges, and utility is measured on four benchmarks.

The central result is an asymmetry. ORPO is the strongest attack tool, while DPO recovers safety best at a modest cost to utility. The study gives practitioners empirical guidance for vetting and realigning untrusted third-party models.

Key points

  • MisQA dataset: 390 unsafe questions across 13 categories, taken from Shen et al. 2024. Unsafe answers were generated from ChatGPT via jailbreak prompts and checked manually. Each question is stored with a safe refusal, giving (question, unsafe answer, safe answer) triplets.
  • Models and evaluation: Llama-3.1-8B-Instruct, GLM-4-9B-Chat, Gemma-2-9B-it and Mistral-7B-Instruct-v0.3. Utility is measured with MMLU, GSM8K, BoolQ and PIQA.
  • Resistance and sample efficiency: Gemma2 resisted misalignment most strongly, and ORPO was the only method that worked on it. LoRA needed the fewest harmful examples: just 13 (one per category) seriously degraded the safety of Llama3.1 and GLM4.
  • Proposed mechanism:
    • ORPO combines token-level imitation with a preference term, so it both suppresses refusals and actively writes in harmful behavior.
    • DPO's purely pairwise, margin-based signal generalizes better to the more varied safe responses in realignment data.
  • Side effects:
    • Realigning a model that had already resisted misalignment can unexpectedly make it less safe.
    • Repeated misalign/realign rounds wear down utility and make both the attacker's and the defender's goals progressively harder to reach.
    • RLHF and proprietary models were not tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Nanyang Technological UniversitySingapore · Singapore184 works · 27,484 citations🎓University of Electronic Science and Technology of ChinaChengdu · China38 works · 2,088 citations🔬Helmholtz Center for Information SecuritySaarbrücken · Germany20 works · 1,172 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksBias, fairness & safety · 587 works

Cite

Cite as Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes and Yang Zhang. 2026. The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.164

@inproceedings{zhang2026art,
  title = {{The Art of (Mis)alignment: How Fine-Tuning Methods Effectively Misalign and Realign LLMs in Post-Training}},
  author = {Rui Zhang, Hongwei Li, Yun Shen, Xinyue Shen, Wenbo Jiang, Guowen Xu, Yang Liu, Michael Backes and Yang Zhang},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.164},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.164}
}

References

69 references, 34 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  2. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022b. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR abs/2204.05862. find
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022c. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR abs/2204.05862. find
  4. Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In AAAI Conference on Artificial Intelligence (AAAI). AAAI.
  5. Fan, and 1 others. 2024. The llama 3 herd of models. CoRR abs/2407.21783.
  6. Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, and 1 others. 2023. Open problems and fundamental limitations of reinforcement learning from human feedback. CoRR abs/2307.15217. find
  7. European Commission. 2021. Proposal for a regulation of the european parliament and of the council laying down harmonised rules on artificial intelligence (artificial intelligence act) and amending certain union legislative acts. https://eur-lex.eur opa.eu/legal-content/EN/TXT/?uri=CELEX: 52021PC020… ↗
  8. Junjie Chu, Yugeng Liu, Ziqing Yang, Xinyue Shen, Michael Backes, and Yang Zhang. 2024. Comprehensive assessment of jailbreak attacks against llms. CoRR abs/2402.05668. — [arXiv:2402.05668](https://arxiv.org/abs/2402.05668) find
  9. Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, and 1 others. 2024. Chatglm: A family of large language models from glm-130b to glm-4 all tools. CoRR abs/2406.12793.
  10. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo…
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training Verifiers to Solve Math Word Problems. CoRR abs/2110.14168.
  12. Yichen Gong, Delong Ran, Xinlei He, Tianshuo Cong, Anyu Wang, and Xiaoyun Wang. 2025. Safety misalignment against large language models. In Network and Distributed System Security Symposium (NDSS). find
  13. Yichen Gong, Delong Ran, Jinyuan Liu, Conglei Wang, Tianshuo Cong, Anyu Wang, Sisi Duan, and Xiaoyun Wang. 2023. FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts. CoRR abs/2311.05608.
  14. Danny Halawi, Alexander Wei, Eric Wallace, Tony T Wang, Nika Haghtalab, and Jacob Steinhardt. 2024. Covert malicious finetuning: Challenges in safeguarding llm adaptation. CoRR abs/2406.20053. find
  15. OpenCompass Contributors. 2023. Opencompass: A universal evaluation platform for foundation models. https://github.com/open-compass/opencompa ss. ↗
  16. Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. 2024. Parameter-efficient fine-tuning for large models: A comprehensive survey. CoRR abs/2403.14608.
  17. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023a. Safe rlhf: Safe reinforcement learning from human feedback. CoRR abs/2310.12773.
  18. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring Massive Multitask Language Understanding. In International Conference on Learning Representations (ICLR).
  19. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023b. Safe rlhf: Safe reinforcement learning from human feedback. In International Conference on Learning Representations (ICLR).
  20. Jade Hardouin Daniel Huynh. 2023. A Real-World Incident from Mithril Security. https://blog.m ithrilsecurity.io/poisongpt-how-we-hid-a -lobotomized-llm-on-hugging-face-to-sprea d-fake-news/. ↗
  21. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Finetuning of Quantized LLMs. CoRR abs/2305.14314.
  22. Yilun Du, Shuang Li, Antonio Torralba, Joshua B Tenenbaum, and Igor Mordatch. 2023. Improving factuality and reasoning in language models through multiagent debate. CoRR abs/2305.14325.
  23. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Jiwoo Hong, Noah Lee, and James Thorne. 2024. Orpo: Monolithic preference optimization without reference model. In Conference on Empirical Methods in Natural… find
  24. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations (ICLR).
  25. Qiang Hu, Xiaofei Xie, Sen Chen, and Lei Ma. 2024. Large language model supply chain: Open problems from the security perspective. CoRR abs/2411.01604. — [arXiv:2411.01604](https://arxiv.org/abs/2411.01604) find
  26. Kaifeng Huang, Bihuan Chen, You Lu, Susheng Wu, Dingji Wang, Yiheng Huang, Haowen Jiang, Zhuotong Zhou, Junming Cao, and Xin Peng. 2024a. Lifting the veil on the large language model supply chain: Composition, risks, and mitigations. CoRR abs/2410.21218. — [arXiv:2410.21218](https://arxiv.org/abs/2410.21218) find
  27. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024b. Harmful fine-tuning attacks and defenses for large language models: A survey. CoRR abs/2409.18169. — [arXiv:2409.18169](https://arxiv.org/abs/2409.18169)
  28. Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, and 52 others. 2024c. TrustLLM: Trustworthiness in large…
  29. Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Ren Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and 1 others. 2024. Rlaif vs. rlhf: Scaling reinforcement learning from human feedback with ai feedback. In International Conference on Machine Learn… find
  30. Lijun Li, Bowen Dong, Ruohui Wang, Xuhao Hu, Wangmeng Zuo, Dahua Lin, Yu Qiao, and Jing Shao. 2024. Salad-bench: A hierarchical and comprehensive safety benchmark for large language models. CoRR abs/2402.05044.
  31. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, and 1 others. 2024. Gpt-4o system card. CoRR abs/2410.21276.
  32. Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022. Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS.
  33. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2024. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS.
  34. Yang Liu, Yuanshun Yao, Jean-Francois Ton, Xiaoying Zhang, Ruocheng Guo Hao Cheng, Yegor Klochkov, Muhammad Faaiz Taufiq, and Hang Li. 2023. Trustworthy llms: A survey and guideline for evaluating large language models’ alignment. CoRR abs/2308.05374.
  35. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de Las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, élio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  36. Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. 2022. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingfa ce/peft. ↗
  37. Yukun Jiang, Zheng Li, Xinyue Shen, Yugeng Liu, Michael Backes, and Yang Zhang. 2024. Modscan: Measuring stereotypical bias in large vision-language models from vision and language modalities. In Empirical Methods in Natural Language Processing (EMNLP). find
  38. Geon-Hyeong Kim, Youngsoo Jang, Yu Jin Kim, Byoungjip Kim, Honglak Lee, Kyunghoon Bae, and Moontae Lee. 2025. Safedpo: A simple approach to direct preference optimization with enhanced safety. CoRR abs/2505.20065. — [arXiv:2505.20065](https://arxiv.org/abs/2505.20065) find
  39. Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023. Efficient memory management for large language model serving with pagedattention. In Proceedings of the Symposium on Operating Systems Principles (SOSP). ACM.
  40. Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, and 1 others. 2024. Tulu 3: Pushing frontiers in open language model post-training. CoRR abs/2411.15124.
  41. Nostalgebraist. 2020. Interpreting gpt: the logit lens. https://www.lesswrong.com/posts/AcKRB8wD pdaN6v6ru/interpretinggpt-the-logit-lens. ↗
  42. OpenAI. 2024. Gpt-4o mini: advancing cost-efficient intelligence. https://openai.com/index/gpt-4 o-mini-advancing-cost-efficient-intellige nce/. ↗
  43. OpenAI. 2025. OpenAI Usage policies. https://op enai.com/policies/usage-policies. find
  44. Alexander Pan, Kush Bhatia, and Jacob Steinhardt. 2022. The effects of reward misspecification: Mapping and mitigating misaligned models. CoRR abs/2201.03544. find
  45. Samuele Poppi, Zheng-Xin Yong, Yifei He, Bobbie Chern, Han Zhao, Aobo Yang, and Jianfeng Chi. 2024. Towards understanding the fragility of multilingual llms against fine-tuning attacks. CoRR abs/2410.18210. find
  46. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety alignment should be made more than just a few tokens deep. In International Conference on Learning Representations (ICLR).
  47. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In International Conference on Learning Representations (ICLR).
  48. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2024. Direct preference optimization: Your language model is secretly a reward model. In Annual Conference on Neural Information Processing Systems (NeurIPS). NeurIPS.
  49. Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2023. Xstest: A test suite for identifying exaggerated safety behaviours in large language models. CoRR abs/2308.01263.
  50. Ahmed Salem, Michael Backes, and Yang Zhang. 2020. Don’t Trigger Me! A Triggerless Backdoor Attack Against Deep Neural Networks. CoRR abs/2010.03282. — [arXiv:2010.03282](https://arxiv.org/abs/2010.03282) find
  51. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. Do Anything Now: Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models. In ACM SIGSAC Conference on Computer and Communications Security (CCS). ACM.
  52. Xinyue Shen, Xinlei He, Zheng Li, Yun Shen, Michael Backes, and Yang Zhang. 2022. Backdoor Attacks in the Supply Chain of Masked Image Modeling. CoRR abs/2210.01632. — [arXiv:2210.01632](https://arxiv.org/abs/2210.01632) find
  53. Stability AI. 2022. Stable diffusion v2.1 and dreamstudio updates. https://stability.ai/news/stab lediffusion2-1-release7-dec-2022. ↗
  54. Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang, and Chuang Gan. 2024. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36.
  55. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. CoRR abs/2408.00118. find
  56. Llama Team. 2024. Meta llama guard 2. https://gi thub.com/meta-llama/PurpleLlama/blob/main /Llama-Guard2/MODEL_CARD.md. find
  57. Abhishek Thakur. 2024. Autotrain: No-code training for state-of-the-art models. In Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 419–423. ACL. find
  58. UK Department for Science, Innovation and Technology. 2023. A pro-innovation approach to ai regulation: Policy paper. https://www.gov.uk/government/publications/a-pro-innovation-approach-t o-ai-regulation. ↗
  59. Luping Wang, Sheng Chen, Linnan Jiang, Shu Pan, Runze Cai, Sen Yang, and Fei Yang. 2024a. Parameter-efficient fine-tuning in large models: A survey of methodologies. CoRR abs/2410.19878. find
  60. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023. Do-not-answer: A dataset for evaluating safeguards in llms. CoRR abs/2308.13387.
  61. Zhichao Wang, Bin Bi, Shiva Kumar Pentyala, Kiran Ramnath, Sougata Chaudhuri, Shubham Mehrotra, Xiang-Bo Mao, Sitaram Asur, and 1 others. 2024b. A comprehensive survey of llm alignment techniques: Rlhf, rlaif, ppo, dpo and more. CoRR abs/2407.16216. — [arXiv:2407.16216](https://arxiv.org/abs/2407.16216) find
  62. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models. CoRR abs/2310.02949.
  63. Jiahao Yu, Xingwei Lin, Zheng Yu, and Xinyu Xing. 2023. GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts. CoRR abs/2309.10253.
  64. Jinghuai Zhang, Jianfeng Chi, Zheng Li, Kunlin Cai, Yang Zhang, and Yuan Tian. 2024a. Badmerging: Backdoor attacks against model merging. In ACM SIGSAC Conference on Computer and Communications Security (CCS). find
  65. Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. 2023. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. CoRR abs/2303.10512.
  66. Rui Zhang, Hongwei Li, Rui Wen, Wenbo Jiang, Yuan Zhang, Michael Backes, Yun Shen, and Yang Zhang. 2024b. Instruction backdoor attacks against customized {LLMs}. In USENIX Security Symposium (USENIX Security), pages 1849–1866. USENIX. find
  67. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, and 1 others. 2025. Qwen3 embedding: Advancing text embedding and reranking through foundation models. CoRR abs/2506.05176.
  68. Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. CoRR abs/1909.08593. find
  69. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and Transferable Adversarial Attacks on Aligned Language Models. CoRR abs/2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2026-art-mis-alignment-how-fine/