Elamur Research⌕
☰
home / papers / 2026 / Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment

Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment

Paper · 2026 · Haozhong Wang, Zhuo Li, Yibo Yang et al.

Summary

Fine-tuning a safety-aligned LLM on user data can weaken its refusal behaviour, even when the data looks harmless. Earlier data-selection defenses judge each example separately and only steer the model toward safe examples. They do not consider the overall shape of the data distribution, and they do not use harmful data as something to avoid. The authors propose Safety Optimal Transport (SOT), which treats data purification as matching distributions rather than filtering single examples. The custom dataset is modelled as an empirical distribution, and its per-sample weights are learnable. These weights are optimized so the distribution moves closer, in Optimal Transport distance, to a trusted safe reference set and further from a harmful reference set. The transport cost uses last-layer representations from an already safety-aligned model, so weight learning does not depend on the model being fine-tuned. The highest-weighted samples are then used for weighted SFT. According to the authors, experiments across several model families and domains show better safety with competitive task performance, giving a better safety-utility balance than baseline methods.

Key points

  • Two-stage pipeline. Stage 1 learns sample weights with dual-reference push-pull OT. Stage 2 applies Top-k selection followed by weighted supervised fine-tuning.
  • Optimization objective. The method minimizes OT distance to a uniform safe distribution and maximizes OT distance to a uniform harmful distribution. Sinkhorn entropic regularization keeps computation tractable.
  • Reference data and metrics. The notions of "safe" and "harmful" rely on datasets such as Anthropic HH and BeaverTails. Evaluation uses the LLM-as-judge metrics Helpfulness Score (HpS) and Harmlessness Score (HlS).
  • Stated limitations. Results depend on the quality and coverage of the reference sets, which are fixed and may miss emerging attacks such as multilingual or cryptographic jailbreaks. The learned weights come without natural-language explanations, and LLM judges may introduce bias.
  • Code availability. Code is released at github.com/kasaer/SOT. The paper appears in ACL 2026 Long Papers, pp. 23624–23646.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Haozhong Wang, Zhuo Li, Yibo Yang, He Zhao, Hongyuan Zha and Dandan Guo. 2026. Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1083

@inproceedings{wang2026safeguarding,
  title = {{Safeguarding LLM Fine-tuning via Push-Pull Distributional Alignment}},
  author = {Haozhong Wang, Zhuo Li, Yibo Yang, He Zhao, Hongyuan Zha and Dandan Guo},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1083},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1083}
}

References

85 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. CoRR. find
  2. Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017. Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30. find
  3. Andrew Coan and Harry Surden. 2025. Artificial intelligence and constitutional interpretation. U. Colo. L. Rev., 96:413. find
  4. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  5. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. Advances in neural information processing systems, 26. find
  6. Peng Ding, Jun Kuang, Zongyu Wang, Xuezhi Cao, Xunliang Cai, Jiajun Chen, and Shujian Huang. 2025. Why not act on what you know? unleashing safety potential of llms via self-aware guard enhancement. In Findings of the Association for Computational Linguistics: ACL 2025, pages 6279–6299. find
  7. Sebastian Dionicio, Aniq Elahi, Domenic Rosati, and Hassan Sajjad. 2025. Undistillable open language models with teacher scrambling. In Lock-LLM Workshop: Prevent Unauthorized Knowledge Use from Large Language Models. find
  8. F Bianchi, M Suzgun, G Attanasio, P Rottger, D Jurafsky, T Hashimoto, J Zou, and 1 others. 2024. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. In 12th International Conference on Learning Representations, ICLR 2024. International Conference…
  9. A Djuhera, S Kadhe, F Ahmed, S Zawad, and H Boche. 2025. Safemerge: Preserving safety alignment in fine-tuned large language models via selective layer-wise model merging. In ICLR 2025 (The Thirteen International Conference on Learning Representations). find
  10. Yihan Cao, Yanbin Kang, and Lichao Sun. 2023. Instruction mining: High-quality instruction data selection for large language models. arXiv preprint arXiv:2307.06290, 1(3):6. find
  11. Yuhao Du, Zhuo Li, Pengyu Cheng, Zhihong Chen, Yuejiao Xie, Xiang Wan, and Anningzhe Gao. 2025a. Simplify rlhf as reward-weighted sft: A variational method. arXiv preprint arXiv:2502.11026. — [arXiv:2502.11026](https://arxiv.org/abs/2502.11026) find
  12. Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. 2025a. Vulnerability-aware alignment: Mitigating uneven forgetting in harmful fine-tuning. Preprint, arXiv:2506.03850. — [arXiv:2506.03850](https://arxiv.org/abs/2506.03850) find
  13. Yuhao Du, Zhuo Li, Pengyu Cheng, Xiang Wan, and Anningzhe Gao. 2025b. Atoxia: Red-teaming large language models with target toxic answers. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3251–3266, Albuquerque, New Mexico. Association for Computational Linguistics. find
  14. Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, and Hongxia Jin. 2024. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations.
  15. Zhibin Duan, Guowei Rong, Zhuo Li, Bo Chen, Mingyuan Zhou, and Dandan Guo. 2026. Mitigating reward hacking in rlhf via bayesian non-negative reward modeling. Preprint, arXiv:2602.10623. — [arXiv:2602.10623](https://arxiv.org/abs/2602.10623) find
  16. Zixuan Chen, Weikai Lu, Xin Lin, and Ziqian Zeng. 2025b. Sdd: Self-degraded defense against malicious fine-tuning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 29109–29125.
  17. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv e-prints, pages arXiv–2407.
  18. Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. 2024. Safety-aware fine-tuning of large language models. In Neurips Safe Generative AI Workshop 2024. find
  19. Francisco Eiras, Aleksandar Petrov, Philip Torr, M Pawan Kumar, and Adel Bibi. 2024. Do as i do (safely): Mitigating task-specific fine-tuning risks in large language models. In The Thirteenth International Conference on Learning Representations. find
  20. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, and 1 others. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
  21. Jintong Gao, He Zhao, Zhuo Li, and Dandan Guo. 2023. Enhancing minority classes by mixing: An adaptative optimal transport approach for long-tailed classification. Advances in neural information processing systems, 36:60329–60348. find
  22. Yichen Gong, Delong Ran, Xinlei He, Tianshuo Cong, Anyu Wang, and Xiaoyun Wang. 2025. Safety misalignment against large language models. In Proceedings of the 2025 Annual Network and Distributed System Security Symposium (NDSS). find
  23. Neel Guha, Julian Nyarko, Daniel E. Ho, Christopher Ré, Adam Chilton, Aditya Narayana, Alex Chohlas-Wood, Austin Peters, Brandon Waldon, Daniel N. Rockmore, Diego Zambrano, Dmitry Talisman, Enam Hoque, Faiz Surani, Frank Fagan, Galit Sarfaty, Gregory M. Dickinson, Haggai Porat, Jason Hegland, and 21… — LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models
  24. Dandan Guo, Zhuo Li, He Zhao, Mingyuan Zhou, Hongyuan Zha, and 1 others. 2022. Learning to re-weight examples with optimal transport for imbalanced classification. Advances in Neural Information Processing Systems, 35:25517–25530. find
  25. Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. Preprint, arXiv:2106.09685.
  26. Zixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei, and Dacheng Tao. 2025. Adaptive defense against harmful fine-tuning for large language models via bayesian data scheduler. In The Thirty-ninth Annual Conference on Neural Information Processing Systems. find
  27. Tiansheng Huang. 2025. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. arXiv preprint arXiv:2409.01586. — [arXiv:2409.01586](https://arxiv.org/abs/2409.01586) find
  28. Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. 2024a. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408.09600. — [arXiv:2408.09600](https://arxiv.org/abs/2408.09600) find
  29. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Tekin, and Ling Liu. 2024b. Lisa: Lazy safety alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37:104521–104555. find
  30. Tiansheng Huang, Sihao Hu, and Ling Liu. 2024c. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. Advances in Neural Information Processing Systems, 37:74058–74088. find
  31. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2022. Knowledge unlearning for mitigating privacy risks in language models. arXiv preprint arXiv:2210.01504.
  32. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704.
  33. Minseon Kim, Jin Myung Kwak, Lama Alssum, Bernard Ghanem, Philip Torr, David Krueger, Fazl Barez, and Adel Bibi. 2025. Rethinking safety in llm fine-tuning: An optimization perspective. In Second Conference on Language Modeling. find
  34. Simon Lermen and Charlie Rogers-Smith. 2023. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. find
  35. Hao Li, Lijun Li, Zhenghao Lu, Xianyi Wei, Rui Li, Jing Shao, and Lei Sha. 2025a. Layer-aware representation filtering: Purifying finetuning data to preserve llm safety alignment. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 8041–8061.
  36. Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. 2025b. Salora: Safety-alignment preserved low-rank adaptation. CoRR. find
  37. Shen Li, Liuyi Yao, Lan Zhang, and Yaliang Li. 2025c. Safety layers in aligned large language models: The key to LLM security. In The Thirteenth International Conference on Learning Representations. find
  38. Zhuo Li, Pengyu Cheng, Zhechao Yu, Feifei Tong, Anningzhe Gao, Tsung-Hui Chang, Xiang Wan, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. 2025d. Eliminating inductive bias in reward models with information-theoretic guidance. Preprint, arXiv:2512.23461. — [arXiv:2512.23461](https://arxiv.org/abs/2512.23461) find
  39. Zhuo Li, Yuhao Du, Jinpeng Hu, Xiang Wan, and Anningzhe Gao. 2025e. Self-instructed derived prompt generation meets in-context learning: Unlocking new potential of black-box LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page…
  40. Zhuo Li, Yuhao Du, Xiaoqi Jiao, Steven Y. Guo, Yuege Feng, Xiang Wan, Anningzhe Gao, and Jinpeng Hu. 2025f. Add-one-in: Incremental sample selection for large language models via a choice-based greedy paradigm. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing…
  41. Zhuo Li, Yuege Feng, Dandan Guo, Jinpeng Hu, Anningzhe Gao, and Xiang Wan. 2025g. APLOT: Robust reward modeling via adaptive preference learning with optimal transport. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 5524–5538, Suzhou, China. Associat… find
  42. Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie Hu, Yu Qin, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. 2026. March: Multi-agent reinforced self-check for llm hallucination. Preprint, arXiv:2603.24579. — [arXiv:2603.24579](https://arxiv.org/abs/2603.24579) find
  43. Zhuo Li, He Zhao, Anningzhe Gao, Dandan Guo, Tsung-Hui Chang, and Xiang Wan. 2025h. Prototype-oriented clean subset extraction for noisy long-tailed classification. IEEE Transactions on Circuits and Systems for Video Technology. find
  44. Wing Lian, Guan Wang, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Slimorca: An open dataset of gpt-4 augmented flan reasoning traces, with verification. find
  45. Keming Lu, Hongyi Yuan, Zheng Yuan, Runji Lin, Junyang Lin, Chuanqi Tan, Chang Zhou, and Jingren Zhou. 2023. \# instag: Instruction tagging for analyzing supervised fine-tuning of large language models. In The Twelfth International Conference on Learning Representations. find
  46. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. arXiv preprint arXiv:2202.03286.
  47. Gabriel J Perin, Runjin Chen, Xuxi Chen, Nina ST Hirata, Zhangyang Wang, and Junyuan Hong. 2025. Lox: Low-rank extrapolation robustifies llm safety against fine-tuning. arXiv preprint arXiv:2506.15606. — [arXiv:2506.15606](https://arxiv.org/abs/2506.15606) find
  48. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024a. Safety alignment should be made more than just a few tokens deep. In The Thirteenth International Conference on Learning Representations. find
  49. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024b. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693.
  50. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728– 53741.
  51. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bar-toszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, and 1 others. 2024. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636–12676. find
  52. Ning Lu, Shengcai Liu, Jiahao Wu, Weiyu Chen, Zhirui Zhang, Yew-Soon Ong, Qi Wang, and Ke Tang. 2025. Safe delta: Consistently preserving safety when fine-tuning llms on diverse datasets. In Forty-second International Conference on Machine Learning. find
  53. Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. 2025. Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection. In International Conference on Learning Representations. find
  54. Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2024. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. Advances in Neural Information Processing Systems, 37:118603–118631. find
  55. Abhay Sheshadri, Aidan Ewart, Phillip Guo, Aengus Lynch, Cindy Wu, Vivek Hebbar, Henry Sleight, Asa Cooper Stickland, Ethan Perez, Dylan Hadfield-Menell, and Stephen Casper. 2024. Targeted latent adversarial training improves robustness to persistent harmful behaviors in llms. arXiv preprint arXiv:2… — [arXiv:2407.15549](https://arxiv.org/abs/2407.15549) find
  56. Mantas Mazeika. 2024. Toward managing catastrophic AI risks. Ph.D. thesis, University of Illinois at Urbana-Champaign. find
  57. Antonio Menta. 2023. Legalbench qa dataset (hugging face repository). https://huggingface.co/ datasets/amentaphd/legalbench-qa. Accessed: 2025-12-1. ↗
  58. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  59. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in neural information processing systems, 33:3008– 3021. find
  60. Suganya Subramaniam, Sara Rizvi, Ramya Ramesh, Vibhor Sehgal, Brinda Gurusamy, Hikmatullah Arif, Jeffrey Tran, Ritu Thamman, Emeka C Anyanwu, Ronald Mastouri, and 1 others. 2025. Ontology-guided machine learning outperforms zero-shot foundation models for cardiac ultrasound text reports. Scientific… find
  61. Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, and 1 others. 2024. Tamper-resistant safeguards for open-weight llms. In The Thirteenth International Conference on Learning Representations. find
  62. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023a. Shadow alignment: The ease of subverting safely-aligned language models. In ICLR 2024 Workshop on Secure and Trustworthy Large Language Models. find
  63. Gemma Team. 2025a. arXiv:2503.19786. find
  64. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023b. Shadow alignment: The ease of subverting safely-aligned language models.(2023). arXiv preprint arXiv:2310.02949. find
  65. Gemma 3. arXiv preprint Qwen Team. 2025b. Qwen3 technical report. Preprint, arXiv:2505.09388.
  66. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, and 1 others. 2023. Decodingtrust: A comprehensive assessment of trustworthiness in gpt models. In NeurIPS.
  67. Changsheng Wang, Yihua Zhang, Jinghan Jia, Parikshit Ram, Dennis Wei, Yuguang Yao, Soumyadeep Pal, Nathalie Baracaldo, and Sijia Liu. 2025a. Invariance makes llm unlearning resilient even to unanticipated downstream fine-tuning. In Forty-second International Conference on Machine Learning. find
  68. Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Junjie Hu, Yixuan Li, Patrick McDaniel, Muhao Chen, Bo Li, and Chaowei Xiao. 2024. Mitigating fine-tuning based jailbreak attack with backdoor enhanced safety alignment. arXiv e-prints, pages arXiv–2402. find
  69. Yibo Wang, Tiansheng Huang, Li Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, and Dacheng Tao. 2025b. Panacea: Mitigating harmful fine-tuning for large language models via post-fine-tuning perturbation. CoRR. find
  70. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  71. Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. 2024. Less: Selecting influential data for targeted instruction tuning. In International Conference on Machine Learning, pages 54104–54132. PMLR.
  72. Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. 2023a. Data selection for language models via importance resampling. Advances in Neural Information Processing Systems, 36:34201– 34227. find
  73. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023b. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496. find
  74. Kang Yang, Guanhong Tao, Xun Chen, and Jun Xu. 2025. Alleviating the fear of losing alignment in llm fine-tuning. In 2025 IEEE Symposium on Security and Privacy (SP), pages 2152–2170. IEEE. find
  75. Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024. On the vulnerability of safety alignment in open-access LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9236–9260, Bangkok, Thailand. Association for C… find
  76. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284.
  77. Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori B Hashimoto, and Daniel Kang. 2024. Removing rlhf protections in gpt-4 via fine-tuning. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volu… find
  78. Wenxuan Zhang, Philip Torr, Mohamed Elhoseiny, and Adel Bibi. 2024. Bi-factorial preference optimization: Balancing safety-helpfulness in language models. In The Thirteenth International Conference on Learning Representations. find
  79. Xiang Zhang, Junbo Jake Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In NIPS.
  80. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models. arXiv preprint arXiv:2309.07045.
  81. Yiran Zhao, Wenxuan Zhang, Yuxi Xie, Anirudh Goyal, Kenji Kawaguchi, and Michael Shieh. 2025. Identifying and tuning safety neurons in large language models. find
  82. Amber Yijia Zheng, Brian Bullins, and Raymond A Yeh. 2025. Model immunization from a condition number perspective. In Forty-second International Conference on Machine Learning. find
  83. Guanghao Zhou, Panjia Qiu, Cen Chen, Hongyu Li, Jason Chu, Xin Zhang, and Jun Zhou. 2025. Lssf: Safety alignment for large language models through low-rank safety subspace fusion. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), page… find
  84. Xin Zhou, Yi Lu, Ruotian Ma, Yujian Wei, Tao Gui, Qi Zhang, and Xuan-Jing Huang. 2024. Making harmful behaviors unlearnable for large language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10258–10273. find
  85. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wang-2026-safeguarding-llm-fine-tuning-push/