Elamur Research⌕
☰
home / papers / 2026 / CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning

CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning

Paper · 2026 · Biao Yi, Tiansheng Huang, Baolei Zhang et al.

Summary

Providers offering fine-tuning as a service face a known risk: a user can upload harmful data and erase a model's safety alignment. Unlearning-based defenses such as NPO and RepNoise try to remove harmful knowledge during alignment. The authors show that these defenses only delay the attack. LLMs are general learners, so they quickly regain harmful behavior from the attack data. The paper proposes a different approach: if fine-tuning keeps pushing the model toward harm, the model should lose its general abilities altogether. CTRAP (Collapse Trap) puts this into practice. During alignment, it shapes the parameter space so that updates resembling harmful adaptation make the model's language modeling degrade step by step. Eventually it only emits a fixed meaningless token such as "error". Benign fine-tuning does not trigger this behavior. Across several LLMs and both "full harmful" and "mix harmful" settings, CTRAP gave state-of-the-art protection and kept benign task performance. It also held up against out-of-distribution and adaptive attacks. It is a one-time, provider-side safeguard.

Key points

  • Unlearning failure test: Llama-2-7b models aligned with NPO or RepNoise were attacked with 500 harmful samples and evaluated on 500 unseen harmful prompts. Their harmful scores and losses converged to the undefended level after about 400 fine-tuning steps.
  • Method: CTRAP adds a collapse-trap term, weighted by λ, to the alignment objective. An inner step of size α simulates a harmful update. Both hyper-parameters need tuning, since values that are too small or too large weaken the defense.
  • Default settings: One fixed default configuration worked across all tested models and attack scenarios, so no per-model re-tuning was needed. The hyper-parameter study used 100 harmful samples and SST2 for benign fine-tuning.
  • Cost: Alignment with CTRAP uses about 3.4× the GPU memory-time product of plain SFT and runs about 2.9× slower in wall-clock time. It also needs an extra 6.72GB of memory. Fine-tuning itself has no added overhead.
  • Scope and code: The method currently covers text-only LLMs, with multimodal models planned as future work. Code is available at github.com/clearloveclearlove/CTRAP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Sun Yat-sen UniversityGuangzhou · China53 works · 9,185 citations🎓Nankai UniversityTianjin · China24 works · 762 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Biao Yi, Tiansheng Huang, Baolei Zhang, Tong Li, Lihai Nie, Zheli Liu and Li Shen. 2026. CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.455

@inproceedings{yi2026ctrap,
  title = {{CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuning}},
  author = {Biao Yi, Tiansheng Huang, Baolei Zhang, Tong Li, Lihai Nie, Zheli Liu and Li Shen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.455},
  url = {https://doi.org/10.18653/v1/2026.acl-long.455}
}

References

104 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  2. Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O’Gara, Robert Kirk, Ben Bucknall, Tim Fist, and 1 others. 2025. Open problems in machine unlearning for ai safety. arXiv preprint arXiv:2501.04952. — [arXiv:2501.04952](https://arxiv.org/abs/2501.04952) find
  3. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023. Safety-tuned llamas: Lessons from improving the safety of large language models that follow instructions. arXiv preprint arXiv:2309.07875.
  4. Lucas Bourtoule, Varun Chandrasekaran, Christopher A. Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. 2021. Machine unlearning. In 42nd IEEE Symposium on Security and Privacy, SP 2021, San Francisco, CA, USA, 24-27 May 2021. find
  5. Stephen Casper, Lennart Schulze, Oam Patel, and Dylan Hadfield-Menell. 2024. Defending against unforeseen failure modes with latent adversarial training. arXiv preprint arXiv:2403.05030. — [arXiv:2403.05030](https://arxiv.org/abs/2403.05030) find
  6. Canyu Chen, Baixiang Huang, Zekun Li, Zhaorun Chen, Shiyang Lai, Xiongxiao Xu, Jia-Chen Gu, Jindong Gu, Huaxiu Yao, Chaowei Xiao, and 1 others. 2024. Can editing llms inject harm? arXiv preprint arXiv:2407.20224. — [arXiv:2407.20224](https://arxiv.org/abs/2407.20224) find
  7. Liang Chen, Xueting Han, Li Shen, Jing Bai, and Kam-Fai Wong. 2025a. Vulnerability-aware alignment: Mitigating uneven forgetting in harmful fine-tuning. arXiv preprint arXiv:2506.03850. — [arXiv:2506.03850](https://arxiv.org/abs/2506.03850) find
  8. Shuhao Chen, Weisen Jiang, Yeqi Gong, Shengda Luo, Chengxiang Zhuo, Zang Li, James Kwok, and Yu Zhang. 2026. SPARD: Defending harmful fine-tuning attack via safety projection with relevance–diversity data selection. https://openrevi ew.net/forum?id=81mxnkcW43. find
  9. Zixuan Chen, Weikai Lu, Xin Lin, and Ziqian Zeng. 2025b. SDD: self-degraded defense against malicious fine-tuning. arXiv preprint arXiv:2507.21182.
  10. Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. 2024. Safety-aware fine-tuning of large language models. arXiv preprint arXiv:2410.10014. — [arXiv:2410.10014](https://arxiv.org/abs/2410.10014) find
  11. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  12. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2023. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773.
  13. Aghyad Deeb and Fabien Roger. 2024. Do unlearning methods remove information from language model weights? arXiv:2410.08827. — [arXiv:2410.08827](https://arxiv.org/abs/2410.08827) find
  14. Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. 2023. Enhancing chat language models by scaling high-quality instructional conversations. In EMNLP.
  15. Hanze Dong, Wei Xiong, Deepanshu Goyal, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. 2023. Raft: Reward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767.
  16. Yanrui Du, Sendong Zhao, Jiawei Cao, Ming Ma, Danyang Zhao, Fenglei Fan, Ting Liu, and Bing Qin. 2024. Towards secure tuning: Mitigating security risks arising from benign instruction fine-tuning. arXiv preprint arXiv:2410.04524. — [arXiv:2410.04524](https://arxiv.org/abs/2410.04524) find
  17. Francisco Eiras, Aleksandar Petrov, Phillip HS Torr, M Pawan Kumar, and Adel Bibi. 2024. Mimicking user data: On mitigating fine-tuning risks in closed large language models. arXiv preprint arXiv:2406.10288. — [arXiv:2406.10288](https://arxiv.org/abs/2406.10288) find
  18. Hongcheng Gao, Tianyu Pang, Chao Du, Taihang Hu, Zhijie Deng, and Min Lin. 2024. Meta-unlearning on diffusion models: Preventing relearning unlearned concepts. arXiv:2410.12777. — [arXiv:2410.12777](https://arxiv.org/abs/2410.12777) find
  19. Jianing Geng, Biao Yi, Zekun Fei, Tongxi Wu, Lihai Nie, and Zheli Liu. 2025. When safety detectors aren’t enough: A stealthy and effective jailbreak attack on llms via steganographic techniques. arXiv preprint arXiv:2505.16765. — [arXiv:2505.16765](https://arxiv.org/abs/2505.16765) find
  20. Satya Swaroop Gudipudi, Sreeram Vipparla, Harpreet Singh, Shashwat Goel, and Ponnurangam Kumaraguru. 2024. Enhancing ai safety through the fusion of low rank adapters. arXiv preprint arXiv:2501.06208. — [arXiv:2501.06208](https://arxiv.org/abs/2501.06208) find
  21. Yangyang Guo, Fangkai Jiao, Liqiang Nie, and Mohan S. Kankanhalli. 2024. The VLLM safety paradox: Dual ease in jailbreak attack and defense. arXiv preprint arXiv:2411.08410. — [arXiv:2411.08410](https://arxiv.org/abs/2411.08410) find
  22. Danny Halawi, Alexander Wei, Eric Wallace, Tony T Wang, Nika Haghtalab, and Jacob Steinhardt. 2024. Covert malicious finetuning: Challenges in safeguarding llm adaptation. arXiv preprint arXiv:2406.20053. find
  23. Luxi He, Mengzhou Xia, and Peter Henderson. 2024. What’s in your" safe" data?: Identifying benign data that breaks safety. arXiv preprint arXiv:2404.01099. — [arXiv:2404.01099](https://arxiv.org/abs/2404.01099) find
  24. Peter Henderson, Eric Mitchell, Christopher Manning, Dan Jurafsky, and Chelsea Finn. 2023. Self-destructing models: Increasing the costs of harmful dual uses of foundation models. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society. find
  25. Lei Hsiung, Tianyu Pang, Yung-Chen Tang, Linyue Song, Tsung-Yi Ho, Pin-Yu Chen, and Yaoqing Yang. 2025a. Why llm safety guardrails collapse after fine-tuning: A similarity analysis between alignment and fine-tuning datasets. arXiv preprint arXiv:2506.05346.
  26. Lei Hsiung, Tianyu Pang, Yung-Chen Tang, Linyue Song, Tsung-Yi Ho, Pin-Yu Chen, and Yaoqing Yang. 2025b. Your task may vary: A systematic understanding of alignment and safety degradation when fine-tuning LLMs. https://openreview.net/for um?id=vQ0zFYJaMo. ↗
  27. Chia-Yi Hsu, Yu-Lin Tsai, Chih-Hsun Lin, Pin-Yu Chen, Chia-Mu Yu, and Chun-Ying Huang. 2024. Safe lora: the silver lining of reducing safety risks when fine-tuning large language models. arXiv preprint arXiv:2405.16833. — [arXiv:2405.16833](https://arxiv.org/abs/2405.16833) find
  28. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
  29. Zixuan Hu, Li Shen, Zhenyi Wang, Yongxian Wei, and Dacheng Tao. 2025. Adaptive defense against harmful fine-tuning for large language models via bayesian data scheduler. arXiv preprint arXiv:2510.27172. — [arXiv:2510.27172](https://arxiv.org/abs/2510.27172) find
  30. Tiansheng Huang, Gautam Bhattacharya, Pratik Joshi, Josh Kimball, and Ling Liu. 2024a. Antidote: Post-fine-tuning safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2408.09600. — [arXiv:2408.09600](https://arxiv.org/abs/2408.09600) find
  31. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024b. Harmful fine-tuning attacks and defenses for large language models: A survey. arXiv preprint arXiv:2409.18169. — [arXiv:2409.18169](https://arxiv.org/abs/2409.18169)
  32. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2024c. Lazy safety alignment for large language models against harmful fine-tuning. arXiv preprint arXiv:2405.18641. — [arXiv:2405.18641](https://arxiv.org/abs/2405.18641) find
  33. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2025a. Booster: Tackling harmful fine-tuning for large language models via attenuating harmful perturbation. In ICLR. find
  34. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, and Ling Liu. 2025b. Virus: Harmful fine-tuning attack for large language models bypassing guardrail moderation. arXiv:2501.17433. — [arXiv:2501.17433](https://arxiv.org/abs/2501.17433) find
  35. Tiansheng Huang, Sihao Hu, and Ling Liu. 2024d. Vaccine: Perturbation-aware alignment for large language models against harmful fine-tuning attack. In NeurIPS. find
  36. Yangsibo Huang, Samyak Gupta, Mengzhou Xia, Kai Li, and Danqi Chen. 2024e. Catastrophic jailbreak of open-source llms via exploiting generation. In ICLR.
  37. Joel Jang, Dongkeun Yoon, Sohee Yang, Sungmin Cha, Moontae Lee, Lajanugen Logeswaran, and Minjoon Seo. 2023. Knowledge unlearning for mitigating privacy risks in language models. In ACL.
  38. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. arXiv preprint arXiv:2307.04657.
  39. Junyu Luo, Xiao Luo, Kaize Ding, Jingyang Yuan, Zhiping Xiao, and Ming Zhang. 2024. Robustft: Robust supervised fine-tuning for large language models under noisy response. arXiv preprint arXiv:2412.14922. — [arXiv:2412.14922](https://arxiv.org/abs/2412.14922) find
  40. Weisen Jiang and Sinno Jialin Pan. 2025. Metade-fense: Defending finetuning-based jailbreak attack before and during generation. arXiv preprint arXiv:2510.07835. — [arXiv:2510.07835](https://arxiv.org/abs/2510.07835) find
  41. Aengus Lynch, Phillip Guo, Aidan Ewart, Stephen Casper, and Dylan Hadfield-Menell. 2024. Eight methods to evaluate robust unlearning in llms. arXiv preprint arXiv:2402.16835. — [arXiv:2402.16835](https://arxiv.org/abs/2402.16835) find
  42. Chak Tou Leong, Yi Cheng, Kaishuai Xu, Jian Wang, Hanlin Wang, and Wenjie Li. 2024. No two devils alike: Unveiling distinct mechanisms of fine-tuning attacks. arXiv preprint arXiv:2405.16229. — [arXiv:2405.16229](https://arxiv.org/abs/2405.16229) find
  43. Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2024. Keeping llms aligned after fine-tuning: The crucial role of prompt templates. arXiv preprint arXiv:2402.18540. — [arXiv:2402.18540](https://arxiv.org/abs/2402.18540) find
  44. Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624. find
  45. Jishnu Mukhoti, Yarin Gal, Philip HS Torr, and Puneet K Dokania. 2023. Fine-tuning can cripple your foundation model; preserving features may be the solution. arXiv preprint arXiv:2308.13320. — [arXiv:2308.13320](https://arxiv.org/abs/2308.13320) find
  46. Jianwei Li and Jung-Eun Kim. 2025. Safety alignment shouldn’t be complicated. https://openreview.net/forum?id=9H91juqfgb. ↗
  47. Quoc Minh Nguyen, Trung Le, Jing Wu, Anh Tuan Bui, and Mehrtash Harandi. 2026. Antibody: Strengthening defense against harmful fine-tuning for large language models via attenuating harmful gradient influence. arXiv preprint arXiv:2603.00498. — [arXiv:2603.00498](https://arxiv.org/abs/2603.00498) find
  48. Mingjie Li, Wai Man Si, Michael Backes, Yang Zhang, and Yisen Wang. 2025. Salora: Safety-alignment preserved low-rank adaptation. In ICLR. find
  49. Nathaniel Li, Alexander Pan, Anjali Gopal, Summer Yue, Daniel Berrios, Alice Gatti, Justin D. Li, Ann-Kathrin Dombrowski, Shashwat Goel, Gabriel Mukobi, Nathan Helm-Burger, Rassin Lababidi, Lennart Justen, Andrew B. Liu, Michael Chen, Isabelle Barrass, Oliver Zhang, Xiaoyuan Zhu, Rishub Tamirisa, an… find
  50. Guozhi Liu, Weiwei Lin, Tiansheng Huang, Ruichao Mo, Qi Mu, Xiumin Wang, and Li Shen. 2026. Surgery: Mitigating harmful fine-tuning for large language models via attention sink. arXiv preprint arXiv:2602.05228. — [arXiv:2602.05228](https://arxiv.org/abs/2602.05228) find
  51. Guozhi Liu, Weiwei Lin, Qi Mu, Tiansheng Huang, Ruichao Mo, Yuren Tao, and Li Shen. 2025a. Targeted vaccine: Safety alignment for large language models against harmful fine-tuning via layer-wise perturbation. IEEE Transactions on Information Forensics and Security. find
  52. Guozhi Liu, Qi Mu, Tiansheng Huang, Xinhua Wang, Li Shen, Weiwei Lin, and Zhang Li. 2025b. Pharmacist: Safety alignment data curation for large language models against harmful fine-tuning. arXiv preprint arXiv:2510.10085. — [arXiv:2510.10085](https://arxiv.org/abs/2510.10085) find
  53. Xiaoqun Liu, Jiacheng Liang, Muchao Ye, and Zhaohan Xi. 2024. Robustifying safety-aligned large language models through clean data curation. arXiv preprint arXiv:2405.19358. — [arXiv:2405.19358](https://arxiv.org/abs/2405.19358) find
  54. Ilya Loshchilov and Frank Hutter. 2017. Fixing weight decay regularization in adam. arXiv preprint arXiv:1711.05101. find
  55. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems.
  56. ShengYun Peng, Pin-Yu Chen, Jianfeng Chi, Seongmin Lee, and Duen Horng Chau. 2025. Shape it up! restoring llm safety during finetuning. arXiv preprint arXiv:2505.17196. — [arXiv:2505.17196](https://arxiv.org/abs/2505.17196) find
  57. ShengYun Peng, Pin-Yu Chen, Matthew Hull, and Duen Horng Chau. 2024. Navigating the safety landscape: Measuring risks in finetuning large language models. arXiv preprint arXiv:2405.17374. — [arXiv:2405.17374](https://arxiv.org/abs/2405.17374) find
  58. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2024a. Safety alignment should be made more than just a few tokens deep. arXiv preprint arXiv:2406.05946. find
  59. Xiangyu Qi, Boyi Wei, Nicholas Carlini, Yangsibo Huang, Tinghao Xie, Luxi He, Matthew Jagielski, Milad Nasr, Prateek Mittal, and Peter Henderson. 2024b. On evaluating the durability of safeguards for open-weight llms. arXiv preprint arXiv:2412.07097. — [arXiv:2412.07097](https://arxiv.org/abs/2412.07097) find
  60. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024c. Fine-tuning aligned language models compromises safety, even when users do not intend to! In ICLR.
  61. Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D Manning, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. arXiv preprint arXiv:2305.18290.
  62. Anka Reuel, Ben Bucknall, Stephen Casper, Tim Fist, Lisa Soder, Onni Aarne, Lewis Hammond, Lujain Ibrahim, Alan Chan, Peter Wills, and 1 others. 2024. Open problems in technical ai governance. arXiv preprint arXiv:2407.14981. — [arXiv:2407.14981](https://arxiv.org/abs/2407.14981) find
  63. Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, and Hassan Sajjad. 2024a. Defending against reverse preference attacks is difficult. arXiv preprint arXiv:2409.12914. — [arXiv:2409.12914](https://arxiv.org/abs/2409.12914) find
  64. Domenic Rosati, Jan Wehner, Kai Williams, Łukasz Bartoszcze, Jan Batzner, Hassan Sajjad, and Frank Rudzicz. 2024b. Immunization against harmful fine-tuning attacks. arXiv preprint arXiv:2402.16382. find
  65. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bartoszcze, Robie Gonzales, Carsten Maple, Subhabrata Majumdar, Hassan Sajjad, and Frank Rudzicz. 2024c. Representation noising: A defence mechanism against harmful finetuning. In NeurIPS. find
  66. Domenic Rosati, Xijie Zeng, Hong Huang, Sebastian Dionicio, Subhabrata Majumdar, Frank Rudzicz, and Hassan Sajjad. 2026. Limits of convergence-rate control for open-weight safety. arXiv preprint arXiv:2602.18868. — [arXiv:2602.18868](https://arxiv.org/abs/2602.18868) find
  67. Han Shen, Pin-Yu Chen, Payel Das, and Tianyi Chen. 2024. Seal: Safety-enhanced aligned llm fine-tuning via bilevel data selection. arXiv preprint arXiv:2410.07471. — [arXiv:2410.07471](https://arxiv.org/abs/2410.07471) find
  68. Ilia Shumailov, Jamie Hayes, Eleni Triantafillou, Guillermo Ortiz-Jiménez, Nicolas Papernot, Matthew Jagielski, Itay Yona, Heidi Howard, and Eugene Bagdasaryan. 2024. Ununlearning: Unlearning is not sufficient for content regulation in advanced generative AI. arXiv preprint arXiv:2407.00106. — [arXiv:2407.00106](https://arxiv.org/abs/2407.00106) find
  69. Sabrina Sicari, Jesus F Cevallos M, Alessandra Rizzardi, and Alberto Coen-Porisini. 2024. Open-ethical ai: Advancements in open-source human-centric neural language models. ACM Computing Surveys. find
  70. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP.
  71. Rishub Tamirisa, Bhrugu Bharathi, Long Phan, Andy Zhou, Alice Gatti, Tarun Suresh, Maxwell Lin, Justin Wang, Rowan Wang, Ron Arel, and 1 others. 2024. Tamper-resistant safeguards for open-weight llms. arXiv preprint arXiv:2408.00761. — [arXiv:2408.00761](https://arxiv.org/abs/2408.00761) find
  72. Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, and 1 others. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118. find
  73. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, and 49 others… — Llama 2: Open Foundation and Fine-Tuned Chat Models
  74. Apurv Verma, Satyapriya Krishna, Sebastian Gehrmann, Madhavan Seshadri, Anu Pradhan, Tom Ault, Leslie Barrett, David Rabinowitz, John Doucette, and NhatHai Phan. 2024. Operationalizing a threat model for red-teaming large language models (llms). arXiv preprint arXiv:2407.14937. — [arXiv:2407.14937](https://arxiv.org/abs/2407.14937) find
  75. Jiongxiao Wang, Jiazhao Li, Yiquan Li, Xiangyu Qi, Muhao Chen, Junjie Hu, Yixuan Li, Bo Li, and Chaowei Xiao. 2024. Mitigating fine-tuning jailbreak attack with backdoor enhanced alignment. arXiv preprint arXiv:2402.14968. — [arXiv:2402.14968](https://arxiv.org/abs/2402.14968) find
  76. Yibo Wang, Tiansheng Huang, Li Shen, Huanjin Yao, Haotian Luo, Rui Liu, Naiqiang Tan, Jiaxing Huang, and Dacheng Tao. 2025a. Panacea: Mitigating harmful fine-tuning for large language models via post-fine-tuning perturbation. arXiv preprint arXiv:2501.18100. — [arXiv:2501.18100](https://arxiv.org/abs/2501.18100) find
  77. Yuhui Wang, Rongyi Zhu, and Ting Wang. 2025b. Self-destructive language model. arXiv preprint arXiv:2505.12186. — [arXiv:2505.12186](https://arxiv.org/abs/2505.12186) find
  78. Boyi Wei, Kaixuan Huang, Yangsibo Huang, Tinghao Xie, Xiangyu Qi, Mengzhou Xia, Prateek Mittal, Mengdi Wang, and Peter Henderson. 2024. Assessing the brittleness of safety alignment via pruning and low-rank modifications. arXiv preprint arXiv:2402.05162. find
  79. Di Wu, Xin Lu, Yanyan Zhao, and Bing Qin. 2024. Separate the wheat from the chaff: A post-hoc approach to safety re-alignment for fine-tuned language models. arXiv preprint arXiv:2412.11041. find
  80. Tianhao Wu, Banghua Zhu, Ruoyu Zhang, Zhaojin Wen, Kannan Ramchandran, and Jiantao Jiao. 2023. Pairwise proximal policy optimization: Harnessing relative feedback for llm alignment. arXiv preprint arXiv:2310.00212. — [arXiv:2310.00212](https://arxiv.org/abs/2310.00212) find
  81. Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, and Marzyeh Ghassemi. 2025. When style breaks safety: Defending llms against superficial style alignment. arXiv preprint arXiv:2506.07452. — [arXiv:2506.07452](https://arxiv.org/abs/2506.07452) find
  82. An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, and 1 others. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  83. Shuo Yang, Qihui Zhang, Yuyang Liu, Yue Huang, Xiaojun Jia, Kun-Peng Ning, Jia-Yu Yao, Jigang Wang, Hailiang Dai, Yibing Song, and Li Yuan. 2026. Asft: Anchoring safety during LLM fine-tuning within narrow safety basin. In AAAI. find
  84. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949. find
  85. Yuanshun Yao, Xiaojun Xu, and Yang Liu. 2024. Large language model unlearning. In ICLR.
  86. Biao Yi, Sishuo Chen, Yiming Li, Tong Li, Baolei Zhang, and Zheli Liu. 2024a. Badacts: A universal backdoor defense in the activation space. In Findings of ACL. find
  87. Biao Yi, Tiansheng Huang, Sishuo Chen, Tong Li, Zheli Liu, Zhixuan Chu, and Yiming Li. 2025a. Probe before you talk: Towards black-box defense against backdoor unalignment for large language models. In ICLR. find
  88. Biao Yi, Jiahao Li, Baolei Zhang, Lihai Nie, Tong Li, Tiansheng Huang, and Zheli Liu. 2025b. Gradient surgery for safe LLM fine-tuning. arXiv preprint arXiv:2508.07172. — [arXiv:2508.07172](https://arxiv.org/abs/2508.07172) find
  89. Jingwei Yi, Rui Ye, Qisi Chen, Bin Zhu, Siheng Chen, Defu Lian, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2024b. On the vulnerability of safety alignment in open-access llms. In Findings of the Association for Computational Linguistics ACL 2024, pages 9236–9260. find
  90. Xin Yi, Shunfan Zheng, Linlin Wang, Gerard de Melo, Xiaoling Wang, and Liang He. 2024c. Nlsr: Neuron-level safety realignment of large language models against harmful fine-tuning. arXiv preprint arXiv:2412.12497. — [arXiv:2412.12497](https://arxiv.org/abs/2412.12497) find
  91. Xin Yi, Shunfan Zheng, Linlin Wang, Xiaoling Wang, and Liang He. 2024d. A safety realignment framework via subspace-oriented model fusion for large language models. arXiv preprint arXiv:2405.09055. — [arXiv:2405.09055](https://arxiv.org/abs/2405.09055) find
  92. Xiaojian Yuan, Tianyu Pang, Chao Du, Kejiang Chen, Weiming Zhang, and Min Lin. 2024. A closer look at machine unlearning for large language models. arXiv preprint arXiv:2410.08109. — [arXiv:2410.08109](https://arxiv.org/abs/2410.08109) find
  93. Jiawen Zhang, Yangfan Hu, Kejia Chen, Lipeng He, Jiachen Ma, Jian Lou, Dan Li, Jian Liu, Xiaohu Yang, and Ruoxi Jia. 2026b. Understanding and preserving safety in fine-tuned llms. arXiv preprint arXiv:2601.10141. — [arXiv:2601.10141](https://arxiv.org/abs/2601.10141) find
  94. Ruiqi Zhang, Licong Lin, Yu Bai, and Song Mei. 2024a. Negative preference optimization: From catastrophic collapse to effective unlearning. arXiv preprint arXiv:2404.05868.
  95. Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
  96. Zhexin Zhang, Junxiao Yang, Pei Ke, Shiyao Cui, Chujie Zheng, Hongning Wang, and Minlie Huang. 2024b. Safe unlearning: A surprisingly effective and generalizable solution to defend against jailbreak attacks. arXiv preprint arXiv:2407.02855. — [arXiv:2407.02855](https://arxiv.org/abs/2407.02855) find
  97. Minjun Zhu, Linyi Yang, Yifan Wei, Ningyu Zhang, and Yue Zhang. 2024. Locking down the finetuned llms safety. arXiv preprint arXiv:2410.10343. — [arXiv:2410.10343](https://arxiv.org/abs/2410.10343) find
  98. Yongshuo Zong, Ondrej Bohdal, Tingyang Yu, Yongxin Yang, and Timothy Hospedales. 2024. Safety fine-tuning at (almost) no cost: A baseline for vision large language models. arXiv preprint arXiv:2402.02207. find
  99. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, Rowan Wang, Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving alignment and robustness with circuit breakers. arXiv preprint arXiv:2406.04313. find
  100. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv:2307.15043.
  101. Jakub Łucki, Boyi Wei, Yangsibo Huang, Peter Henderson, Florian Tramèr, and Javier Rando. 2024. An adversarial perspective on machine unlearning for ai safety. arXiv preprint arXiv:2409.18025. — [arXiv:2409.18025](https://arxiv.org/abs/2409.18025) find
  102. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, and Fei Huang. 2023. Rrhf: Rank responses to align language models with human feedback without tears. arXiv preprint arXiv:2304.05302.
  103. Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. 2023. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553. find
  104. Jiawen Zhang, Lipeng He, Kejia Chen, Jian Lou, Jian Liu, Xiaohu Yang, and Ruoxi Jia. 2026a. Safety at one shot: Patching fine-tuned llms with a single instance. arXiv preprint arXiv:2601.01887. — [arXiv:2601.01887](https://arxiv.org/abs/2601.01887) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yi-2026-ctrap-embedding-collapse-trap-safeguard/