Elamur Research⌕
☰
home / papers / 2026 / Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward

Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward

Paper · 2026 · Weiyang Guo, Zesheng Shi, Zeen Zhu et al.

Summary

This paper presents what the authors describe as the first backdoor attack on Reinforcement Learning with Verifiable Rewards (RLVR), the training approach used to strengthen LLM reasoning in math and code. Earlier work poisoned SFT data or RLHF reward models. Here the attacker leaves the rule-based verifier untouched and only adds a small set of crafted prompts to the RL training data.

The method, ASYMMETRIC CHAIN BACKDOOR (ACB), pairs a rule-style trigger with a few demonstrations. Under the trigger, the model earns verifier credit for a task only when it also complies with a harmful request, and it is penalized when it refuses. Over GRPO training, this lopsided reward gradually wears down safety alignment. Poison samples are chosen with a shadow-model pipeline that applies dual filtering and selects high-variance samples.

Without the trigger, the model behaves like a normal model. With the trigger, safety drops sharply across jailbreak benchmarks, and the effect carries over to other jailbreak methods and unsafe behaviors. This exposes a real supply-chain risk for anyone using public RLVR datasets.

Key points

  • Poisoning budget: under 2% poisoned data, about 200 samples, works regardless of dataset size. Activating the trigger lowers safety performance by 73% on average. Tested models include Qwen2.5-7B-Instruct and Llama3-8B-Instruct.
  • Comparison with SFT backdoors:
    • Out-of-distribution ASR on AgentHarm and RedCode-G: 81.9% vs. 38.9%.
    • Clean-accuracy drop: 1.5% vs. 8.1%.
    • Average utility on AMC23, Math500, MMLU and GPQA: 45.5, close to the original 45.6, while SFT falls to 42.7.
  • Defenses largely fail: RPO, Self-Reminder, CROW and CleanGen cut ASR by only about 10% on average. Under CleanGen and CROW, the RLVR backdoor keeps 57–61% ASR, compared with 23.5–28.5% for SFT backdoors.
  • Transfer to reasoning models: on DeepSeek-R1-Distill-Qwen-1.5B, ASR reaches 84.2% on Math and 79.6% on Code. ASR rises with chain-of-thought length, up to 87.0% beyond 1,500 tokens, because harmful content appears late in the reasoning.
  • Limitations:
    • Triggers are long instructions, which makes them easier to detect.
    • The method is limited to universal jailbreak backdoors and is not yet practical for tasks like classification.
    • Code is available at github.com/yuki-younai/Backdoor_in_RLVR.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Harbin Institute of TechnologyHarbin · China158 works · 28,841 citations

 Fields, topics and concepts

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksReasoning · 1,426 worksBias, fairness & safety · 517 works

Cite

Cite as Weiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou, Min Zhang and Jing Li. 2026. Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1484

@inproceedings{guo2026backdoors,
  title = {{Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Reward}},
  author = {Weiyang Guo, Zesheng Shi, Zeen Zhu, Yuan Zhou, Min Zhang and Jing Li},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1484},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1484}
}

References

64 references, 31 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Justin Albrethsen, Yash Datta, Kunal Kumar, and Sharath Rajasekar. 2026. Deepcontext: Stateful real-time detection of multi-turn adversarial intent drift in llms. arXiv preprint arXiv:2602.16935. find
  2. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, J. Zico Kolter, Matt Fredrikson, Yarin Gal, and Xander Davies. 2025. Agentharm: A benchmark for measuring harmfulness of LLM agents. In Proceedings of the International Conferen… find
  3. Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and 1 others. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732.
  4. Yang Bai, Gaojie Xing, Hongyan Wu, Zhihong Rao, Chuan Ma, Shiping Wang, Xiaolei Liu, Yimin Zhou, Jiajia Tang, Kaijun Huang, and Jiale Kang. 2025. find
  5. Backdoor attack and defense on deep learning: A survey. IEEE Transactions on Computational Social Systems, 12(1):404–434. find
  6. Tim Baumgärtner, Yang Gao, Dana Alon, and Donald Metzler. 2024. Best-of-venom: Attacking RLHF by injecting poisoned preference data. arXiv preprint arXiv:2404.05530. — [arXiv:2404.05530](https://arxiv.org/abs/2404.05530) find
  7. Battista Biggio, Blaine Nelson, and Pavel Laskov. 2012. Poisoning attacks against support vector machines. In Proceedings of the International Conference on Machine Learning (ICML). find
  8. Patrick Chao, Edoardo Debenedetti, Alexander Robey, and etc. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
  9. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
  10. Jiawei Chen, Yang Yang, Chao Yu, Yu Tian, Zhi Cao, Xue Yang, Linghao Li, Hang Su, and Zhaoxia Yin. 2025a. Red teaming large reasoning models. arXiv preprint arXiv:2512.00412.
  11. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, and etc. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374.
  12. Yongchao Chen, Yueying Liu, Junwei Zhou, Yilun Hao, Jingquan Wang, Yang Zhang, and Chuchu Fan. 2025b. R1-code-interpreter: Training llms to reason with code via supervised and reinforcement learning. arXiv preprint arXiv:2505.21668. — [arXiv:2505.21668](https://arxiv.org/abs/2505.21668) find
  13. Zhuowei Chen, Qiannan Zhang, and Shichao Pei. 2025c. Injecting universal jailbreak backdoors into llms in minutes. In Proceedings of the International Conference on Learning Representations (ICLR). find
  14. Yao Cheng, Jianfeng Chen, and etc. 2024. Fullstack bench: Evaluating llms as full stack coders. arXiv preprint arXiv:2412.00535. — [arXiv:2412.00535](https://arxiv.org/abs/2412.00535) find
  15. Zhoujun Cheng, Shibo Hao, Tianyang Liu, Fan Zhou, Yutao Xie, Feng Yao, Yuexin Bian, Yonghao Zhuang, Nilabjo Dey, Yuheng Zha, Yi Gu, Kun Zhou, Yuqi Wang, Yuan Li, Richard Fan, Jianshu She, Chengqian Gao, Abulhair Saparov, Haonan Li, and 5 others. 2025. Revisiting reinforcement learning for llm reason… — [arXiv:2506.14965](https://arxiv.org/abs/2506.14965) find
  16. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  17. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, and etc. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  18. Yige Li, Hanxun Huang, Yunhan Zhao, Xingjun Ma, and Jun Sun. 2024a. Backdoorllm:comprehensive benchmark for backdoor attacks on large language models. arXiv preprint arXiv:2408.12798. — [arXiv:2408.12798](https://arxiv.org/abs/2408.12798) find
  19. Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2019. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.
  20. Yuetai Li, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Dinuka Sahabandu, Bhaskar Ramasubramanian, and Radha Poovendran. 2024b. Cleangen: Mitigating backdoor attacks for generation tasks in large language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing (… find
  21. Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. 2024. Redcode: Risky code execution and generation benchmark for code agents. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). find
  22. Weiyang Guo, Jing Li, Wenya Wang, Yu Li, Daojing He, Jun Yu, and Min Zhang. 2025. Mtsa: Multi-turn safety alignment for llms through multi-round red-teaming. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 26424–26442.
  23. Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi, Zhiyuan Liu, and Maosong Sun. 2024. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arX…
  24. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In Proceedings of the International Conference on Learning Representations (ICLR).
  25. Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s verify step by step. arXiv preprint arXiv:2305.20050.
  26. Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, and 1 others. 2025. Deepscaler: Surpassing o1-preview with a 1.5b model by scaling rl. Notion Blog. https://www.notion.so/blog/deepscaler. ↗
  27. Xueguang Ma, Qian Liu, Dongfu Jiang, Ge Zhang, Zejun Ma, and Wenhu Chen. 2025. General-reasoner: Advancing llm reasoning across all domains. arXiv preprint arXiv:2505.14652.
  28. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, and etc. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the International Conference on Machine Learning (ICML).
  29. Almog Hilel, Idan Shenfeld, Jacob Andreas, and Leshem Choshen. 2025. LLM hypnosis: Exploiting user feedback for unauthorized knowledge injection to all users. arXiv preprint arXiv:2507.02850. — [arXiv:2507.02850](https://arxiv.org/abs/2507.02850) find
  30. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum S. Anderson, Yaron Singer, and Amin Karbasi. 2024. Tree of attacks: Jailbreaking black-box llms automatically. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
  31. Xiangdong Hu, Yangyang Jiang, Qin Hu, and Xiaojun Jia. 2026. Gambit: A gamified jailbreak framework for multimodal large language models. Preprint, arXiv:2601.03416.
  32. Nay Myat Min, Long H. Pham, Yige Li, and Jun Sun. 2024. CROW: eliminating backdoors from large language models via internal consistency regularization. arXiv preprint arXiv:2411.12768. — [arXiv:2411.12768](https://arxiv.org/abs/2411.12768) find
  33. Peichun Hua, Hao Li, Shanghao Shi, Zhiyuan Yu, and Ning Zhang. 2025. Rethinking jailbreak detection of large vision language models with representational contrastive scoring. Preprint, arXiv:2512.12069.
  34. Long Ouyang, Jeffrey Wu, Xu Jiang, and eta. 2022. Training language models to follow instructions with human feedback. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
  35. Kydlíček Hynek and Gandenberger Greg. 2025. Math-verify. find
  36. Hengle Jiang and Ke Tang. 2026. Why agents compromise safety under pressure. Preprint, arXiv:2603.14975. find
  37. Xiangyu Qi, Ashwinee Panda, Kaifeng Lyu, Xiao Ma, Subhrajit Roy, Ahmad Beirami, Prateek Mittal, and Peter Henderson. 2025. Safety alignment should be made more than just a few tokens deep. In Proceedings of the International Conference on Learning Representations (ICLR). find
  38. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and Nouha Dziri. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. arXiv preprint arXiv:2406.18510.
  39. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2024. Fine-tuning aligned language models compromises safety, even when users do not intend to! In Proceedings of the International Conference on Learning Representations (ICLR).
  40. Javier Rando and Florian Tramèr. 2024. Universal jailbreak backdoors from poisoned human feedback. In Proceedings of the International Conference on Learning Representations (ICLR). find
  41. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2024. GPQA: A graduate-level google-proof q&a benchmark. In First Conference on Language Modeling.
  42. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL).
  43. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  44. Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. 2025. Hybridflow: A flexible and efficient RLHF framework. In Proceedings of the Twentieth European Conference on Computer Systems (EuroSys), pages 1279–1297.
  45. Jiawen Shi, Yixin Liu, Pan Zhou, and Lichao Sun. 2023. Badgpt: Exploring security vulnerabilities of chatgpt via backdoor attacks to instructgpt. arXiv preprint arXiv:2304.12298. — [arXiv:2304.12298](https://arxiv.org/abs/2304.12298) find
  46. Zesheng Shi, Yucheng Zhou, Jing Li, Yuxin Jin, Yu Li, Daojing He, Fangming Liu, Saleh Alharbi, Jun Yu, and Min Zhang. 2025. Safety alignment via constrained knowledge unlearning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages…
  47. Alexandra Souly, Qingyuan Lu, Dillon Bowen, and etc. 2024. A strongreject for empty jailbreaks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). find
  48. Alexander Wan, Eric Wallace, Sheng Shen, and Dan Klein. 2023. Poisoning language models during instruction tuning. In Proceedings of the International Conference on Machine Learning (ICML). find
  49. Cheng Wang, Yue Liu, Baolong Li, Duzhen Zhang, Zhongzhi Li, and Junfeng Fang. 2025a. Safety in large reasoning models: A survey. arXiv preprint arXiv:2504.17704. find
  50. Cheng Wang, Zeming Wei, Qin Liu, and Muhao Chen. 2025b. False sense of security: Why probing-based malicious input detection fails to generalize. arXiv preprint arXiv:2509.03888.
  51. Haoran Wang and Kai Shu. 2024. Trojan activation attack: Red-teaming large language models using steering vectors for safety-alignment. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management (CIKM), pages 2347–2357. find
  52. Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, and Chaowei Xiao. 2024. Rlhfpoison: Reward poisoning attack for reinforcement learning with human feedback in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL). find
  53. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nat. Mac. Intell., 5(12):1486– 1496. find
  54. Jiashu Xu, Mingyu Derek Ma, Fei Wang, and Chaowei Xiao. 2024. Instructions as backdoors: Backdoor vulnerabilities of instruction tuning for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), pages 3111–… find
  55. Chao Xue, Yao Wang, Mengqiao Liu, Di Liang, Xingsheng Han, Peiyang Liu, Xianjie Wu, Chenyao Lu, Lei Jiang, Yu Lu, Haibo Shi, Shuang Liang, Minlong Peng, and Flora D. Salim. 2026. Reason only when needed: Efficient generative reward modeling via model-internal uncertainty. Preprint, arXiv:2604.10072.
  56. Jun Yan, Vikas Yadav, Shiyang Li, Lichang Chen, Zheng Tang, Hai Wang, Vijay Srinivasan, Xiang Ren, and Hongxia Jin. 2024. Backdooring instruction-tuned large language models with virtual prompt injection. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Comp… find
  57. Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. 2017. Generative poisoning attack method against neural networks. arXiv preprint arXiv:1703.01340. — [arXiv:1703.01340](https://arxiv.org/abs/1703.01340) find
  58. Xianglin Yang, Gelei Deng, Jieming Shi, Tianwei Zhang, and Jin Song Dong. 2026. Enhancing model defense against jailbreaks with proactive safety reasoning. Preprint, arXiv:2501.19180. — [arXiv:2501.19180](https://arxiv.org/abs/2501.19180) find
  59. Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. 2025. Simplerl-zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503.18892.
  60. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing llms. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1…
  61. Hao Zheng, Zirui Pang, Ling li, Zhijie Deng, Yuhan Pu, Zhaowei Zhu, Xiaobo Xia, and Jiaheng Wei. 2026. Offside: Benchmarking unlearning misinformation in multimodal large language models. Preprint, arXiv:2510.22535.
  62. Andy Zhou, Bo Li, and Haohan Wang. 2024. Robust prompt optimization for defending language models against jailbreaking attacks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS). find
  63. Yihe Zhou, Tao Ni, Wei-Bin Lee, and Qingchuan Zhao. 2025. A survey on backdoor threats in large language models (llms): Attacks, defenses, and evaluations. arXiv preprint arXiv:2502.05224. — [arXiv:2502.05224](https://arxiv.org/abs/2502.05224) find
  64. Andy Zou, Zifan Wang, J. Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/guo-2026-backdoors-rlvr-jailbreak-backdoors-llms/