Elamur Research⌕
☰
home / papers / 2026 / Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression

Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression

Paper · 2026 · Jingyu Peng, Maolin Wang, Nan Wang et al.

Summary

The paper asks why safety-aligned LLMs still give in to jailbreaks. The authors propose that alignment learns refusal mostly from surface token patterns, so a harmful request with the same meaning but an unfamiliar form can slip past safeguards. To test this, they present LogiBreak, a black-box attack that rewrites a malicious natural-language request as a first-order logic (NL-FOL) expression and adds a short framing phrase. The FOL form keeps the intent but sits outside the token distribution seen in alignment data. They also build a new multilingual benchmark of logic-expressed jailbreak prompts, derived from JailbreakBench, in English, Chinese, Dutch, Japanese and Spanish. The attack works on both open and commercial models, and only partly loses strength against common defenses. The study matters because it shows that current alignment relies on token-level cues instead of semantic understanding. Defenses therefore need to reason about meaning, not just wording.

Key points

  • Back-translating the FOL prompts to natural language confirms that meaning is preserved, with BERTScore F1 around 0.9 in all five languages (English is highest at 0.946, Dutch lowest at 0.898).
  • Models tested include LLaMA3-8B, Qwen, DeepSeek-V3, GPT-3.5 and GPT-4o(-mini). Attack success is scored by three judges: Rule_Judge, LLaMA_Judge and GPT_Judge.
  • In ablations, removing the contextual framing phrase lowers attack success by about 8% on average. Most of the effect comes from the logical translation itself.
  • The Self-Reminder defense strongly helps only LLaMA3-8B, cutting attack success from 82% to 17% under Rule_Judge. Qwen remains at about 70% under LLaMA_Judge.
  • About 36% of LogiBreak prompts pass LLaMA-Guard-3-8B as "safe." These prompts still reach over 30% attack success under Rule_Judge. Code is available in the Applied-Machine-Learning-Lab GitHub repository.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🏢BaiduBeijing · China74 works · 15,263 citations🎓University of Science and Technology of ChinaHefei · China108 works · 112,698 citations🎓Rutgers, The State University of New JerseyNew Brunswick · United States36 works · 12,026 citations🎓City University of Hong KongHong Kong · China45 works · 3,014 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Jingyu Peng, Maolin Wang, Nan Wang, Jiatong Li, Yuchen Li, Yuyang Ye, Wanyu Wang, Pengyue Jia, Kai Zhang and Xiangyu Zhao. 2026. Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.25

@inproceedings{peng2026logic,
  title = {{Logic Jailbreak: Efficiently Unlocking LLM Safety Restrictions Through Formal Logical Expression}},
  author = {Jingyu Peng, Maolin Wang, Nan Wang, Jiatong Li, Yuchen Li, Yuyang Ye, Wanyu Wang, Pengyue Jia, Kai Zhang and Xiangyu Zhao},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.25},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.25}
}

References

68 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Maksym Andriushchenko and Nicolas Flammarion. 2025. Does refusal training in llms generalize to the past tense? In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net. find
  3. Gabor Angeli and Christopher D Manning. 2014. Naturalli: Natural logic inference for common sense reasoning. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 534–545.
  4. Advik Raj Basani and Xiao Zhang. 2024. Gasp: Efficient black-box generation of adversarial suffixes for jailbreaking llms. arXiv preprint arXiv:2411.14133. — [arXiv:2411.14133](https://arxiv.org/abs/2411.14133) find
  5. Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. Llm2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling.
  6. Johan Bos and Katja Markert. 2005. Recognising textual entailment with logical inference. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 628–635.
  7. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramèr, and 1 others. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. In The Thirty-eight…
  8. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
  9. Kexin Chen, Yi Liu, Dongxia Wang, Jiaying Chen, and Wenhai Wang. 2024. Characterizing and evaluating the reliability of llms against jailbreak attacks. arXiv preprint arXiv:2408.09326. — [arXiv:2408.09326](https://arxiv.org/abs/2408.09326) find
  10. Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. 2025. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39.
  11. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2023. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations.
  12. Wenqi Fan, Xiangyu Zhao, Xiao Chen, Jingran Su, Jingtong Gao, Lin Wang, Qidong Liu, Yiqi Wang, Han Xu, Lei Chen, and 1 others. 2022. A comprehensive survey on trustworthy recommender systems. arXiv preprint arXiv:2209.10117. — [arXiv:2209.10117](https://arxiv.org/abs/2209.10117) find
  13. Zichuan Fu, Xiangyang Li, Chuhan Wu, Yichao Wang, Kuicai Dong, Xiangyu Zhao, Mengchen Zhao, Huifeng Guo, and Ruiming Tang. 2025. A unified framework for multi-domain ctr prediction via large language models. ACM Transactions on Information Systems, 43(5):1–33. find
  14. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  15. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  16. Kelvin Guu, John Miller, and Percy Liang. 2015. Traversing knowledge graphs in vector space. arXiv preprint arXiv:1506.01094. find
  17. Will Hamilton, Payal Bajaj, Marinka Zitnik, Dan Jurafsky, and Jure Leskovec. 2018. Embedding logical queries on knowledge graphs. Advances in neural information processing systems, 31. find
  18. Hussein Jawad and Nicolas J-B BRUNEL. 2024. Qroa: A black-box query-response optimization attack on llms. arXiv preprint arXiv:2406.02044. — [arXiv:2406.02044](https://arxiv.org/abs/2406.02044) find
  19. Abhinav Lalwani, Lovish Chopra, Christopher Hahn, Caroline Trippel, Zhijing Jin, and Mrinmaya Sachan. 2024. Nl2fol: translating natural language to first-order logic for logical fallacy detection. arXiv preprint arXiv:2405.02318.
  20. Junxian Li, Beining Xu, Simin Chen, Jiatong Li, Jingdi Lei, Haodong Zhao, and Di Zhang. 2025b. Iag: Input-aware backdoor attack on vlm-based visual grounding. arXiv preprint arXiv:2508.09456. — [arXiv:2508.09456](https://arxiv.org/abs/2508.09456) find
  21. Pengyu Li, Lingling Zhang, Zhitao Gao, Yanrui Wu, Yuxuan Dong, Huan Liu, Bifan Wei, and Jun Liu. 2026b. AGT AO : Robust and stabilized llm unlearning via adversarial gating training with adaptive orthogonality. Preprint, arXiv:2602.01703. — [arXiv:2602.01703](https://arxiv.org/abs/2602.01703) find
  22. Yuchen Li, Rui Kong, Xinran Chen, Chengzhe Zhang, Jiamin Chen, Cheng Deng, Xinyu Ma, Haojie Zhang, Tianhao Peng, Hengyi Cai, and 1 others. 2026c. Probe-and-fetch: Dynamic kv cache pruning for accelerated long-context inference in web-scale ai search. In Proceedings of the ACM Web Conference 2026, pa… find
  23. Yuchen Li, Rui Kong, Zhonghao Lyu, Qiyang Li, Xinran Chen, Hengyi Cai, Lingyong Yan, Shuaiqiang Wang, Jiashu Zhao, Guangxu Zhu, and 1 others. 2026d. Flexspec: Frozen drafts meet evolving targets in edge-cloud collaborative llm speculative decoding. IEEE Transactions on Mobile Computing. find
  24. Yuchen Li, Zhonghao Lyu, Yongqi Zhang, Hao Zhang, Tianhao Peng, Haoyi Xiong, Shuaiqiang Wang, Linghe Kong, Guihai Chen, and Dawei Yin. 2025c. S3prank: Towards satisfaction-oriented learning to rank with semi-supervised pre-training. IEEE Transactions on Knowledge and Data Engineering. find
  25. Yuchen Li, Hao Zhang, Yongqi Zhang, Hengyi Cai, Mingxin Cai, Shuaiqiang Wang, Haoyi Xiong, Linghe Kong, Dawei Yin, and Lei Chen. 2025d. Rankexpert: A mixture of textual-and-behavioral experts for multi-objective learning-to-rank in web search. In Proceedings of the 31st ACM SIGKDD Conference on Know… find
  26. Zihao Li, Yucheng Shi, Zirui Liu, Fan Yang, Ninghao Liu, and Mengnan Du. 2024. Quantifying multilingual performance of large language models across languages. CoRR. find
  27. Jinu Lee, Qi Liu, Runzhi Ma, Vincent Han, Ziqi Wang, Heng Ji, and Julia Hockenmaier. 2025. Entailment-preserving first-order logic representations in natural language entailment. arXiv preprint arXiv:2502.16757. find
  28. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  29. Jiahui Li, Yongchang Hao, Haoyu Xu, Xing Wang, and Yu Hong. 2025a. Exploiting the index gradients for optimization-based jailbreaking on large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4535–4547, Abu Dhabi, UAE. Association for Computati… find
  30. Qidong Liu, Xian Wu, Wanyu Wang, Yejing Wang, Yuanshao Zhu, Xiangyu Zhao, Feng Tian, and Yefeng Zheng. 2025. Llmemb: Large language model can be a good embedding generator for sequential recommendation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12183– 12191. find
  31. Junxian Li, Tu Lan, Haozhen Tan, Yan Meng, and Haojin Zhu. 2026a. Slowba: An efficiency backdoor attack towards vlm-based gui agents. arXiv preprint arXiv:2603.08316. — [arXiv:2603.08316](https://arxiv.org/abs/2603.08316) find
  32. Qidong Liu, Xian Wu, Xiangyu Zhao, Yuanshao Zhu, Zijian Zhang, Feng Tian, and Yefeng Zheng. 2024b. Large language model distilling medication recommendation model. arXiv preprint arXiv:2402.02803. — [arXiv:2402.02803](https://arxiv.org/abs/2402.02803) find
  33. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024c. Autodan: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations.
  34. Yi Liu, Gelei Deng, Zhengzi Xu, Yuekang Li, Yaowen Zheng, Ying Zhang, Lida Zhao, Tianwei Zhang, Kailong Wang, and Yang Liu. 2023. Jailbreaking chatgpt via prompt engineering: An empirical study. arXiv preprint arXiv:2305.13860.
  35. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024d. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1831–1847. find
  36. AI @ Meta Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  37. Xuantao Lu, Jingping Liu, Zhouhong Gu, Hanwen Tong, Chenhao Xie, Junyang Huang, Yanghua Xiao, and Wenguang Wang. 2022. Parsing natural language into propositional and first-order logic with dual reinforcement learning. In Proceedings of the 29th International Conference on Computational Linguistics,… find
  38. Bill MacCartney and Christopher D Manning. 2014. Natural logic and natural language inference. In Computing Meaning: Volume 4, pages 129–147. Springer. find
  39. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, and 1 others. 2024. Harmbench: a standardized evaluation framework for automated red teaming and robust refusal. In Proceedings of the 41st International Conference on Machine L…
  40. OpenAI. 2023. Gpt-3.5. https://openai.com. Accessed: April 25, 2025. ↗
  41. Jingyu Peng, Hongyu Chen, Jiancheng Dong, Maolin Wang, Wenxi Li, Yuchen Li, Kai Zhang, and Xiangyu Zhao. 2026. Mosaic: Composable safety alignment with modular control tokens. arXiv preprint arXiv:2603.16210. — [arXiv:2603.16210](https://arxiv.org/abs/2603.16210) find
  42. Jingyu Peng, Maolin Wang, Xiangyu Zhao, Kai Zhang, Wanyu Wang, Pengyue Jia, Qidong Liu, Ruocheng Guo, and Qi Liu. 2025. Stepwise reasoning disruption attack of LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5040– 5058,…
  43. William C Purdy. 1991. A logic for natural language. Notre Dame Journal of Formal Logic, 32(3):409– 425. find
  44. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685.
  45. Hrituraj Singh, Milan Aggrawal, and Balaji Krishnamurthy. 2020. Exploring neural models for parsing natural language into first-order logic. arXiv preprint arXiv:2002.06544. — [arXiv:2002.06544](https://arxiv.org/abs/2002.06544) find
  46. Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11). find
  47. Hanbing Wang, Xiaorui Liu, Wenqi Fan, Xiangyu Zhao, Venkataramana Kini, Devendra Pratap Yadav, Fei Wang, Zhen Wen, and Hui Liu. 2025. Rethinking large language model architectures for sequential recommendations. In Proceedings of the 14th International Joint Conference on Natural Language Processing… find
  48. Yuhao Wang, Xiangyu Zhao, Bo Chen, Qidong Liu, Huifeng Guo, Huanshuo Liu, Yichao Wang, Rui Zhang, and Ruiming Tang. 2023. Plate: A prompt-enhanced paradigm for multi-scenario recommendations. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Ret… find
  49. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  50. Yoad Winter. 2016. Elements of formal semantics: An introduction to the mathematical theory of meaning in natural language. Edinburgh University Press. find
  51. Tianle Xia, Liang Ding, Guojia Wan, Yibing Zhan, Bo Du, and Dacheng Tao. 2025. Improving complex reasoning over knowledge graph with logic-aware curriculum tuning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 12881–12889. find
  52. Yongli Xiang, Ziming Hong, Zhaoqing Wang, Xiangyu Zhao, Bo Han, and Tongliang Liu. 2026. When safety collides: Resolving multi-category harmful conflicts in text-to-image diffusion via adaptive safety guidance. Preprint, arXiv:2602.20880. — [arXiv:2602.20880](https://arxiv.org/abs/2602.20880) find
  53. Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. 2023. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12):1486–1496. find
  54. Yuan Xin, Dingfan Chen, Linyi Yang, Michael Backes, and Xiao Zhang. 2025. Jailbreaking attacks vs. content safety filters: How far are we in the llm safety arms race? arXiv preprint arXiv:2512.24044.
  55. Siheng Xiong, Ali Payani, Ehsan Shareghi, Faramarz Fekri, and 1 others. 2024. Harnessing the power of large language models for natural language to first-order logic translation. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages… find
  56. Derong Xu, Xinhang Li, Ziheng Zhang, Zhenxi Lin, Zhihong Zhu, Zhi Zheng, Xian Wu, Xiangyu Zhao, Tong Xu, and Enhong Chen. 2025. Harnessing large language models for knowledge graph question answering via adaptive multi-aspect retrieval-augmentation. In Proceedings of the AAAI Conference on Artificia… find
  57. Derong Xu, Ziheng Zhang, Zhihong Zhu, Zhenxi Lin, Qidong Liu, Xian Wu, Tong Xu, Wanyu Wang, Yuyang Ye, Xiangyu Zhao, and 1 others. 2024. Editing factual knowledge and explanatory ability of medical large language models. In Proceedings of the 33rd ACM international conference on information and know… find
  58. Liu, and Peng Jiang. 2025b. Llm-powered user simulator for recommender system. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 13339–13347. find
  59. Yuqi Zhou, Lin Lu, Ryan Sun, Pan Zhou, and Lichao Sun. 2024a. Virtual context enhancing jailbreak attacks with special token injection. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11843–11857, Miami, Florida, USA. Association for Computational Linguistics. find
  60. Zhenhong Zhou, Haiyang Yu, Xinghua Zhang, Rongwu Xu, Fei Huang, and Yongbin Li. 2024b. How alignment and jailbreak work: Explain LLM safety through intermediate hidden states. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2461–2488, Miami, Florida, USA. Association… find
  61. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  62. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. find
  63. Sibo Yi, Yule Liu, Zhen Sun, Tianshuo Cong, Xinlei He, Jiaxing Song, Ke Xu, and Qi Li. 2024. Jailbreak attacks and defenses against large language models: A survey. arXiv preprint arXiv:2407.04295.
  64. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446.
  65. Jiahao Yu, Haozheng Luo, Jerry Yao-Chieh Hu, Wenbo Guo, Han Liu, and Xinyu Xing. 2024. Enhancing jailbreak attack against large language models through silent tokens. arXiv preprint arXiv:2405.20653. — [arXiv:2405.20653](https://arxiv.org/abs/2405.20653) find
  66. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. In International Conference on Learning Representations.
  67. Chao Zhang, Haoxin Zhang, Shiwei Wu, Di Wu, Tong Xu, Xiangyu Zhao, Yan Gao, Yao Hu, and Enhong Chen. 2025a. Notellm-2: Multimodal large representation models for recommendation. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V. 1, pages 2815–2826. find
  68. Jiajun Zhang, Zeyu Cui, Jiaxi Yang, Lei Zhang, Yuheng Jing, Zeyao Ma, Tianyi Bai, Zilei Wang, Qiang Liu, Liang Wang, Binyuan Hui, and Junyang Lin. 2026. From completion to editing: Unlocking context-aware code infilling via search-and-replace instruction tuning. Preprint, arXiv:2601.13384.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/peng-2026-logic-jailbreak-efficiently-unlocking-llm/