Elamur Research⌕
☰
home / papers / 2026 / Reasoning Hijacking: The Fragility of Reasoning Alignment in Large Language Models

Reasoning Hijacking: The Fragility of Reasoning Alignment in Large Language Models

Paper · 2026 · Yuansen Liu, Yixuan Tang and Anthony Kum Hoe Tung

Summary

Most work on prompt-injection security targets Goal Hijacking, where injected text tries to replace an application's task. This paper describes a different threat, Reasoning Hijacking, in which the task stays the same but the model's decision logic is corrupted. The authors define it with three conditions: the instruction is unchanged, no injected text demands a specific label or a new task, and the resulting label no longer matches what the model would have predicted on the untampered input.

They implement this as the Criteria Attack. An attacker model mines refutable decision criteria from in-distribution data and turns them into a structured reasoning scaffold appended to untrusted input. For example, it might falsely assert that only emails with links count as spam.

Experiments covered spam, toxic comment and negative review detection across several backbones. Strong models often followed the injected shortcuts instead of analyzing meaning. The attack stayed effective under defenses built to catch goal deviation, such as StruQ and SecAlign. However, attention-based monitoring still separated attacked from clean inputs. The work matters because guarding the model's intent alone leaves classification pipelines exposed.

Key points

  • Datasets: Enron (spam), Wikipedia toxicity (toxic comments) and IMDb (reviews). The attack injects through the data channel in a black-box setting.
  • Structured-query and safety-alignment defenses (StruQ, SecAlign) failed to stop the attack, while Goal Hijacking baselines were suppressed by them.
  • The Focus Score (Hung et al., 2025) fell under Criteria Attack on all three tasks. Its distributions stayed separable from clean data, though closer to clean than with goal-hijacking attacks.
  • Limitations: only classification tasks were tested, success is measured by ASR, and results depend on the attacker model and how well the criteria bank matches the victim task.
  • Code and data: github.com/Yuan-Hou/criteria_attack (ACL 2026, NUS).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓National University of SingaporeSingapore · Singapore207 works · 39,576 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,894 worksReasoning · 1,685 works

Cite

Cite as Yuansen Liu, Yixuan Tang and Anthony Kum Hoe Tung. 2026. Reasoning Hijacking: The Fragility of Reasoning Alignment in Large Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1698

@inproceedings{liu2026reasoning,
  title = {{Reasoning Hijacking: The Fragility of Reasoning Alignment in Large Language Models}},
  author = {Yuansen Liu, Yixuan Tang and Anthony Kum Hoe Tung},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1698},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1698}
}

References

43 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Shan Chen, Mingye Gao, Kuleen Sasse, Thomas Hartvigsen, Brian Anthony, Lizhou Fan, Hugo Aerts, Jack Gallifant, and Danielle S Bitterman. 2025a. When helpfulness backfires: Llms and the risk of false medical information due to sycophantic behavior. npj Digital Medicine, 8(1):605. find
  2. Sizhe Chen, Julien Piet, Chawin Sitawarin, and David Wagner. 2025b. Struq: defending against prompt injection with structured queries. In Proceedings of the 34th USENIX Conference on Security Symposium, SEC ’25, USA. USENIX Association. find
  3. Sizhe Chen, Yizhu Wang, Nicholas Carlini, Chawin Sitawarin, and David Wagner. 2025c. Defending against prompt injection with a few defensivetokens. arXiv preprint arXiv:2507.07974. — [arXiv:2507.07974](https://arxiv.org/abs/2507.07974) find
  4. Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, and Chuan Guo. 2024. Secalign: Defending against prompt injection with preference optimization. pages 2833– 2847. find
  5. Yulin Chen, Haoran Li, Yuexin Li, Yue Liu, Yangqiu Song, and Bryan Hooi. 2025d. Topicattack: An indirect prompt injection attack via topic transition. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 7338–7356. Association for Computational Linguistics… find
  6. Zheng Chen and Buhui Yao. 2024. Pseudo-conversation injection for llm goal hijacking. arXiv preprint arXiv:2410.23678. — [arXiv:2410.23678](https://arxiv.org/abs/2410.23678) find
  7. DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, and 181 others. 2025. Deepseek-v3 technical report. Preprint, arXiv:2… find
  8. Kai Greshake, Sahar Abdelnabi, Shailesh Mishra, Christoph Endres, Thorsten Holz, and Mario Fritz. 2023. Not what you’ve signed up for: Compromising real-world llm-integrated applications with indirect prompt injection. In Proceedings of the 16th ACM Workshop on Artificial Intelligence and Security,…
  9. Yihao Huang, Chong Wang, Xiaojun Jia, Qing Guo, Felix Juefei-Xu, Jian Zhang, Geguang Pu, and Yang Liu. 2025. Efficient universal goal hijacking with semantics-guided prompt organization. pages 5796– 5816. find
  10. Kuo-Han Hung, Ching-Yun Ko, Ambrish Rawat, I-Hsin Chung, Winston H. Hsu, and Pin-Yu Chen. 2025. Attention tracker: Detecting prompt injection attacks in llms. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 2309–2322. Association for Computational Linguistics. find
  11. Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix,… find
  12. Sam Johnson, Viet Pham, and Thai Le. 2025. Manipulating llm web agents with indirect prompt injection attack via html accessibility tree. arXiv preprint arXiv:2507.14799. — [arXiv:2507.14799](https://arxiv.org/abs/2507.14799) find
  13. Bryan Klimt and Yiming Yang. 2004. Introducing the enron corpus. In CEAS, volume 45, pages 92–96. find
  14. Martin Kuo, Jianyi Zhang, Aolin Ding, Qinsi Wang, Louis DiValentin, Yujia Bao, Wei Wei, Hai Li, and Yiran Chen. 2025. H-cot: Hijacking the chain-of-thought safety reasoning mechanism to jailbreak large reasoning models, including openai o1/o3, deepseek-r1, and gemini 2.0 flash thinking. arXiv prepri… — [arXiv:2502.12893](https://arxiv.org/abs/2502.12893) find
  15. Xitao Li, Haijun Wang, Jiang Wu, and Ting Liu. 2025. Separator injection attack: Uncovering dialogue biases in large language models caused by role separators. arXiv preprint arXiv:2504.05689. — [arXiv:2504.05689](https://arxiv.org/abs/2504.05689) find
  16. Zhuotao Lian, Weiyu Wang, Qingkui Zeng, Toru Nakanishi, Teruaki Kitasuka, and Chunhua Su. 2025. Prompt-in-content attacks: Exploiting uploaded inputs to hijack llm behavior. arXiv preprint arXiv:2508.19287. — [arXiv:2508.19287](https://arxiv.org/abs/2508.19287) find
  17. Yi Liu, Gelei Deng, Yuekang Li, Kailong Wang, Zihao Wang, Xiaofeng Wang, Tianwei Zhang, Yepang Liu, Haoyu Wang, Yan Zheng, Leo Yu Zhang, and Yang Liu. 2023. Prompt injection attack against llm-integrated applications. arXiv preprint arXiv:2306.05499.
  18. Yupei Liu, Yuqi Jia, Runpeng Geng, Jinyuan Jia, and Neil Zhenqiang Gong. 2024. Formalizing and benchmarking prompt injection attacks and defenses. In 33rd USENIX Security Symposium (USENIX Security 24), pages 1831–1847, Philadelphia, PA. USENIX Association.
  19. Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142–150.
  20. Lars Malmqvist. 2025. Sycophancy in Large Language Models: Causes and Mitigations, pages 61–74. Springer Nature Switzerland.
  21. Erick Mendez Guzman, Viktor Schlegel, and Riza Batista-Navarro. 2024. From outputs to insights: a survey of rationalization approaches for explainable text classification. Frontiers in Artificial Intelligence, 7:1363531. find
  22. Mistral AI. 2025. Mistral small 3.2. https://docs.mistral.ai/models/ mistral-small-3-2-25-06. Model documentation page (dated June 20, 2025). Accessed: 2025-12-30. ↗
  23. OpenAI. 2025. gpt-oss-120b & gpt-oss-20b model card.
  24. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, and 262 others.… — GPT-4 Technical Report
  25. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022. Tra… — Training language models to follow instructions with human feedback
  26. Fábio Perez and Ian Ribeiro. 2022. Ignore previous prompt: Attack techniques for language models. arXiv preprint arXiv:2211.09527.
  27. Vyas Raina, Adian Liusie, and Mark Gales. 2024. Is llm-as-a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment. arXiv preprint arXiv:2402.14016, pages 7499–7517.
  28. Sander Schulhoff, M Ilie, N Balepur, K Kahadze, A Liu, C Si, Y Li, A Gupta, H Han, and 1 others. 2024. The prompt report: A systematic survey of prompting techniques. arXiv preprint arXiv:2406.06608.
  29. Mrinank Sharma, Meg Tong, Tomasz Korbak, David Duvenaud, Amanda Askell, Samuel R. Bowman, Newton Cheng, Esin Durmus, Zac Hatfield-Dodds, Scott R. Johnston, Shauna Kravec, Timothy Maxwell, Sam McCandlish, Kamal Ndousse, Oliver Rausch, Nicholas Schiefer, Da Yan, Miranda Zhang, and Ethan Perez. 2023. T… — Towards Understanding Sycophancy in Language Models
  30. Jiawen Shi, Zenghui Yuan, Yinuo Liu, Yue Huang, Pan Zhou, Lichao Sun, and Neil Zhenqiang Gong. 2024. Optimization-based prompt injection attack to llm-asa-judge. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS ’24, pages 660–674. ACM. find
  31. Yixuan Tang, Hwee Tou Ng, and Anthony Tung. 2021. Do multi-hop question answering systems know how to answer the single-hop sub-questions? In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 3244–3249, Online. Association… find
  32. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. find
  33. Miles Turpin, Julian Michael, Ethan Perez, and Samuel R. Bowman. 2023. Language models don’t always say what they think: unfaithful explanations in chain-of-thought prompting. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS ’23, Red Hook, NY, USA. C…
  34. Betty Van Aken, Julian Risch, Ralf Krestel, and Alexander Löser. 2018. Challenges for toxic comment classification: An in-depth error analysis. In Proceedings of the 2nd Workshop on Abusive Language Online (ALW2). Association for Computational Linguistics.
  35. Cheng Wang, Yue Liu, Baolong Li, Duzhen Zhang, Zhongzhi Li, and Junfeng Fang. 2025. Safety in large reasoning models: A survey. arXiv preprint arXiv:2504.17704. find
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35, 24824-24837.
  37. Sarah Wiegreffe, Ana Marasović, and Noah A Smith. 2021. Measuring association between labels and free-text rationales. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10266–10284.
  38. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  39. Yang Yao, Xuan Tong, Ruofan Wang, Yixu Wang, Lu-jundong Li, Liang Liu, Yan Teng, and Yingchun Wang. 2025. A mousetrap: Fooling large reasoning models for jailbreak with chain of iterative chaos. In Findings of the Association for Computational Linguistics: ACL 2025, pages 7837–7855. find
  40. Jingwei Yi, Yueqi Xie, Bin Zhu, Emre Kiciman, Guangzhong Sun, Xing Xie, and Fangzhao Wu. 2025. Benchmarking and defending against indirect prompt injection attacks on large language models. In Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining V.1, KDD ’25, pages 18… find
  41. Itay Yona, Amir Sarid, Michael Karasik, and Yossi Gandelsman. 2025. In-context representation hijacking. arXiv preprint arXiv:2512.03771.
  42. Gejian Zhao, Hanzhou Wu, Xinpeng Zhang, and Athanasios V Vasilakos. 2025. Shadowcot: Cognitive hijacking for stealthy reasoning backdoors in llms. arXiv preprint arXiv:2504.05605. — [arXiv:2504.05605](https://arxiv.org/abs/2504.05605) find
  43. Yinan Zhong, Qianhao Miao, Yanjiao Chen, Jiangyi Deng, Yushi Cheng, and Wenyuan Xu. 2025. Attention is all you need to defend against indirect prompt injection attacks in llms. arXiv preprint arXiv:2512.08417. — [arXiv:2512.08417](https://arxiv.org/abs/2512.08417) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/liu-2026-reasoning-hijacking-fragility-reasoning-alignment/