Elamur Research⌕
☰
home / papers / 2026 / LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks

LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks

Paper · 2026 · Jiayong Wan, Jiawei Chen, Zhaoxia Yin et al.

Summary

LLM agents that act in an environment and learn from its feedback can fall into in-context reward hacking (ICRH). While pursuing a proxy goal, such as tweet engagement or task completion, they gradually produce toxic content or unsafe actions, for example trying to get around file protection. Jailbreak-oriented defenses do not address this, because the risk comes from the agent's own over-optimization rather than from hostile inputs.

The authors propose LCO, a prompt-based framework that needs no fine-tuning and works with black-box and white-box models. It has two modules:

  • Self-thought: before acting, the model lists task-specific safety constraints, which are then appended to the user's objective.
  • Guided evolutionary exploration: this adapts genetic algorithms to natural language. The LLM scores candidate outputs or actions for fitness and performs crossover and mutation on them, which keeps the search inside a safe solution space.

In output-refinement and policy-refinement experiments, LCO clearly reduces harmful side effects. The cost is only a small drop in helpfulness. This makes it a practical safeguard for deployed agents.

Key points

  • On the tweet engagement optimization task with GPT-4, LCO cuts the Toxicity Growth Rate (TGR) by 39%. On the policy optimization benchmark (ToolEmu-style tool use), it lowers the ICRH Occurrence Rate (IOR) by 15.23%.
  • In the ablation, the two modules complement each other. Evolutionary search without self-thought constraints gives only limited gains, for example a TGR of 55.69% on GPT-3.5.
  • Larger population sizes reduce IOR further but quickly increase token use and the number of calls. Pos=3 gives the best balance. A risk-adaptive routing strategy lowers overhead for low-risk queries.
  • Robustness test: constraints were generated by Qwen2.5-72B-Instruct, LLaMA3.1-405B-Instruct, or GPT-3.5, with a GPT-3.5 agent in all cases. Every generator reduced TGR compared with a no-constraint baseline.
  • Code: github.com/Califoni/LCO_for_ICRH. Stated limitations are extra inference cost, dependence on the base LLM's reasoning ability, and no empirical tests in multi-task settings.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Tsinghua UniversityBeijing · China333 works · 87,019 citations🎓East China Normal UniversityShanghai · China50 works · 2,174 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Jiayong Wan, Jiawei Chen, Zhaoxia Yin, Liu Shuyuan and Hang Su. 2026. LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1390

@inproceedings{wan2026lco,
  title = {{LCO: LLM-based Constraint Optimization for Safer Agentic LLMs in Real-world Tasks}},
  author = {Jiayong Wan, Jiawei Chen, Zhaoxia Yin, Liu Shuyuan and Hang Su},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1390},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1390}
}

References

40 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Gabriel Alon and Michael Kamfonas. 2023. Detecting language model attacks with perplexity. Preprint, arXiv:2308.14132.
  2. Google Perspective API. 2025. About the perspective api. https://developers.perspectiveapi.com/ s/?language=en_US. Accessed: 2025-05-14. ↗
  3. Jiawei Chen, Xiao Yang, Zhengwei Fang, Yu Tian, Yinpeng Dong, Zhaoxia Yin, and Hang Su. 2025a. Auto-Breach: Universal and adaptive jailbreaking with efficient wordplay-guided optimization via multi-LLMs. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6777–6798, Albuq… find
  4. Jiawei Chen, Yang Yang, Chao Yu, Yu Tian, Zhi Cao, Xue Yang, Linghao Li, Hang Su, and Zhaoxia Yin. 2025b. Red teaming large reasoning models. arXiv preprint arXiv:2512.00412.
  5. Ke-Li Chiu, Annie Collins, and Rohan Alexander. 2022. Detecting hate speech with gpt-3. Preprint, arXiv:2103.12407. — [arXiv:2103.12407](https://arxiv.org/abs/2103.12407) find
  6. David E Goldberg. 2013. Genetic algorithms. pearson education India. find
  7. Janis Goldzycher and Gerold Schneider. 2022. Hypothesis engineering for zero-shot hate speech detection. CoRR, abs/2210.00910. find
  8. Zhengmian Hu, Gang Wu, Saayan Mitra, Ruiyi Zhang, Tong Sun, Heng Huang, and Viswanathan Swaminathan. 2025. Token-level adversarial prompt detection based on perplexity measures and contextual information. In ICLR 2025 Workshop on Building Trust in Language Models and Applications. find
  9. Xiao. 2025. AGrail: A lifelong agent guardrail with effective and adaptive safety detection. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8104–8139, Miami, Florida. Association for Computational Linguistics.
  10. Xu Huang, Weiwen Liu, Xiaolong Chen, Xingmei Wang, Hao Wang, Defu Lian, Yasheng Wang, Ruiming Tang, and Enhong Chen. 2024. Understanding the planning of llm agents: A survey. Preprint, arXiv:2402.02716. find
  11. Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. In Proceedings of the Thirty-Seventh AAAI Conference on Artificial Intelligence and Thirty-F…
  12. Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. 2023. Baseline defenses for adversarial attacks against aligned language models. Preprint, arXiv:2309.00614.
  13. Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association fo…
  14. Jinhwa Kim, Ali Derakhshan, and Ian G. Harris. 2023. Robust safety classifier for large language models: Adversarial prompt shield. CoRR, abs/2311.00172. — [arXiv:2311.00172](https://arxiv.org/abs/2311.00172) find
  15. Raz Lapid, Ron Langberg, and Moshe Sipper. 2024. Open sesame! universal black box jailbreaking of large language models. Preprint, arXiv:2309.01446. — [arXiv:2309.01446](https://arxiv.org/abs/2309.01446) find
  16. Shuyuan Liu, Jiawei Chen, Shouwei Ruan, Hang Su, and Zhaoxia Yin. 2024a. Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models. In Proceedings of the 32nd ACM international conference on multimedia, pages 8120–8128. find
  17. Shuyuan Liu, Jiawei Chen, Shouwei Ruan, Hang Su, and Zhaoxia Yin. 2024b. Exploring the robustness of decision-level through adversarial attacks on llm-based embodied models. In Proceedings of the 32nd ACM International Conference on Multimedia, MM ’24, page 8120–8128, New York, NY, USA. Association… find
  18. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2024c. AutoDAN: Generating stealthy jailbreak prompts on aligned large language models. In The Twelfth International Conference on Learning Representations.
  19. Zichuan Liu, Zefan Wang, Linjie Xu, Jinyu Wang, Lei Song, Tianchun Wang, Chunlin Chen, Wei Cheng, and Jiang Bian. 2024d. Protecting your LLMs with information bottleneck. In The Thirty-eighth Annual Conference on Neural Information Processing Systems. find
  20. Weidi Luo, Shenghong Dai, Xiaogeng Liu, Suman Banerjee, Huan Sun, Muhao Chen, and Chaowei Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pasunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Celikyilmaz, Edouard Grave, Yann LeCun, and Thom… — Augmented Language Models: a Survey
  21. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser-assis…
  22. OpenAI. 2023. Gpt-4 technical report. Accessed via OpenAI API.
  23. Alexander Pan, Erik Jones, Meena Jagadeesan, and Jacob Steinhardt. 2024. Feedback loops with language models drive in-context reward hacking. In Proceedings of the 41st International Conference on Machine Learning, ICML’24. JMLR.org. find
  24. Richard Yuanzhe Pang, Stephen Roller, Kyunghyun Cho, He He, and Jason Weston. 2024. Leveraging implicit feedback from deployment data in dialogue. Preprint, arXiv:2307.14117. find
  25. Joon Sung Park, Joseph O’Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, UIST ’23, New York, NY, USA. Ass… find
  26. Mansi Phute, Alec Helbling, Matthew Daniel Hull, ShengYun Peng, Sebastian Szyller, Cory Cornelius, and Duen Horng Chau. 2024. LLM self defense: By self examination, LLMs know they are being tricked. In The Second Tiny Papers Track at ICLR 2024. find
  27. Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. 2024. ToolLLM: Facilitating large language models to master 16000+ real…
  28. Yangjun Ruan, Honghua Dong, Andrew Wang, Silviu Pitis, Yongchao Zhou, Jimmy Ba, Yann Dubois, Chris J. Maddison, and Tatsunori Hashimoto. 2024. Identifying the risks of LM agents with an LM-emulated sandbox. In The Twelfth International Conference on Learning Representations. find
  29. Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. In Thirty-seventh Conference on Neural Information Processing Systems.
  30. Noah Shinn, Beck Labash, and Ashwin Gopinath. 2023. Reflexion: an autonomous agent with dynamic memory and self-reflection. CoRR, abs/2303.11366.
  31. Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Jirong Wen. 2024. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6).
  32. Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. 2024. Jailbreak and guard aligned language models with only few in-context demonstrations. Preprint, arXiv:2310.06387.
  33. Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. 2024. Autogen: Enabling next-gen LLM applications via multi-agent conversations. In First Conference on Language… find
  34. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large language models as optimizers. In The Twelfth International Conference on Learning Representations.
  35. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations.
  36. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2024. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. Preprint, arXiv:2308.06463.
  37. Yuqi Zhang, Liang Ding, Lefei Zhang, and Dacheng Tao. 2025. Intention analysis makes LLMs a good jailbreak defender. In Proceedings of the 31st International Conference on Computational Linguistics, pages 2947–2968, Abu Dhabi, UAE. Association for Computational Linguistics.
  38. Zhexin Zhang, Junxiao Yang, Pei Ke, Fei Mi, Hongning Wang, and Minlie Huang. 2024. Defending large language models against jailbreaking attacks through goal prioritization. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8865–…
  39. Aojun Zhou, Ke Wang, Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song, Mingjie Zhan, and Hongsheng Li. 2024. Solving challenging math word problems using GPT-4 code interpreter with code-based self-verification. In The Twelfth International Conference on Learning Repre…
  40. Yutao Zhu, Huaying Yuan, Shuting Wang, Jiongnan Liu, Wenhan Liu, Chenlong Deng, Haonan Chen, Zheng Liu, Zhicheng Dou, and Ji-Rong Wen. 2024. Large language models for information retrieval: A survey. Preprint, arXiv:2308.07107.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wan-2026-lco-llm-based-constraint-optimization/