Elamur Research⌕
☰
home / papers / 2026 / SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents

SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents

Paper · 2026 · Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed et al.

Summary

The paper looks at the safety of LLM search agents, which repeatedly write queries, retrieve documents, and reason over them. On red-teaming prompts, these agents produce harmful content more often than their underlying base models. Retrieved material, such as crime descriptions, can pull an agent toward unsafe answers, and finetuning for QA accuracy alone makes this worse. The authors link harm to two things: the decision to search at all, and issuing unsafe queries.

They propose SafeSearch, a multi-objective reinforcement learning framework. It trains on a shuffled mix of QA data, rewarded by exact match, and red-teaming data, rewarded by an LLM judge for safety and for helpfulness that stays within policy. A further query-level reward penalizes unsafe search queries and rewards benign ones. Applied to Qwen models, SafeSearch lowers harmful output rates by over 70% across three red-teaming sets. Its QA accuracy stays on par with a utility-only agent, and it gives constructive alternatives instead of flat refusals. This suggests safety alignment for agents need not cost utility.

Key points

  • Baseline risk: Search agents reached up to 2× the harmful output rate of base LLMs, and up to 3× after utility-oriented RL finetuning.
  • Models and utility benchmarks: The backbones are Qwen-2.5-3B-Instruct and Qwen-2.5-7B-Instruct. Utility is measured on TriviaQA, HotpotQA, and Bamboogle, using a local BM25 retriever over Wikipedia-2018.
  • Reward design:
    • KL-regularized RL over mixed utility and safety data.
    • An LLM judge gives a binary safety label and a 1–4 helpfulness score, and the two are combined as safety-weighted helpfulness.
    • The query-level shaping term is weighted by λ_q.
  • Results and ablations:
    • Harmfulness drops by over 70% on average, and by up to 90% compared with a utility-finetuned agent.
    • The query-level reward improves both safety and helpfulness.
    • Filtering only queries or only documents gives marginal gains, though controlling queries works better than controlling documents.
    • Results are fairly robust to λ_q.
  • Code: github.com/amazon-science/SafeSearch.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States209 works · 71,394 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksInformation Systems · 1,730 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 587 works

Cite

Cite as Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang and Joo-Kyung Kim. 2026. SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.146

@inproceedings{zhan2026safesearch,
  title = {{SafeSearch: Do Not Trade Safety for Utility in LLM Search Agents}},
  author = {Qiusi Zhan, Angeline Budiman-Chan, Abdelrahman Zayed, Xingzhi Guo, Daniel Kang and Joo-Kyung Kim},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.146},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.146}
}

References

42 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bang An, Shiyue Zhang, and Mark Dredze. 2025. RAG llms are not safer: A safety analysis of retrieval-augmented generation for large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technolog… find
  2. Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2024. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In ICLR.
  3. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022a. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  4. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022b. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  5. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, and 1 others. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machin… find
  6. Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, and Mengdi Wang. 2024. MaxMin-RLHF: Alignment with diverse human preferences. In ICML. find
  7. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2025. Or-bench: An over-refusal benchmark for large language models. In Forty-second International Conference on Machine Learning, ICML 2025, Vancouver, BC, Canada, July 13-19, 2025. OpenReview.net.
  8. Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. 2024. Safe RLHF: Safe reinforcement learning from human feedback. In ICLR.
  9. DeepSeek-AI. 2025. Deepseek-v3.2: Pushing the frontier of open large language models.
  10. Tiansheng Huang, Sihao Hu, Fatih Ilhan, Selim Furkan Tekin, Zachary Yahn, Yichang Xu, and Ling Liu. 2025. Safety tax: Safety alignment makes your large reasoning models less reasonable. arXiv preprint arXiv:2503.00555. — [arXiv:2503.00555](https://arxiv.org/abs/2503.00555) find
  11. Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, and Dongsheng Ding. 2024. One-shot safety alignment for large language models via optimal dualization. Advances in Neural Information Processing Systems, 37:84350–84383. find
  12. Fengqing Jiang, Zhangchen Xu, Yuetai Li, Luyao Niu, Zhen Xiang, Bo Li, Bill Yuchen Lin, and Radha Poovendran. 2025. Safechain: Safety of language models with long chain-of-thought reasoning capabilities. In ACL Findings. find
  13. Liwei Jiang, Kavel Rao, Seungju Han, Allyson Ettinger, Faeze Brahman, Sachin Kumar, Niloofar Mireshghallah, Ximing Lu, Maarten Sap, Yejin Choi, and 1 others. 2024. Wildteaming at scale: From in-the-wild jailbreaks to (adversarially) safer language models. Advances in Neural Information Processing Sy…
  14. Bowen Jin, Jinsung Yoon, Priyanka Kargupta, Sercan O Arik, and Jiawei Han. 2025a. An empirical study on reinforcement learning for reasoning-search interleaved llm agents. arXiv preprint arXiv:2505.15117. — [arXiv:2505.15117](https://arxiv.org/abs/2505.15117) find
  15. Bowen Jin, Hansi Zeng, Zhenrui Yue, Jinsung Yoon, Sercan Arik, Dong Wang, Hamed Zamani, and Jiawei Han. 2025b. Search-R1: Training llms to reason and leverage search engines with reinforcement learning. In COLM.
  16. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In ACL.
  17. Maurice G Kendall. 1938. A new measure of rank correlation. Biometrika, 30(1-2):81–93. find
  18. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural information process…
  19. Steve Li and Leonard Tang. 2024. Redteaming Resistance Benchmark. https://github.com/ haizelabs/redteaming-resistance-benchmark. GitHub repository, version v1.0.0 (commit 26dc059). ↗
  20. Xiaoxi Li, Guanting Dong, Jiajie Jin, Yuyao Zhang, Yujia Zhou, Yutao Zhu, Peitian Zhang, and Zhicheng Dou. 2025. Search-o1: Agentic search-enhanced large reasoning models. arXiv preprint arXiv:2501.05366.
  21. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. 2024. Rule based rewards for language model safety. Advances in Neural Information Processing Systems, 37:108877–108901. find
  22. OpenAI. 2025. gpt-oss-120b gpt-oss-20b model card. Preprint, arXiv:2508.10925.
  23. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  24. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2023. Measuring and narrowing the compositionality gap in language models. In EMNLP Findings. find
  25. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  26. Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728–53741.
  27. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36:68539–68551.
  28. John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2015. High-dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.02438. find
  29. John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347. find
  30. Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, and 1 others. 2024. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300.
  31. Huatong Song, Jinhao Jiang, Yingqian Min, Jie Chen, Zhipeng Chen, Wayne Xin Zhao, Lei Fang, and Ji-Rong Wen. 2025. R1-searcher: Incentivizing the search capability in llms via reinforcement learning. arXiv preprint arXiv:2503.05592.
  32. Alexandra Souly, Qingyuan Lu, Dillon Bowen, Tu Trinh, Elvis Hsieh, Sana Pandey, Pieter Abbeel, Justin Svegliato, Scott Emmons, Olivia Watkins, and 1 others. 2024. A strongreject for empty jailbreaks. Advances in Neural Information Processing Systems, 37:125416–125440. find
  33. Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Ameya Sunil Mahabaleshwarkar, Gerald Shen, Jiaqi Zeng, Zijia Chen, Yoshi Suhara, and 1 others. 2024. Mistral-nemo-minitron-8b-instruct. Technical report, NVIDIA / NVlabs. Technical Report, “LLM Pruning and Distillat…
  34. Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. 2025. Equilibrate rlhf: Towards balancing helpfulness-safety trade-off in large language models. arXiv preprint arXiv:2502.11555. — [arXiv:2502.11555](https://arxiv.org/abs/2502.11555) find
  35. Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In ACL.
  36. Akifumi Wachi, Thien Tran, Rei Sato, Takumi Tanabe, and Youhei Akimoto. 2024. Stepwise alignment for constrained language model policy optimization. Advances in Neural Information Processing Systems, 37:104471–104520. find
  37. Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. 2024. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP.
  38. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In EMNLP.
  39. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. React: Synergizing reasoning and acting in language models. In International Conference on Learning Representations (ICLR).
  40. Cheng Yu, Benedikt Stroebl, Diyi Yang, and Orestis Papakyriakopoulos. 2025. Safety devolution in ai agents. arXiv preprint arXiv:2505.14215. find
  41. Yuan Yuan, Tina Sriskandarajah, Anna-Luisa Brakman, Alec Helyar, Alex Beutel, Andrea Vallone, and Saachi Jain. 2025. From hard refusals to safe-completions: Toward output-centric safety training. arXiv preprint arXiv:2508.09224. — [arXiv:2508.09224](https://arxiv.org/abs/2508.09224) find
  42. Yuxiang Zheng, Dayuan Fu, Xiangkun Hu, Xiaojie Cai, Lyumanshan Ye, Pengrui Lu, and Pengfei Liu. 2025. Deepresearcher: Scaling deep research via reinforcement learning in real-world environments. arXiv preprint arXiv:2504.03160.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhan-2026-safesearch-do-not-trade-safety/