Elamur Research⌕
☰
home / papers / 2025 / SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents

Paper · 2025 · Kunlun Zhu, Jiaxun Zhang, Ziheng Qi et al.

Summary

LLM-based "AI scientist" systems can now propose hypotheses, run tool-assisted experiments and draft papers. They may also comply with harmful requests or handle dangerous research topics carelessly. Most prior safety work targets single models rather than multi-agent research pipelines.

The authors build SafeScientist, an end-to-end research agent inspired by AI Scientist and Tiny Scientist. In it, expert agents debate an idea, invoke tools and write a paper. Safeguards sit at every stage:

  • a Prompt Monitor based on LLaMA-Guard
  • an Agent Collaboration Monitor
  • a Tool-Use Monitor
  • a Paper Ethic Reviewer

To evaluate such systems, they introduce SciSafetyBench. It pairs high-risk research tasks with simulated scientific tools and tool-misuse scenarios. The pipeline is also stress-tested with jailbreak-style attacks such as DAN, Base64 and Inception.

Compared with AI scientist frameworks that lack these protections, SafeScientist improves safety by roughly 35% without degrading research output. The work is an early, domain-grounded effort to make autonomous research agents refuse or responsibly manage risky science.

Key points

  • SciSafetyBench: 240 high-risk tasks spanning six scientific fields (Medicine, Biology, Chemistry, Physics, Material Science and Computer Science), organized by four sources of risk. It also includes 30 scientific tools and 120 tool-related risk tasks.
  • Overall safety gain: about 35% (34.69% in the introduction) over unprotected AI scientist frameworks.
  • Tool-Use Monitor under malicious instructions:
    • Tool Call Safety Rate rose from 5.8% to 47.5%.
    • Human Correctness Rate rose from 0% to 60.0%.
    • Benign cases also improved (43.3%→50.0% and 70.6%→75.0%).
  • Ethical reviewer: on 20 tasks per domain, refined papers scored on average 44.4% higher on the ethics rubric than the initial drafts.
  • Collaboration defense: a defender agent at the idea-summarization stage lifted idea-safety scores to roughly 4.0–4.2, e.g. 4.22 in Chemistry with Attacker 2 + Defender 2. Code and data are at github.com/ulab-uiuc/SafeScientist.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Illinois Urbana-ChampaignUrbana · United States228 works · 74,591 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 19,503 worksArtificial Intelligence · 18,805 worksSocial Sciences · 2,009 worksSafety Research · 665 works

🧭 Topics 1Research subjects and languages the work is about.Bias, fairness & safety · 590 works

Cite

Cite as Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu and Jiaxuan You. 2025. SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents. arXiv:2505.23559.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2505.23559 · arXiv 2505.23559

@misc{zhu2025safescientist,
  title = {{SafeScientist: Toward Risk-Aware Scientific Discoveries by LLM Agents}},
  author = {Kunlun Zhu, Jiaxun Zhang, Ziheng Qi, Nuoxing Shang, Zijia Liu, Peixuan Han, Yue Su, Haofei Yu and Jiaxuan You},
  year = {2025},
  doi = {10.48550/arxiv.2505.23559},
  eprint = {2505.23559},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2505.23559}
}

References

80 references, 33 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Walid Al-Zyoud, Alshaimaa M Qunies, Ayana UC Walters, and Nigel K Jalsa. 2019. Perceptions of chemical safety in laboratories. Safety, 5(2):21. find
  2. Maksym Andriushchenko, Alexandra Souly, Mateusz Dziemian, Derek Duenas, Maxwell Lin, Justin Wang, Dan Hendrycks, Andy Zou, Zico Kolter, Matt Fredrikson, and 1 others. 2024. Agentharm: A benchmark for measuring harmfulness of llm agents. arXiv preprint arXiv:2410.09024. — [arXiv:2410.09024](https://arxiv.org/abs/2410.09024) find
  3. Andy Arditi, Oscar Obeso, Aaquib Syed, Daniel Paleka, Nina Panickssery, Wes Gurnee, and Neel Nanda. 2024. Refusal in language models is mediated by a single direction. arXiv preprint arXiv:2406.11717.
  4. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, and 1 others. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862. find
  5. Yoshua Bengio, Michael Cohen, Damiano Fornasiere, Joumana Ghosn, Pietro Greiner, Matt MacDermott, Sören Mindermann, Adam Oberman, Jesse Richardson, Oliver Richardson, Marc-Antoine Rondeau, Pierre-Luc St-Charles, and David Williams-King. 2025a. Superintelligent agents pose catastrophic risks: Can sci…
  6. Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, and 1 others. 2025b. International ai safety report. arXiv preprint arXiv:2501.17805. find
  7. Xiusi Chen, Hongzhi Wen, Sreyashi Nag, Chen Luo, Qingyu Yin, Ruirui Li, Zheng Li, and Wei Wang. 2024a. Iteralign: Iterative constitutional alignment of large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human L… find
  8. Zhaorun Chen, Zhen Xiang, Chaowei Xiao, Dawn Song, and Bo Li. 2024b. Agentpoison: Red-teaming llm agents via poisoning memory or knowledge bases. Advances in Neural Information Processing Systems, 37:130185–130213. find
  9. Yuheng Cheng, Ceyao Zhang, Zhengwen Zhang, Xiangrui Meng, Sirui Hong, Wenhao Li, Zihao Wang, Zekai Wang, Feng Yin, Junhua Zhao, and 1 others. 2024. Exploring large language model based intelligent agents: Definitions, methods, and prospects. arXiv preprint arXiv:2401.03428. — [arXiv:2401.03428](https://arxiv.org/abs/2401.03428) find
  10. Hyeong Kyu Choi, Xuefeng Du, and Yixuan Li. 2024. Safety-aware fine-tuning of large language models. arXiv preprint arXiv:2410.10014. — [arXiv:2410.10014](https://arxiv.org/abs/2410.10014) find
  11. Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, and Florian Tramèr. 2024. Agentdojo: A dynamic environment to evaluate attacks and defenses for llm agents. arXiv preprint arXiv:2406.13352. — [arXiv:2406.13352](https://arxiv.org/abs/2406.13352) find
  12. Google DeepMind. 2025. Gemini 2.5 pro. https://deepmind.google/technologies/gemini/pro/. ↗
  13. Zehang Deng, Yongjian Guo, Changzhou Han, Wan-lun Ma, Junwu Xiong, Sheng Wen, and Yang Xiang. 2024. Ai agents under threat: A survey of key security challenges and future pathways. arXiv preprint arXiv:2406.02630. — [arXiv:2406.02630](https://arxiv.org/abs/2406.02630) find
  14. Shen Dong, Shaochen Xu, Pengfei He, Yige Li, Jiliang Tang, Tianming Liu, Hui Liu, and Zhen Xiang. 2025. A practical memory injection attack against llm agents. arXiv preprint arXiv:2503.03704. — [arXiv:2503.03704](https://arxiv.org/abs/2503.03704) find
  15. Yi Dong, Ronghui Mu, Yanghao Zhang, Siqi Sun, Tianle Zhang, Changshun Wu, Gaojie Jin, Yi Qi, Jinwei Hu, Jie Meng, and 1 others. 2024. Safeguarding large language models: A survey. arXiv preprint arXiv:2406.02622. — [arXiv:2406.02622](https://arxiv.org/abs/2406.02622)
  16. Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. 2023. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models. arXiv preprint arXiv:2305.08283.
  17. Tao Feng, Chuanyang Jin, Jingyu Liu, Kunlun Zhu, Haoqin Tu, Zirui Cheng, Guanyu Lin, and Jiaxuan You. 2024. How far are we from AGI: Are LLMs all we need? Transactions on Machine Learning Research. Survey Certification. find
  18. Juraj Gottweis, Wei-Hung Weng, Alexander Daryin, Tao Tu, Anil Palepu, Petar Sirkovic, Artiom Myaskovsky, Felix Weissenberger, Keran Rong, Ryutaro Tanno, and 1 others. 2025. Towards an ai co-scientist. arXiv preprint arXiv:2502.18864.
  19. Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680.
  20. Chi Han, Jialiang Xu, Manling Li, Yi Fung, Chenkai Sun, Nan Jiang, Tarek Abdelzaher, and Heng Ji. 2023. Word embeddings are steers for language models. arXiv preprint arXiv:2305.12798.
  21. Peixuan Han, Cheng Qian, Xiusi Chen, Yuji Zhang, Denghui Zhang, and Heng Ji. 2025. Internal activation as the polar star for steering unsafe llm behavior. arXiv preprint arXiv:2502.01042. — [arXiv:2502.01042](https://arxiv.org/abs/2502.01042) find
  22. Pengfei He, Yupin Lin, Shen Dong, Han Xu, Yue Xing, and Hui Liu. 2025. Red-teaming llm multi-agent systems via communication attacks. arXiv preprint arXiv:2502.14847. find
  23. Yifeng He, Ethan Wang, Yuyang Rong, Zifei Cheng, and Hao Chen. 2024. Security of ai agents. arXiv preprint arXiv:2406.08689. — [arXiv:2406.08689](https://arxiv.org/abs/2406.08689) find
  24. David Huang, Avidan Shah, Alexandre Araujo, David Wagner, and Chawin Sitawarin. 2025. Stronger universal and transferable attacks by suppressing refusals. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Tec… find
  25. Jen-tse Huang, Jiaxu Zhou, Tailin Jin, Xuhui Zhou, Zixi Chen, Wenxuan Wang, Youliang Yuan, Maarten Sap, and Michael R Lyu. 2024. On the resilience of multi-agent systems with malicious agents. arXiv preprint arXiv:2408.00989. — [arXiv:2408.00989](https://arxiv.org/abs/2408.00989) find
  26. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and 1 others. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
  27. Tianjie Ju, Yiting Wang, Xinbei Ma, Pengzhou Cheng, Haodong Zhao, Yulong Wang, Lifeng Liu, Jian Xie, Zhuosheng Zhang, and Gongshen Liu. 2024. Flooding spread of manipulated knowledge in llm-based multi-agent communities. arXiv preprint arXiv:2407.07791. — [arXiv:2407.07791](https://arxiv.org/abs/2407.07791) find
  28. Daniel Kang, Xuechen Li, Ion Stoica, Carlos Guestrin, Matei Zaharia, and Tatsunori Hashimoto. 2024. Exploiting programmatic behavior of llms: Dual-use through standard security attacks. In 2024 IEEE Security and Privacy Workshops (SPW), pages 132–143. IEEE. find
  29. Aounon Kumar, Chirag Agarwal, Suraj Srinivas, Aaron Jiaxun Li, Soheil Feizi, and Himabindu Lakkaraju. 2023. Certifying llm safety against adversarial prompting. arXiv preprint arXiv:2309.02705.
  30. Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. 2022. Backdoor learning: A survey. IEEE transactions on neural networks and learning systems, 35(1):5–22. find
  31. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  32. Bang Liu, Xinfeng Li, Jiayi Zhang, Jinlin Wang, Tanjin He, Sirui Hong, Hongzhang Liu, Shaokun Zhang, Kaitao Song, Kunlun Zhu, Yuheng Cheng, Suyuchen Wang, Xiaoqiang Wang, Yuyu Luo, Haibo Jin, Peiyan Zhang, Ollie Liu, Jiaqi Chen, Huan Zhang, and 28 others. 2025. Advances and challenges in foundation… — [arXiv:2504.01990](https://arxiv.org/abs/2504.01990) find
  33. Chris Lu, Cong Lu, Robert Tjarko Lange, Jakob Foerster, Jeff Clune, and David Ha. 2024. The ai scientist: Towards fully automated open-ended scientific discovery. arXiv preprint arXiv:2408.06292.
  34. Ziming Luo, Zonglin Yang, Zexin Xu, Wei Yang, and Xinya Du. 2025. Llm4sr: A survey on large language models for scientific research. arXiv preprint arXiv:2501.04306.
  35. Junyuan Mao, Fanci Meng, Yifan Duan, Miao Yu, Xiaojun Jia, Junfeng Fang, Yuxuan Liang, Kun Wang, and Qingsong Wen. 2025. Agentsafe: Safeguarding large language model-based multi-agent systems via hierarchical data management. arXiv preprint arXiv:2503.04392. — [arXiv:2503.04392](https://arxiv.org/abs/2503.04392) find
  36. Tong Mu, Alec Helyar, Johannes Heidecke, Joshua Achiam, Andrea Vallone, Ian Kivlichan, Molly Lin, Alex Beutel, John Schulman, and Lilian Weng. 2024. Rule based rewards for language model safety. arXiv preprint arXiv:2411.01111. find
  37. Donghyun Lee and Mo Tiwari. 2024. Prompt infection: Llm-to-llm prompt injection within multi-agent systems. Preprint, arXiv:2410.07283. — [arXiv:2410.07283](https://arxiv.org/abs/2410.07283) find
  38. OpenAI. 2024. Openai o3 and o4-mini system card. https://cdn.openai.com/pdf/ 2221c875-02dc-4789-800b-e7758f3722c1/ o3-and-o4-mini-system-card.pdf. ↗
  39. Tianlong Li, Shihan Dou, Wenhao Liu, Muling Wu, Changze Lv, Rui Zheng, Xiaoqing Zheng, and Xuanjing Huang. 2024. Rethinking jailbreaking through the lens of representation engineering. arXiv preprint arXiv:2401.06824. — [arXiv:2401.06824](https://arxiv.org/abs/2401.06824) find
  40. OpenAI. 2025. Openai gpt-4.5 system card. https://cdn.openai.com/ gpt-4-5-system-card-2272025.pdf. ↗
  41. Xuan Li, Zhanke Zhou, Jianing Zhu, Jiangchao Yao, Tongliang Liu, and Bo Han. 2023. Deepinception: Hypnotize large language model to be jailbreaker. arXiv preprint arXiv:2311.03191.
  42. Yatao Li and Jianfeng Zhan. 2022. Saibench: Benchmarking ai for science. BenchCouncil Transactions on Benchmarks, Standards and Evaluations, 2(2):100063. find
  43. Nardine Osman and Mark d’Inverno. 2023. Human values in multiagent systems. arXiv preprint arXiv:2305.02739. — [arXiv:2305.02739](https://arxiv.org/abs/2305.02739) find
  44. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  45. Yansheng Qiu, Haoquan Zhang, Zhaopan Xu, Ming Li, Diping Song, Zheng Wang, and Kaipeng Zhang. 2025. Ai idea bench 2025: Ai research idea generation benchmark. arXiv preprint arXiv:2504.14191. — [arXiv:2504.14191](https://arxiv.org/abs/2504.14191) find
  46. Domenic Rosati, Jan Wehner, Kai Williams, Lukasz Bar-toszcze, Robie Gonzales, Subhabrata Majumdar, Hassan Sajjad, Frank Rudzicz, and 1 others. 2024. Representation noising: A defence mechanism against harmful finetuning. Advances in Neural Information Processing Systems, 37:12636–12676. find
  47. Sakana. 2024. The ai scientist: Towards fully automated open-ended scientific discovery.
  48. Samuel Schmidgall, Yusheng Su, Ze Wang, Ximeng Sun, Jialian Wu, Xiaodong Yu, Jiang Liu, Zicheng Liu, and Emad Barsoum. 2025. Agent laboratory: Using llm agents as research assistants. arXiv preprint arXiv:2501.04227.
  49. Md Shamsujjoha, Qinghua Lu, Dehai Zhao, and Liming Zhu. 2024. Towards ai-safety-by-design: A taxonomy of runtime guardrails in foundation model based systems. arXiv preprint arXiv:2408.02205. — [arXiv:2408.02205](https://arxiv.org/abs/2408.02205) find
  50. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2024. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, pages 1671–1685.
  51. Lucheng Sun, Tiejun Wu, and Ya Zhang. 2023. A defense strategy for false data injection attacks in multi-agent systems. International Journal of Systems Science, 54(16):3071–3084. find
  52. Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, and 1 others. 2024. Prioritizing safeguarding over autonomy: Risks of llm agents for science. arXiv preprint arXiv:2402.04247. — [arXiv:2402.04247](https://arxiv.org/abs/2402.04247) find
  53. Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshorn, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. 2022. Galactica: A large language model for science. arXiv preprint arXiv:2211.09085.
  54. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  55. Yu Tian, Xiao Yang, Jingyuan Zhang, Yinpeng Dong, and Hang Su. 2023. Evil geniuses: Delving into the safety of llm-based agents. arXiv preprint arXiv:2311.11855. — [arXiv:2311.11855](https://arxiv.org/abs/2311.11855) find
  56. Benyou Wang, Qianqian Xie, Jiahuan Pei, Zhihong Chen, Prayag Tiwari, Zhao Li, and Jie Fu. 2023a. Pre-trained language models in biomedical domain: A systematic survey. ACM Computing Surveys, 56(3):1– 52. find
  57. Shilong Wang, Guibin Zhang, Miao Yu, Guancheng Wan, Fanci Meng, Chongye Guo, Kun Wang, and Yang Wang. 2025. G-safeguard: A topology-guided security lens and treatment on llm-based multi-agent systems. arXiv preprint arXiv:2502.11127.
  58. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2023b. Do-not-answer: A dataset for evaluating safeguards in llms. arXiv preprint arXiv:2308.13387.
  59. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36:80079–80110.
  60. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36.
  61. Yixuan Weng, Minjun Zhu, Guangsheng Bao, Hongbo Zhang, Jindong Wang, Yue Zhang, and Linyi Yang. 2024. Cycleresearcher: Improving automated research via automated review. arXiv preprint arXiv:2411.00816.
  62. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, and 1 others. 2024. Sorry-bench: Systematically evaluating large language model safety refusal behaviors. arXiv preprint arXiv:2406.14598.
  63. Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. 2024. Gibbs sampling from human feedback: A provable kl-constrained framework for rlhf. In Proc. The Forty-first International Conference on Machine Learning (ICML2024). find
  64. Sheng Yin, Xianghe Pang, Yuanzhuo Ding, Menglan Chen, Yutong Bi, Yichen Xiong, Wenhao Huang, Zhen Xiang, Jing Shao, and Siheng Chen. 2024. Safeagentbench: A benchmark for safe task planning of embodied llm agents. arXiv preprint arXiv:2412.13178. — [arXiv:2412.13178](https://arxiv.org/abs/2412.13178) find
  65. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-resource languages jailbreak gpt-4. arXiv preprint arXiv:2310.02446.
  66. Haofei Yu, Zhaochen Hong, Zirui Cheng, Kunlun Zhu, Keyang Xuan, Jinwei Yao, Tao Feng, and Jiaxuan You. 2024. Researchtown: Simulator of human research community. arXiv preprint arXiv:2412.17767.
  67. Haofei Yu, Keyang Xuan, Fenghai Li, Zijie Lei, and Jiaxuan You. 2025. Tinyscientist: A lightweight framework for building research agents. https://github.com/ulab-uiuc/tiny-scientist. Accessed: 2025-04-14. ↗
  68. Jiakang Yuan, Xiangchao Yan, Botian Shi, Tao Chen, Wanli Ouyang, Bo Zhang, Lei Bai, Yu Qiao, and Bowen Zhou. 2025. Dolphin: Closed-loop open-ended auto-research through thinking, practice, and feedback. arXiv preprint arXiv:2501.03916.
  69. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, and 1 others. 2024. R-judge: Benchmarking safety risk awareness for llm agents. arXiv preprint arXiv:2401.10019.
  70. Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. 2024a. Breaking agents: Compromising autonomous llm agents through malfunction amplification. arXiv preprint arXiv:2407.20859.
  71. Hanrong Zhang, Jingyuan Huang, Kai Mei, Yifei Yao, Zhenting Wang, Chenlu Zhan, Hongwei Wang, and Yongfeng Zhang. 2024b. Agent security bench (asb): Formalizing and benchmarking attacks and defenses in llm-based agents. arXiv preprint arXiv:2410.02644. — [arXiv:2410.02644](https://arxiv.org/abs/2410.02644) find
  72. Qiang Zhang, Keyan Ding, Tianwen Lv, Xinda Wang, Qingyu Yin, Yiwen Zhang, Jing Yu, Yuhao Wang, Xiaotong Li, Zhuoyi Xiang, and 1 others. 2025. Scientific large language models: A survey on biological & chemical domains. ACM Computing Surveys, 57(6):1–38.
  73. Yu Zhang, Xiusi Chen, Bowen Jin, Sheng Wang, Shuiwang Ji, Wei Wang, and Jiawei Han. 2024c. A comprehensive survey of scientific large language models and their applications in scientific discovery. arXiv preprint arXiv:2406.10833. find
  74. Aaron Y Zhao, Nathan E DeSousa, Hanne C Henriksen, Ann Marie May, Xianming Tan, and David S Lawrence. 2024a. An assessment of laboratory safety training in undergraduate education. Journal of Chemical Education, 101(4):1626–1634. find
  75. Shuai Zhao, Meihuizi Jia, Zhongliang Guo, Leilei Gan, Xiaoyu Xu, Xiaobao Wu, Jie Fu, Yichao Feng, Fengjun Pan, and Luu Anh Tuan. 2024b. A survey of backdoor attacks and defenses on large language models: Implications for security measures. Authorea Preprints. find
  76. Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. On prompt-driven safeguarding for large language models. In Forty-first International Conference on Machine Learning.
  77. Xuhui Zhou, Hyunwoo Kim, Faeze Brahman, Liwei Jiang, Hao Zhu, Ximing Lu, Frank Xu, Bill Yuchen Lin, Yejin Choi, Niloofar Mireshghallah, Ronan Le Bras, and Maarten Sap. 2024. Haicosystem: An ecosystem for sandboxing safety risks in human-ai interactions. Preprint, arXiv:2409.16427. — [arXiv:2409.16427](https://arxiv.org/abs/2409.16427) find
  78. Andy Zou, Long Phan, Justin Wang, Derek Duenas, Maxwell Lin, Maksym Andriushchenko, J Zico Kolter, Matt Fredrikson, and Dan Hendrycks. 2024. Improving alignment and robustness with circuit breakers. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.
  79. Andy Zou, Zifan Wang, Nicholas Carlini, Milad Nasr, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043. .1
  80. Comparisons with Different Foundation Models To assess the impact of the underlying LLM on SafeScientist’s performance, we evaluated our framework using various foundation models. The same set of quality and safety metrics as in the main experiment were used. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2025-safescientist-toward-risk-aware-scientific/