Elamur Research⌕
☰
home / papers / 2025 / The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models

Paper · 2025 · Zihui Wu, Haichang Gao, Jianping He et al.

Summary

Most jailbreaking research looks at the chat interface of LLMs. This paper asks whether function calling, where a model fills in arguments for user-defined tools, opens a separate route around safety alignment. The authors design a "jailbreak function" attack. A fictional tool called WriteNovel is defined with a novel-writing scenario in which an evil character must produce content, along with a forced opening phrase (prefix injection) and a required minimum length. The model is then compelled to call it, and the harmful text appears inside the generated argument rather than in the final reply. Tested on six recent LLMs, including GPT-4o, Claude-3.5-Sonnet and Gemini-1.5-pro, the attack succeeds on average more than 90% of the time. The authors attribute this to three factors: weaker alignment of argument generation than of chat replies, the ability of users to force a specific function call, and the lack of the safety filtering applied to chat output. They also test mitigations and find that defensive prompts work well. The work matters because tool use is becoming central to deployed LLM systems.

Key points

  • The attack setup has four components: a template (placed in the function name, the function description and the argument description), custom parameters ("harm_behavior", "content_type"), system parameters (e.g., forcing tool_choice to WriteNovel) and a short trigger prompt.
  • Candidate content types (for example, a tutorial or a news article) are generated automatically. GLM-4-flash proposes a ranked list of 5 for each harmful behaviour.
  • The attack averages a success rate above 90% across six models. The authors also show that function calls are more jailbreak-prone than chat mode.
  • An earlier attempt on GPT-4 (Pelrine et al., 2023) failed because it put the harmful request in the arguments and looked for harmful output in the final response. This paper locates the weakness in argument generation itself.
  • Placing a defensive prompt in the function description was the most consistently effective defence. Putting it in the user prompt still allowed about 50% attack success in some cases. Code is available at github.com/wooozihui/jailbreakfunction. The authors note as a limitation that only one jailbreak function was tested.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Xidian UniversityXi'an · China14 works · 1,357 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksBias, fairness & safety · 517 works

Cite

Cite as Zihui Wu, Haichang Gao, Jianping He and Ping Wang. 2025. The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models. Proceedings of the 31st International Conference on Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{wu2025dark,
  title = {{The Dark Side of Function Calling: Pathways to Jailbreaking Large Language Models}},
  author = {Zihui Wu, Haichang Gao, Jianping He and Ping Wang},
  booktitle = {Proceedings of the 31st International Conference on Computational Linguistics},
  year = {2025},
  url = {https://aclanthology.org/2025.coling-main.39/}
}

References

27 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George J Pappas, and Eric Wong. 2023. Jailbreaking black box large language models in twenty queries. arXiv preprint arXiv:2310.08419.
  2. Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, and Shane Legg. 2018. Scalable agent alignment via reward modeling: a research direction. arXiv preprint arXiv:1811.07871. find
  3. Simon Lermen, Charlie Rogers-Smith, and Jeffrey Ladish. 2023. Lora fine-tuning efficiently undoes safety training in llama 2-chat 70b. arXiv preprint arXiv:2310.20624. find
  4. Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. 2024. A cross-language investigation into jailbreak attacks in large language models. arXiv preprint arXiv:2401.16765. — [arXiv:2401.16765](https://arxiv.org/abs/2401.16765) find
  5. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825.
  6. Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. 2023. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451.
  7. Huijie Lv, Xiao Wang, Yuansen Zhang, Caishuang Huang, Shihan Dou, Junjie Ye, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024. Codechameleon: Personalized encryption framework for jailbreaking large language models. Preprint, arXiv:2402.16717. — [arXiv:2402.16717](https://arxiv.org/abs/2402.16717) find
  8. Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. 2023. Tree of attacks: Jailbreaking black-box llms automatically. arXiv preprint arXiv:2312.02119. https://www.
  9. Haoqin Tu, Bingchen Zhao, Chen Wei, and Cihang Xie. 2023. Sight beyond text: Multi-modal training enhances llms in truthfulness and ethics. CoRR, abs/2309.07120. — [arXiv:2309.07120](https://arxiv.org/abs/2309.07120) find
  10. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2024. Jailbroken: How does llm safety training fail? Advances in Neural Information Processing Systems, 36.
  11. Xianjun Yang, Xiao Wang, Qi Zhang, Linda Petzold, William Yang Wang, Xun Zhao, and Dahua Lin. 2023. Shadow alignment: The ease of subverting safely-aligned language models. arXiv preprint arXiv:2310.02949. find
  12. Jiahao Yu, Xingwei Lin, and Xinyu Xing. 2023. Gptfuzzer: Red teaming large language models with auto-generated jailbreak prompts. arXiv preprint arXiv:2309.10253.
  13. Michael Noukhovitch, Samuel Lavoie, Florian Strub, and Aaron C. Courville. 2023. Language model alignment with elastic reset. In NeurIPS. find
  14. OpenAI. 2023a. Function calling and other api updates. https://openai.com/index/ function-calling-and-other-api-updates/. ↗
  15. OpenAI. 2023b. Introducing gpts. https://openai.com/index/introducing-gpts/. ↗
  16. OpenAI. 2023c. New models and developer products announced at devday. OpenAI Blog. find
  17. OpenAI. 2024. Hello gpt-4o. OpenAI Blog. find
  18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  19. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463.
  20. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms. arXiv preprint arXiv:2401.06373.
  21. Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, and Daniel Kang. 2023. Removing rlhf protections in gpt-4 via fine-tuning. arXiv preprint arXiv:2311.05553. find
  22. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.
  23. Kellin Pelrine, Mohammad Taufeeque, Michał Zaj ac, Euan McLean, and Adam Gleave. 2023. Exploiting novel gpt-4 apis. arXiv preprint arXiv:2312.14302. — [arXiv:2312.14302](https://arxiv.org/abs/2312.14302) find
  24. Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. 2023. Fine-tuning aligned language models compromises safety, even when users do not intend to! arXiv preprint arXiv:2310.03693.
  25. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, et al. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.…
  26. Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, and Lizhuang Ma. 2024. Codeattack: Revealing safety generalization challenges of large language models via code completion. In Findings of the Association for Computational Linguistics ACL 2024, pages 11437–11452. find
  27. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2024. Toolformer: Language models can teach themselves to use tools. Advances in Neural Information Processing Systems, 36.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2025-dark-side-function-calling-pathways/