Elamur Research⌕
☰
home / papers / 2026 / Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models

Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models

Paper · 2026 · Zirui Song, Qian Jiang, Mingxuan Cui et al.

Summary

Large Audio-Language Models (LAMs) can be pushed into producing harmful content, yet their safety has rarely been measured systematically. Earlier audio jailbreak studies mostly read text attacks such as AdvBench aloud and ignored the acoustic signal. This paper introduces AJailBench, presented as the first open benchmark for this problem.

AJailBench-Base turns jailbreak texts into speech with TTS. The texts come from papers, Reddit and automatic generators, and only prompts that still fooled ChatGPT-3.5/4 were kept.

The authors then built the Audio Perturbation Toolkit (APT), which alters recordings in time, frequency and mixing domains. A semantic consistency constraint, based on GPTScore and calibrated with human judgments, keeps the original intent intact. Bayesian optimization searches for subtle but damaging perturbation combinations. The output is AJailBench-APT+.

No tested model was safe across all categories. The perturbed audio lowered safety further for most systems, which suggests current safeguards depend too heavily on clean speech.

Key points

  • AJailBench-Base contains 1,495 adversarial audio prompts across 10 policy-violating categories, grouped under 3 core aspects. Labels follow OpenAI usage policies, were assigned with DeepSeek-V3, and the audio was synthesized with Google Cloud TTS.
  • APT offers 7 perturbation methods in three families. Examples include pitch shifting, temporal scaling, energy distribution changes, fade in/out, trimming, noise injection and extra-auditory priming.
  • Semantic tolerance experiments set the allowed perturbation ranges:
    • Similarity falls sharply when temporal scaling goes below 0.6 or above 1.2.
    • Overlays were limited to 10 rounds, below the observed maximum of 13.
  • Base and APT+ were compared on SALMONN, Qwen2-Audio, DIVA, Gemini-flash and GPT4o, out of seven LAMs evaluated overall. Time stretch and fade were the perturbations Bayesian optimization picked most often.
  • Stated limitations: the work covers English audio only and does not study defenses. Access to the data is gated under a non-commercial, responsible-use license.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢ByteDanceBeijing · China41 works · 4,180 citations🎓Northeastern UniversityBoston · United States57 works · 6,533 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 worksSpeech & spoken language · 480 works

Cite

Cite as Zirui Song, Qian Jiang, Mingxuan Cui, Mingzhe Li, Lang Gao, Zeyu Zhang, Zixiang Xu, Yanbo Wang, Guangxian Ouyang, Zhenhao Chen and Xiuying Chen. 2026. Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1259

@inproceedings{song2026audio,
  title = {{Audio Jailbreak: An Open Comprehensive Benchmark for Jailbreaking Large Audio-Language Models}},
  author = {Zirui Song, Qian Jiang, Mingxuan Cui, Mingzhe Li, Lang Gao, Zeyu Zhang, Zixiang Xu, Yanbo Wang, Guangxian Ouyang, Zhenhao Chen and Xiuying Chen},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1259},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1259}
}

References

60 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Zahid Akhtar, Thanvi Lahari Pendyala, and Virinchi Sai Athmakuri. 2024. Video and audio deepfake datasets and open issues in deepfake technology: being ahead of the curve. Forensic Sciences, 4(3):289–377. find
  3. Zalán Borsos, Raphaël Marinier, Damien Vincent, Eugene Kharitonov, Olivier Pietquin, Matt Sharifi, Olivier Teboul, David Grangier, Marco Tagliasacchi, and Neil Zeghidour. 2022. Audiolm: a language modeling approach to audio generation.(2022). arXiv preprint arXiv:2209.03143.
  4. Rizhao Cai, Zirui Song, Dayan Guan, Zhenhao Chen, Yaohang Li, Xing Luo, Chenyu Yi, and Alex Kot. 2025. Benchlmm: Benchmarking cross-style visual capability of large multimodal models. In European Conference on Computer Vision, pages 340– 358. Springer. find
  5. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J Pappas, Florian Tramer, and 1 others. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. arXiv preprint arXi…
  6. Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, and Bo Xu. 2023. Xllm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv preprint arXiv:2305.04160. — [arXiv:2305.04160](https://arxiv.org/abs/2305.04160) find
  7. Xiuying Chen, Tairan Wang, Taicheng Guo, Kehan Guo, Juexiao Zhou, Haoyang Li, Zirui Song, Xin Gao, and Xiangliang Zhang. 2025. Unveiling the power of language models in chemical research question answering. Communications Chemistry, 8(1):4.
  8. Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759.
  9. Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919.
  10. Soham Deshmukh, Benjamin Elizalde, Rita Singh, and Huaming Wang. 2023. Pengi: An audio language model for audio tasks. Advances in Neural Information Processing Systems, 36:18090–18108. find
  11. Mark Dolson. 1986. The phase vocoder: A tutorial. Computer Music Journal, 10(4):14–27. find
  12. Lijie Fan, Sijia Liu, Pin-Yu Chen, Gaoyuan Zhang, and Chuang Gan. 2021. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in neural information processing systems, 34:21480–21492. find
  13. Qingkai Fang, Shoutao Guo, Yan Zhou, Zhengrui Ma, Shaolei Zhang, and Yang Feng. 2024. Llama-omni: Seamless speech interaction with large language models. arXiv preprint arXiv:2409.06666. find
  14. Yassir Fathullah, Chunyang Wu, Egor Lakomkin, Junteng Jia, Yuan Shangguan, Ke Li, Jinxi Guo, Wenhan Xiong, Jay Mahadeokar, Ozlem Kalinli, and 1 others. 2024. Prompting large language models with speech recognition abilities. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and… find
  15. Peter I Frazier. 2018. A tutorial on bayesian optimization. arXiv preprint arXiv:1807.02811. — [arXiv:1807.02811](https://arxiv.org/abs/1807.02811) find
  16. Jinlan Fu, See Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. Gptscore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576. find
  17. Lang Gao, Xiangliang Zhang, Preslav Nakov, and Xiuying Chen. 2024. Shaping the safety boundaries: Understanding and defending against jailbreaks in large language models. arXiv preprint arXiv:2412.17034.
  18. Sreyan Ghosh, Sonal Kumar, Ashish Seth, Chandra Kiran Reddy Evuru, Utkarsh Tyagi, S Sakshi, Oriol Nieto, Ramani Duraiswami, and Dinesh Manocha. 2024. Gama: A large audio-language model with advanced audio understanding and complex reasoning abilities. arXiv preprint arXiv:2406.11768.
  19. Yuan Gong, Hongyin Luo, Alexander H Liu, Leonid Karlinsky, and James Glass. 2023. Listen, think, and understand. arXiv preprint arXiv:2305.10790. — [arXiv:2305.10790](https://arxiv.org/abs/2305.10790) find
  20. Wenhan Han, Meng Fang, Zihan Zhang, Yu Yin, Zirui Song, Ling Chen, Mykola Pechenizkiy, and Qingyu Chen. 2024. Medinst: Meta dataset of biomedical instructions. arXiv preprint arXiv:2410.13458. find
  21. William Held, Ella Li, Michael Ryan, Weiyan Shi, Yanzhe Zhang, and Diyi Yang. 2024. Distilling an end-to-end voice assistant without instruction training data. arXiv preprint arXiv:2410.02678.
  22. Yue Huang, Chujie Gao, Siyuan Wu, Haoran Wang, Xiangqi Wang, Yujun Zhou, Yanbo Wang, Jiayi Ye, Jiawen Shi, Qihui Zhang, and 1 others. 2025a. On the trustworthiness of generative foundation models: Guideline, assessment, and perspective. arXiv preprint arXiv:2502.14296. find
  23. Yue Huang, Yanbo Wang, Zixiang Xu, Chujie Gao, Siyuan Wu, Jiayi Ye, Xiuying Chen, Pin-Yu Chen, and Xiangliang Zhang. 2025b. Breaking focus: Contextual distraction curse in large language models. arXiv preprint arXiv:2502.01609. — [arXiv:2502.01609](https://arxiv.org/abs/2502.01609) find
  24. John Hughes, Sara Price, Aengus Lynch, Rylan Schaeffer, Fazl Barez, Sanmi Koyejo, Henry Sleight, Erik Jones, Ethan Perez, and Mrinank Sharma. 2024. Best-of-n jailbreaking. arXiv preprint arXiv:2412.03556. — [arXiv:2412.03556](https://arxiv.org/abs/2412.03556) find
  25. Mintong Kang, Chejian Xu, and Bo Li. 2024. Advwave: Stealthy adversarial jailbreak attack against large audio-language models. arXiv preprint arXiv:2412.08608. — [arXiv:2412.08608](https://arxiv.org/abs/2412.08608) find
  26. Kushal Lakhotia, Eugene Kharitonov, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Benjamin Bolte, Tu-Anh Nguyen, Jade Copet, Alexei Baevski, Abdelrahman Mohamed, and 1 others. 2021. On generative spoken language modeling from raw audio. Transactions of the Association for Computational Linguistics, 9:1336–1…
  27. Yanda Li, Chi Zhang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, and Yunchao Wei. 2024a. Appagent v2: Advanced agent for flexible mobile interactions. arXiv preprint arXiv:2408.11824. — [arXiv:2408.11824](https://arxiv.org/abs/2408.11824) find
  28. Yuang Li, Min Zhang, Mengxin Ren, Miaomiao Ma, Daimeng Wei, and Hao Yang. 2024b. Cross-domain audio deepfake detection: Dataset and analysis. arXiv preprint arXiv:2404.04904. find
  29. Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. 2023. The unlocking spell on base llms: Rethinking alignment via in-context learning. arXiv preprint arXiv:2312.01552.
  30. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. find
  31. Yuhan Liu, Zirui Song, Xiaoqing Zhang, Xiuying Chen, and Rui Yan. 2024b. From a tiny slip to a giant leap: An llm-based simulation for fake news evolution. arXiv preprint arXiv:2410.19064. — [arXiv:2410.19064](https://arxiv.org/abs/2410.19064) find
  32. Kangkang Lu, Chuan-Sheng Foo, Kah Kuan Teh, Huy Dat Tran, and Vijay Ramaseshan Chandrasekhar. 2019. Semi-supervised audio classification with consistency-based regularization. In IN-TERSPEECH, volume 1, pages 3654–3658. find
  33. Eliya Nachmani, Alon Levkovitch, Roy Hirsch, Julian Salazar, Chulayuth Asawaroengchai, Soroosh Mariooryad, Ehud Rivlin, RJ Skerry-Ryan, and Michelle Tadmor Ramanovich. 2023. Spoken question answering and speech continuation using spectrogram-powered llm. arXiv preprint arXiv:2305.15255. — [arXiv:2305.15255](https://arxiv.org/abs/2305.15255) find
  34. OpenAI. 2024. GPT-4o System Card. openai.com/index/gpt-4o. https://
  35. Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Zeren Luo, Jingyi Zheng, Wenhan Dong, Xinlei He, Xuechao Wang, Yingjie Xue, Shengmin Xu, and Xinyi Huang. 2026. Jalmbench: Benchmarking jailbreak vulnerabilities in audio language models. Preprint, arXiv:2505.17568. — [arXiv:2505.17568](https://arxiv.org/abs/2505.17568) find
  36. Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. 2023. Robust speech recognition via large-scale weak supervision. In International conference on machine learning, pages 28492–28518. PMLR.
  37. Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy Lillicrap, Jean-baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, and 1 others. 2024. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXi…
  38. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.
  39. Jaechul Roh, Virat Shejwalkar, and Amir Houmansadr. 2025. Multilingual and multi-accent jailbreaking of audio llms. Preprint, arXiv:2504.01094. — [arXiv:2504.01094](https://arxiv.org/abs/2504.01094) find
  40. Xinyue Shen, Zeyuan Chen, Michael Backes, Yun Shen, and Yang Zhang. 2023. " do anything now": Characterizing and evaluating in-the-wild jailbreak prompts on large language models. arXiv preprint arXiv:2308.03825.
  41. Xinyue Shen, Yixin Wu, Michael Backes, and Yang Zhang. 2024. Voice jailbreak attacks against gpt-4o. arXiv preprint arXiv:2405.19103. — [arXiv:2405.19103](https://arxiv.org/abs/2405.19103) find
  42. Zirui Song, Yaohang Li, Meng Fang, Zhenhao Chen, Zecheng Shi, Yuan Huang, and Ling Chen. 2024a. Mmac-copilot: Multi-modal agent collaboration operating system copilot. arXiv preprint arXiv:2404.18074. — [arXiv:2404.18074](https://arxiv.org/abs/2404.18074) find
  43. Zirui Song, Guangxian Ouyang, Meng Fang, Hongbin Na, Zijing Shi, Zhenhao Chen, Yujie Fu, Zeyu Zhang, Shiyu Jiang, Miao Fang, and 1 others. 2024b. Hazards in daily life? enabling robots to proactively detect and resolve anomalies. arXiv preprint arXiv:2411.00781. find
  44. Zirui Song, Bin Yan, Yuhan Liu, Miao Fang, Mingzhe Li, Rui Yan, and Xiuying Chen. 2025a. Injecting domain-specific knowledge into large language models: A comprehensive survey. arXiv preprint arXiv:2502.10708. find
  45. Zirui Song, Jingpu Yang, Yuan Huang, Jonathan Tonglet, Zeyu Zhang, Tao Cheng, Meng Fang, Iryna Gurevych, and Xiuying Chen. 2025b. Geolocation with real human gameplay data: A large-scale dataset and human-like reasoning framework. arXiv preprint arXiv:2502.13759. — [arXiv:2502.13759](https://arxiv.org/abs/2502.13759) find
  46. Tongyi SpeechTeam. 2024. Funaudiollm: Voice understanding and generation foundation models for natural interaction between humans and llms. arXiv preprint arXiv:2407.04051. — [arXiv:2407.04051](https://arxiv.org/abs/2407.04051) find
  47. Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma, and Chao Zhang. 2023. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:2310.13289.
  48. Chenxi Wang, Tianle Gu, Zhongyu Wei, Lang Gao, Zirui Song, and Xiuying Chen. 2025a. Word form matters: Llms’ semantic reconstruction under typoglycemia. arXiv preprint arXiv:2503.01714. find
  49. Tianrui Wang, Long Zhou, Ziqiang Zhang, Yu Wu, Shujie Liu, Yashesh Gaur, Zhuo Chen, Jinyu Li, and Furu Wei. 2023. Viola: Unified codec language models for speech recognition, synthesis, and translation. arXiv preprint arXiv:2305.16107. — [arXiv:2305.16107](https://arxiv.org/abs/2305.16107) find
  50. Yanbo Wang, Jiayi Ye, Siyuan Wu, Chujie Gao, Yue Huang, Xiuying Chen, Yue Zhao, and Xiangliang Zhang. 2025b. Trusteval: A dynamic evaluation toolkit on trustworthiness of generative foundation models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for… find
  51. Shuhei Watanabe. 2023. Tree-structured parzen estimator: Understanding its algorithm components and their roles for better empirical performance. arXiv preprint arXiv:2304.11127. — [arXiv:2304.11127](https://arxiv.org/abs/2304.11127) find
  52. Jian Wu, Yashesh Gaur, Zhuo Chen, Long Zhou, Yimeng Zhu, Tianrui Wang, Jinyu Li, Shujie Liu, Bo Ren, Linquan Liu, and 1 others. 2023a. On decoder-only architecture for speech-to-text and large language model integration. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pa… find
  53. Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023b. Next-gpt: Any-to-any multimodal llm. arXiv preprint arXiv:2309.05519. find
  54. Erjia Xiao, Hao Cheng, Jing Shao, Jinhao Duan, Kaidi Xu, Le Yang, Jindong Gu, and Renjing Xu. 2025. Tune in, act up: Exploring the impact of audio modality-specific edits on large audio language models in jailbreak. arXiv preprint arXiv:2501.13772. — [arXiv:2501.13772](https://arxiv.org/abs/2501.13772) find
  55. Yunfei Xie, Ce Zhou, Lang Gao, Juncheng Wu, Xianhang Li, Hong-Yu Zhou, Sheng Liu, Lei Xing, James Zou, Cihang Xie, and Yuyin Zhou. 2025. Medtrinity-25m: A large-scale multimodal dataset with multigranular annotations for medicine. Preprint, arXiv:2408.02900. — [arXiv:2408.02900](https://arxiv.org/abs/2408.02900) find
  56. Zhifei Xie and Changqiao Wu. 2024. Mini-omni: Language models can hear, talk while thinking in streaming. arXiv preprint arXiv:2408.16725. find
  57. Hao Yang, Lizhen Qu, Ehsan Shareghi, and Gholamreza Haffari. 2024. Audio is the achilles’ heel: Red teaming audio large multimodal models. arXiv preprint arXiv:2410.23861. find
  58. Zonghao Ying, Aishan Liu, Xianglong Liu, and Dacheng Tao. 2024. Unveiling the safety of gpt-4o: An empirical study using jailbreak attacks. arXiv preprint arXiv:2406.06302. — [arXiv:2406.06302](https://arxiv.org/abs/2406.06302) find
  59. Dong Zhang, Shimin Li, Xin Zhang, Jun Zhan, Pengyu Wang, Yaqian Zhou, and Xipeng Qiu. 2023. Speechgpt: Empowering large language models with intrinsic cross-modal conversational abilities. arXiv preprint arXiv:2305.11000.
  60. Andy Zou, Zifan Wang, J Zico Kolter, and Matt Fredrikson. 2023. Universal and transferable adversarial attacks on aligned language models. arXiv preprint arXiv:2307.15043.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/song-2026-audio-jailbreak-open-comprehensive-benchmark/