Elamur Research⌕
☰
home / papers / 2025 / The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It

The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It

Paper · 2025 · Zheng Xin Yong, Beyza Ermis, Marzieh Fadaee et al.

Summary

This survey examines how much LLM safety research covers languages other than English. The authors reviewed nearly 300 papers from 2020–2024 published at ∗ACL conferences and workshops. They found candidate papers by searching abstracts for "safe" and "safety". They then labeled each paper by hand with a safety subtopic (adapted from Cui et al., 2024) and its language coverage: English only, a single non-English language, or multilingual.

The results show that the field is heavily centered on English, and the imbalance is growing. Even widely spoken languages such as Mandarin Chinese get little attention. Non-English languages usually appear only as one entry in broad multilingual benchmarks rather than as the focus of detailed study. Many English-only papers also fail to say that their findings are limited to English.

From this evidence, the authors propose recommendations for conferences and three research directions:

  • culturally grounded safety benchmarks,
  • more diverse multilingual safety training data,
  • better understanding of how alignment transfers across languages, for example through mechanistic interpretability.

The work matters because LLMs are deployed worldwide, and safety gaps in other languages can leave their speakers exposed to harm.

Key points

  • About 28% of keyword-matched papers were unrelated to LLM safety and were discarded. Pairwise inter-annotator agreement averaged 0.80–0.96 (Cohen's κ, or Jaccard similarity for the language lists).
  • The yearly lead of English-only papers over non-English or multilingual papers grew from 5 in 2020 to 83 in 2024.
  • English is studied roughly ten times more often than Chinese (zho), the second most studied language. Only about half of English-only safety papers acknowledge their limited language coverage.
  • Among top Chatbot Arena models that publish a system report, 20 of 24 offer wide multilingual support. Only 5 report doing multilingual safety alignment or red-teaming.
  • Prior work suggests that detoxification and debiasing tend to transfer across languages, while refusal training often does not. The authors identify this as an open question for crosslingual safety generalization.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Brown UniversityProvidence · United States76 works · 25,952 citations🏢CohereSan Francisco · United States15 works · 1,223 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 3Research subjects and languages the work is about.Language models & pretraining · 3,320 worksMultilingual & cross-lingual NLP · 828 worksBias, fairness & safety · 517 works

Cite

Cite as Zheng Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach and Julia Kreutzer. 2025. The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.800

@inproceedings{yong2025state,
  title = {{The State of Multilingual LLM Safety Research: From Measuring The Language Gap To Mitigating It}},
  author = {Zheng Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach and Julia Kreutzer},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.800},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.800}
}

References

131 references, 24 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. 2024. The multilingual alignment prism: Aligning global and local preferences to reduce harm. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing…
  2. Jaimeen Ahn, Hwaran Lee, Jinhwa Kim, and Alice Oh. 2022. Why knowledge distillation amplifies gender bias and how to mitigate from the perspective of DistilBERT. In Proceedings of the 4th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 266–272, Seattle, Washington. Association… find
  3. Mansour Al Ghanim, Saleh Almohaimeed, Mengxin Zheng, Yan Solihin, and Qian Lou. 2024. Jailbreaking LLMs with Arabic transliteration and Arabizi. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18584–18600, Miami, Florida, USA. Association for Computat…
  4. Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, and 1 others. 2023. Palm 2 technical report. arXiv preprint arXiv:2305.10403.
  5. Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, and 1 others. 2024. Foundational challenges in assuring alignment and safety of large language models. arXiv preprint arXiv:2404.09932. find
  6. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, and 1 others. 2022. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073.
  7. Patricia Baquedano-López and Shlomy Kattan. 2007. Growing up in a multilingual community: Insights from language socialization. Handbook of multilingualism and multilingual communication, 5:69–99. find
  8. Fazl Barez, Tingchen Fu, Ameya Prabhu, Stephen Casper, Amartya Sanyal, Adel Bibi, Aidan O’Gara, Robert Kirk, Ben Bucknall, Tim Fist, and 1 others. 2025. Open problems in machine unlearning for ai safety. arXiv preprint arXiv:2501.04952. — [arXiv:2501.04952](https://arxiv.org/abs/2501.04952) find
  9. Dipto Barman, Ziyi Guo, and Owen Conlan. 2024. The dark side of language models: Exploring the potential of llms in multimedia disinformation generation and dissemination. Machine Learning with Applications, page 100545. find
  10. Emily Bender. 2019. The \#benderrule: On naming the languages we study and why it matters. The Gradient. find
  11. Emily M Bender. 2011. On achieving and evaluating language-independence in nlp. Linguistic Issues in Language Technology, 6. find
  12. Yoshua Bengio, Sören Mindermann, and Daniel Privitera. 2025. International ai safety report 2025. find
  13. Mukul Bhutani, Kevin Robinson, Vinodkumar Prabhakaran, Shachi Dave, and Sunipa Dev. 2024. SeeG-ULL multilingual: a dataset of geo-culturally situated stereotypes. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 842–854, Bangk… find
  14. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world‘s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Irelan…
  15. Terra Blevins and Luke Zettlemoyer. 2022. Language contamination helps explains the cross-lingual capabilities of English pretrained models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3563–3574, Abu Dhabi, United Arab Emirates. Association for Co… find
  16. Lorna Carson and Ning Jiang. 2021. Offensive words in chinese dialects. In An Anatomy of Chinese Offensive Words: A Lexical and Semantic Analysis, pages 99– 143. Springer. find
  17. Yik Siu Chan, Narutatsu Ri, Yuxin Xiao, and Marzyeh Ghassemi. 2025. Speak easy: Eliciting harmful jailbreaks from llms with simple interactions. arXiv preprint arXiv:2502.04322. — [arXiv:2502.04322](https://arxiv.org/abs/2502.04322) find
  18. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  19. Team Cohere, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos, and 1 others. 2025. Command a: An enterprise-ready large language model. arXiv preprint arXiv:2504.00698. — [arXiv:2504.00698](https://arxiv.org/abs/2504.00698) find
  20. Marta Costa-jussà, Pierre Andrews, Christine Basta, Juan Ciro, Agnieszka Falenska, Seraphina Goldfarb-Tarrant, Rafael Mosquera, Debora Nozza, and Eduardo Sánchez. 2024. Overview of the shared task on machine translation gender bias evaluation with multilingual holistic bias. In Proceedings of the 5t… find
  21. Marta Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. 2023a. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In Proceedings of the 2023 Con… find
  22. Marta Costa-jussà, Pierre Andrews, Eric Smith, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Daniel Licht, and Carleigh Wood. 2023b. Multilingual holistic bias: Extending descriptors and patterns to unveil demographic biases in languages at scale. In Proceedings of the 2023 Con… find
  23. Marta Costa-jussà, Eric Smith, Christophe Ropers, Daniel Licht, Jean Maillard, Javier Ferrando, and Carlos Escolano. 2023c. Toxicity in multilingual machine translation at scale. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9570–9586, Singapore. Association for Com… find
  24. Tianyu Cui, Yanling Wang, Chuanpu Fu, Yong Xiao, Sijia Li, Xinhao Deng, Yunpeng Liu, Qinglin Zhang, Ziyi Qiu, Peiyang Li, and 1 others. 2024. Risk taxonomy, mitigation, and assessment benchmarks of large language model systems. arXiv preprint arXiv:2401.05778. — [arXiv:2401.05778](https://arxiv.org/abs/2401.05778) find
  25. Bárbara Iansã de Lima Barroso, Cláudia Regina Cabral Galvão, Luiz Bueno da Silva, and Selma Lancman. 2018. A systematic review of translation and cross-cultural adaptation of instruments for the selection of assistive technologies. Occupational Therapy International, 2018(1):4984170. find
  26. Herve Debar, Sven Dietrich, Pavel Laskov, Emil C Lupu, and Eirini Ntoutsi. 2024. Emerging security challenges of large language models. arXiv preprint arXiv:2412.17614. — [arXiv:2412.17614](https://arxiv.org/abs/2412.17614) find
  27. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual jailbreak challenges in large language models. In The Twelfth International Conference on Learning Representations.
  28. Hannah Devinney, Jenny Björklund, and Henrik Björklund. 2024. We don‘t talk about that: Case studies on intersectional analysis of social bias in large language models. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 33–44, Bangkok, Thailand. Associat… find
  29. Jean-Marc Dewaele. 2004. The emotional force of swearwords and taboo words in the speech of multilinguals. Journal of multilingual and multicultural development, 25(2-3):204–222. find
  30. Zhichen Dong, Zhanhui Zhou, Chao Yang, Jing Shao, and Yu Qiao. 2024. Attacks, defenses and evaluations for LLM conversation safety: A survey. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: L… find
  31. Yao Dou, Isadora Krsek, Tarek Naous, Anubha Kabra, Sauvik Das, Alan Ritter, and Wei Xu. 2024. Reducing privacy risks in online self-disclosures with language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13732– 13754… find
  32. Beyza Ermis, Luiza Pozzobon, Sara Hooker, and Patrick Lewis. 2024. From one to many: Expanding the scope of toxicity mitigation in language models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15041–15058, Bangkok, Thailand. Association for Computational Linguistics. find
  33. Daniel L Everett. 2012. Language: The cultural tool. Vintage. find
  34. Mahmud Fasya and Dini Gilang Sari. 2021. Sociocultural factors that determine language choice in a multilingual society. In Fifth International Conference on Language, Literature, Culture, and Education (ICOLLITE 2021), pages 412–418. Atlantis Press. find
  35. Saadia Gabriel, Isha Puri, Xuhai Xu, Matteo Malgaroli, and Marzyeh Ghassemi. 2024. Can AI relate: Testing large language model response for mental health support. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2206–2221, Miami, Florida, USA. Association for Computati… find
  36. Penelope Gardner-Chloros. 2009. Cambridge university press. find
  37. Catalina Goanta, Nikolaos Aletras, Ilias Chalkidis, Sofia Ranchordás, and Gerasimos Spanakis. 2023. Regulation and NLP (RegNLP): Taming large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8712–8724, Singapore. Association for Comput… find
  38. Claudia Gorecki, Julia M Brown, Michelle Briggs, Suzanne Coleman, Carol Dealey, Elizabeth McGinnis, E Andrea Nelson, Nikki Stubbs, Lyn Wilson, and Jane Nixon. 2014. Language translation & cross-cultural adaptation guideline. Recommendations for language translation and cross-cultural adaption of the… find
  39. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  40. Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, and 1 others. 2023. Studying large language model generalization with influence functions. arXiv preprint arXiv:2308.03296. find
  41. Melody Y Guan, Manas Joglekar, Eric Wallace, Saachi Jain, Boaz Barak, Alec Helyar, Rachel Dias, Andrea Vallone, Hongyu Ren, Jason Wei, and 1 others. 2024. Deliberative alignment: Reasoning enables safer language models. arXiv preprint arXiv:2412.16339.
  42. Nuno M. Guerreiro, Duarte M. Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models. Transactions of the Association for Computational Linguistics, 11:1500–1517.
  43. Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, and Mete Ozay. 2024. Model merging and safety alignment: One bad model spoils the bunch. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13033–13046, Miami, Florida,… find
  44. Sabit Hassan, Anthony Sicilia, and Malihe Alikhani. 2024. Active learning for robust and representative LLM generation in safety-critical scenarios. In Proceedings of the 1st Workshop on Customizable NLP: Progress and Challenges in Customizing NLP for a Domain, Application, Group, or Individual (Cus… find
  45. Xuanli He, Jun Wang, Qiongkai Xu, Pasquale Minervini, Pontus Stenetorp, Benjamin IP Rubinstein, and Trevor Cohn. 2024. Tuba: Cross-lingual transferability of backdoor attacks in llms with instruction tuning. arXiv preprint arXiv:2404.19597. find
  46. Harry Ed Hoijer. 1954. Language in culture; conference on the interrelations of language and other aspects of culture. find
  47. Wenyue Hua, Xianjun Yang, Mingyu Jin, Zelong Li, Wei Cheng, Ruixiang Tang, and Yongfeng Zhang. 2024. TrustAgent: Towards safe and trustworthy LLM-based agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10000–10016, Miami, Florida, USA. Association for Computatio… find
  48. Kaiyu Huang, Fengran Mo, Xinyu Zhang, Hongliang Li, You Li, Yuanchi Zhang, Weijian Yi, Yulong Mao, Jinchen Liu, Yuzhuang Xu, and 1 others. 2024a. A survey on large language models with multilingualism: Recent advances and new frontiers. arXiv preprint arXiv:2405.10936. — [arXiv:2405.10936](https://arxiv.org/abs/2405.10936) find
  49. Xin Huang, Tarun Kumar Vangani, Minh Duc Pham, Xunlong Zou, Bin Wang, Zhengyuan Liu, and Ai Ti Aw. 2024b. Meralion-textllm: Cross-lingual understanding of large language models in chinese, indonesian, malay, and singlish. arXiv preprint arXiv:2501.08335. — [arXiv:2501.08335](https://arxiv.org/abs/2501.08335) find
  50. Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. 2024. Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models. In First Conference on Language Modeling. find
  51. Shaoxiong Ji, Zihao Li, Indraneil Paul, Jaakko Paavola, Peiqin Lin, Pinzhen Chen, Dayyán O’Brien, Hengyu Luo, Hinrich Schütze, Jörg Tiedemann, and 1 others. 2024. Emma-500: Enhancing massively multilingual adaptation of large language models. arXiv preprint arXiv:2409.17892. — [arXiv:2409.17892](https://arxiv.org/abs/2409.17892) find
  52. Wenying Jiang. 2000. The relationship between culture and language. ELT journal, 54(4):328–334. find
  53. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  54. Arturs Kanepajs, Vladimir Ivanov, and Richard Moulange. 2024. Towards safe multilingual frontier AI. In Workshop on Socially Responsible Language Modelling Research. find
  55. Teo Keipi, Matti Näsi, Atte Oksanen, and Pekka Räsänen. 2016. Online hate and harmful content: Cross-national perspectives. Taylor & Francis. find
  56. Tseen-Ling Khoo. 2003. Banana Bending: Asian-Australian and Asian-Canadian Literatures. Hong Kong University Press. find
  57. Paria Khoshtab, Danial Namazifard, Mostafa Masoudi, Ali Akhgary, Samin Mahdizadeh Sani, and Yadollah Yaghoobzadeh. 2025. Comparative study of multilingual idioms and similes in large language models. In Proceedings of the 31st International Conference on Computational Linguistics, pages 8680–8698, A…
  58. Minbeom Kim, Jahyun Koo, Hwanhee Lee, Joonsuk Park, Hwaran Lee, and Kyomin Jung. 2024. LifeTox: Unveiling implicit toxicity in life advice. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Sho… find
  59. Hannah Rose Kirk, Alexander Whitefield, Paul Rottger, Andrew M Bean, Katerina Margatina, Rafael Mosquera-Gomez, Juan Ciro, Max Bartolo, Adina Williams, He He, and 1 others. 2024. The prism alignment dataset: What participatory, representative and individualised human feedback reveals about the subje… find
  60. Katerina Korre, Arianna Muti, Federico Ruggeri, and Alberto Barrón-Cedeño. 2025. Untangling hate speech definitions: A semantic componential analysis across cultures and domains. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3184–3198, Albuquerque, New Mexico. Assoc… find
  61. Claire Kramsch. 2014. Language and culture. AILA review, 27(1):30–55. find
  62. Julia Kreutzer, Eleftheria Briakou, Sweta Agrawal, Marzieh Fadaee, and Kocmi Tom. 2025. D\\’ej\\a vu: Multilingual llm evaluation through the lens of machine translation evaluation. arXiv preprint arXiv:2504.11829. find
  63. Udo Kruschwitz and Maximilian Schmidhuber. 2024. LLM-based synthetic datasets: Applications and limitations in toxicity detection. In Proceedings of the Fourth Workshop on Threat, Aggression & Cyberbullying @ LREC-COLING-2024, pages 37–51, Torino, Italia. ELRA and ICCL. find
  64. Sandipan Kundu, Yuntao Bai, Saurav Kadavath, Amanda Askell, Andrew Callahan, Anna Chen, Anna Goldie, Avital Balwit, Azalia Mirhoseini, Brayden McLean, and 1 others. 2023. Specific versus general principles for constitutional ai. arXiv preprint arXiv:2310.13798. find
  65. Yara Kyrychenko, Ke Zhou, Edyta Bogucka, and Daniele Quercia. 2025. C3ai: Crafting and evaluating constitutions for constitutional ai. In Proceedings of the ACM on Web Conference 2025, WWW ’25, page 3204–3218, New York, NY, USA. Association for Computing Machinery. find
  66. Benjamin Larsen and Virginia Dignum. 2024. Ai value alignment: How we can align artificial intelligence with human values. World Economic Forum. find
  67. Xiaochen Li, Zheng Xin Yong, and Stephen Bach. 2024. Preference tuning for toxicity mitigation generalizes across languages. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13422–13440, Miami, Florida, USA. Association for Computational Linguistics. find
  68. Yubo Li, Xiaobin Shen, Xinyu Yao, Xueying Ding, Yidi Miao, Ramayya Krishnan, and Rema Padman. 2025. Beyond single-turn: A survey on multi-turn interactions with large language models. arXiv preprint arXiv:2504.04717. — [arXiv:2504.04717](https://arxiv.org/abs/2504.04717) find
  69. Peiqin Lin, Shaoxiong Ji, Jörg Tiedemann, André FT Martins, and Hinrich Schütze. 2024. Mala-500: Massive language adaptation of large language models. arXiv preprint arXiv:2401.13303.
  70. Edoardo Manino, Julia Rozanova, Danilo Carvalho, Andre Freitas, and Lucas Cordeiro. 2022. Systematicity, compositionality and transitivity of deep NLP models: a metamorphic testing perspective. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2355–2366, Dublin, Ireland.… find
  71. Beverley Maxwell, MO Martin, and DL Kelly. 1996. Translation and cultural adaptation of the survey instruments. Third international mathematics and science study (TIMSS) technical report, 1:159–169. find
  72. Abdelfattah Mazari and Naoual Derraz. 2015. Language and culture. International Journal of Humanities and Cultural Studies, 2(2):350–359. find
  73. Chidozie Emmanuel Mbada, Gafar Atanda Adeo-gun, Michael Opeoluwa Ogunlana, Rufus Adesoji Adedoyin, Adesanmi Akinsulore, Taofeek Oluwole Awotidebe, Opeyemi Ayodiipo Idowu, and Olumide Ayoola Olaoye. 2015. Translation, cross-cultural adaptation and psychometric evaluation of yoruba version of the shor… find
  74. Preslav Nakov, Firoj Alam, Shaden Shaar, Giovanni Da San Martino, and Yifan Zhang. 2021. COVID-19 in Bulgarian social media: Factuality, harmfulness, propaganda, and framing. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 997–1009… find
  75. Zabir Al Nazi and Wei Peng. 2024. Large language models in healthcare and medical domain: A review. In Informatics, volume 11, page 57. MDPI. find
  76. Alexander Tobias Neumann, Yue Yin, Sulayman Sowe, Stefan Decker, and Matthias Jarke. 2024. An llm-driven chatbot in higher education for databases and information systems. IEEE Transactions on Education. find
  77. Hellina Hailu Nigatu and Inioluwa Deborah Raji. 2024. “i searched for a religious song in amharic and got sexual content instead”: Investigating online harm in low-resourced languages on youtube. In Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, pages 141–160. find
  78. Hellina Hailu Nigatu, Atnafu Lambebo Tonja, Benjamin Rosman, Thamar Solorio, and Monojit Choudhury. 2024. The zeno‘s paradox of ‘low-resource’ languages. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17753–17774, Miami, Florida, USA. Association for… find
  79. Chad Nilep. 2006. “code switching” in sociocultural linguistics. Colorado research in linguistics. find
  80. Nobal B. Niraula, Saurab Dulal, and Diwa Koirala. 2021. Offensive language detection in Nepali social media. In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), Online. Association for Computational Linguistics. find
  81. OpenAI. 2022. Introducing ChatGPT. https://openai.com/blog/chatgpt/. Accessed [Insert Date You Accessed This Page]. ↗
  82. OpenAI. 2025. Openai gpt-4.5 system card. Technical report, OpenAI. find
  83. Ankit Pal and Malaikannan Sankarasubbu. 2024. Gemini goes to Med school: Exploring the capabilities of multimodal large language models on medical challenge problems & hallucinations. In Proceedings of the 6th Clinical Natural Language Processing Workshop, pages 21–46, Mexico City, Mexico. Associati… find
  84. Endang Wahyu Pamungkas, Valerio Basile, and Viviana Patti. 2023. Towards multidomain and multilingual abusive language detection: a survey. Personal and Ubiquitous Computing, 27(1):17–43. find
  85. Bruna Pilz, Rodrigo A Vasconcelos, Freddy B Marcondes, Samuel S Lodovichi, Wilson Mello, and Débora B Grossi. 2014. The brazilian version of start back screening tool-translation, cross-cultural adaptation and reliability. Brazilian journal of physical therapy, 18:453–461. find
  86. Giada Pistilli, Alina Leidinger, Yacine Jernite, Atoosa Kasirzadeh, Alexandra Sasha Luccioni, and Margaret Mitchell. 2025. Civics: Building a dataset for examining culturally-informed values in large language models. In Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society, AIES ’24… find
  87. Flor Miriam Plaza-del Arco, Debora Nozza, Marco Guerini, Jeffrey Sorensen, and Marcos Zampieri. 2024. Countering hateful and offensive speech online - open challenges. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts, pages 11–16, Miami, F… find
  88. Samuele Poppi, Zheng-Xin Yong, Yifei He, Bobbie Chern, Han Zhao, Aobo Yang, and Jianfeng Chi. 2025. Towards understanding the fragility of multilingual llms against fine-tuning attacks. Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics:… find
  89. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. 2024. Multilingual large language model: A survey of resources, taxonomy and frontiers. arXiv preprint arXiv:2404.04925. find
  90. Manon Reusens, Philipp Borchert, Margot Mieskes, Jochen De Weerdt, and Bart Baesens. 2023. Investigating bias in multilingual language models: Cross-lingual transfer of debiasing techniques. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2887–2896, S… find
  91. Paul Röttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. 2025. Safetyprompts: a systematic review of open datasets for evaluating and improving large language model safety. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 27617–27627.
  92. Sebastian Ruder, Ivan Vulić, and Anders Søgaard. 2022. Square one bias in NLP: Towards a multi-dimensional exploration of the research manifold. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2340–2354, Dublin, Ireland. Association for Computational Linguistics. find
  93. Laura Ruis, Maximilian Mozes, Juhan Bae, Siddhartha Rao Kamalakara, Dwaraknath Gnaneshwar, Acyr Locatelli, Robert Kirk, Tim Rocktäschel, Edward Grefenstette, and Max Bartolo. 2025. Procedural knowledge in pretraining drives reasoning in large language models. In The Thirteenth International Conferen… find
  94. Mikayel Samvelyan, Sharath Chandra Raparthy, Andrei Lupu, Eric Hambro, Aram Markosyan, Manish Bhatt, Yuning Mao, Minqi Jiang, Jack Parker-Holder, Jakob Foerster, and 1 others. 2024. Rainbow teaming: Open-ended generation of diverse adversarial prompts. Advances in Neural Information Processing Syste… find
  95. Beatrice Savoldi, Marco Gaido, Luisa Bentivogli, Matteo Negri, and Marco Turchi. 2021. Gender bias in machine translation. Transactions of the Association for Computational Linguistics, 9:845–874.
  96. Ayse Pinar Saygin. 2001. Processing figurative language in a multi-lingual task: Translation, transfer and metaphor. In Proceedings of the Workshop on Corpus-based and Processing Approaches to Figurative Language. Citeseer. find
  97. Lingfeng Shen, Weiting Tan, Sihao Chen, Yunmo Chen, Jingyu Zhang, Haoran Xu, Boyuan Zheng, Philipp Koehn, and Daniel Khashabi. 2024. The language barrier: Dissecting safety challenges of LLMs in multilingual contexts. In Findings of the Association for Computational Linguistics: ACL 2024, pages 2668…
  98. Dan Shi, Tianhao Shen, Yufei Huang, Zhigen Li, Yongqi Leng, Renren Jin, Chuang Liu, Xinwei Wu, Zishan Guo, Linhao Yu, Ling Shi, Bojian Jiang, and Deyi Xiong. 2024a. Large language model safety: A holistic survey. Preprint, arXiv:2412.17686. — [arXiv:2412.17686](https://arxiv.org/abs/2412.17686) find
  99. Shaojie Shi, Xiaoyu Tan, Xihe Qiu, Chao Qu, Kexin Nie, Yuan Cheng, Wei Chu, Xu Yinghui, and Yuan Qi. 2024b. ULMR: Unlearning large language models via negative response and model parameter average. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing: Industry Tr… find
  100. Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, and 1 others. 2023. Large language models encode clinical knowledge. Nature, 620(7972):172–180.
  101. Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, and 1 others. 2025. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1–8. find
  102. Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2025. Multilingual blending: Large language model safety alignment evaluation with language mixture. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3433– 3449, Albuquerque, New Mexico. Association for Computational… find
  103. Kamal K Sridhar. 1996. Societal multilingualism. Sociolinguistics and language teaching, 47:70. find
  104. Alex Tamkin, Miles McCain, Kunal Handa, Esin Durmus, Liane Lovitt, Ankur Rathi, Saffron Huang, Alfred Mountfield, Jerry Hong, Stuart Ritchie, and 1 others. 2024. Clio: Privacy-preserving insights into real-world ai use. arXiv preprint arXiv:2412.13678. find
  105. Yan Tao, Olga Viberg, Ryan S Baker, and René F Kizilcec. 2024. Cultural bias and cultural alignment of large language models. PNAS nexus, 3(9):pgae346. find
  106. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  107. Monica M Trieu. 2019. Understanding the use of “twinkie,”“banana,” and “fob”: Identifying the origin, role, and consequences of internalized racism within asian america. Sociology Compass, 13(5):e12679. find
  108. Ahmet Üstün, Viraat Aryabumi, Zheng Yong, Wei-Yin Ko, Daniel D’souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzieh Fadaee, Julia Kreutzer, and Sara Hooker. 2024. Aya model: An instruction fine-tun… find
  109. Jun Wang, Benjamin Rubinstein, and Trevor Cohn. 2022. Measuring and mitigating name biases in neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2576–2590, Dublin, Ireland. Association for Computationa… find
  110. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024a. All languages matter: On the multilingual safety of LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865–5877, Bangkok, Thailand. Association for Computatio… find
  111. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024b. All languages matter: On the multilingual safety of LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865–5877, Bangkok, Thailand. Association for Computatio… find
  112. Xiangwen Wang, Jie Peng, Kaidi Xu, Huaxiu Yao, and Tianlong Chen. 2024c. Reinforcement learning-driven LLM agent for automated attacks on LLMs. In Proceedings of the Fifth Workshop on Privacy in Natural Language Processing, pages 170–177, Bangkok, Thailand. Association for Computational Linguistics. find
  113. Xinpeng Wang, Mingyang Wang, Yihong Liu, Hinrich Schütze, and Barbara Plank. 2025. Refusal direction is universal across safety-aligned languages. arXiv preprint arXiv:2505.17306. — [arXiv:2505.17306](https://arxiv.org/abs/2505.17306) find
  114. Yixu Wang, Yan Teng, Kexin Huang, Chengqi Lyu, Songyang Zhang, Wenwei Zhang, Xingjun Ma, Yu-Gang Jiang, Yu Qiao, and Yingchun Wang. 2024d. Fake alignment: Are LLMs really aligned well? In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistic… find
  115. Zhonghao Wang, Zijia Lu, Bo Jin, and Haiying Deng. 2023. Mediagpt: A large language model for chinese media. arXiv preprint arXiv:2307.10930. — [arXiv:2307.10930](https://arxiv.org/abs/2307.10930) find
  116. Qingsong Wen, Jing Liang, Carles Sierra, Rose Luckin, Richard Tong, Zitao Liu, Peng Cui, and Jiliang Tang. 2024. Ai for education (ai4edu): Advancing personalized education with llm and adaptive learning. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages… find
  117. Genta Winata, Alham Fikri Aji, Zheng Xin Yong, and Thamar Solorio. 2023. The decades progress on code-switching research in NLP: A systematic survey on trends and challenges. In Findings of the Association for Computational Linguistics: ACL 2023, pages 2936–2978, Toronto, Canada. Association for Com… find
  118. Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. The bitter lesson learned from 2,000+ multilingual benchmarks. arXiv preprint arXiv:2504.15521. — [arXiv:2504.15521](https://arxiv.org/abs/2504.15521) find
  119. Hemant Yadav and Sunayana Sitaram. 2022. A survey of multilingual models for automatic speech recognition. arXiv preprint arXiv:2202.12576. find
  120. Mohammad Ali Yaghan. 2008. " arabizi": A contemporary style of arabic slang. Design issues, 24(2):39– 52. find
  121. Yahan Yang, Soham Dan, Dan Roth, and Insup Lee. 2024a. Benchmarking llm guardrails in handling multilingual toxicity. arXiv preprint arXiv:2410.22153. find
  122. Zhaorui Yang, Tianyu Pang, Haozhe Feng, Han Wang, Wei Chen, Minfeng Zhu, and Qian Liu. 2024b. Self-distillation bridges distribution gap in language model fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1028– 1043…
  123. Zheng Xin Yong, Cristina Menghini, and Stephen Bach. 2023a. Low-resource languages jailbreak GPT-4. In Socially Responsible Language Modelling Research.
  124. Zheng Xin Yong, Hailey Schoelkopf, Niklas Muennighoff, Alham Fikri Aji, David Ifeoluwa Adelani, Khalid Almubarak, M Saiful Bari, Lintang Sutawika, Jungo Kasai, Ahmed Baruwa, Genta Winata, Stella Biderman, Edward Raff, Dragomir Radev, and Vassilina Nikoulina. 2023b. BLOOM+1: Adding language support t…
  125. Haneul Yoo, Yongjin Yang, and Hwaran Lee. 2024. Code-switching red-teaming: Llm evaluation for safety and multilingual understanding. arXiv preprint arXiv:2406.15481.
  126. Tongxin Yuan, Zhiwei He, Lingzhong Dong, Yiming Wang, Ruijie Zhao, Tian Xia, Lizhen Xu, Binglin Zhou, Fangqi Li, Zhuosheng Zhang, Rui Wang, and Gongshen Liu. 2024. R-judge: Benchmarking safety risk awareness for LLM agents. In Findings of the Association for Computational Linguistics: EMNLP 2024, pa…
  127. Yi Zeng, Hongpeng Lin, Jingwen Zhang, Diyi Yang, Ruoxi Jia, and Weiyan Shi. 2024. How johnny can persuade LLMs to jailbreak them: Rethinking persuasion to challenge AI safety by humanizing LLMs. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lon…
  128. Yizhou Zhang, Karishma Sharma, Lun Du, and Yan Liu. 2024a. Toward mitigating misinformation and social media manipulation in llm era. In Companion Proceedings of the ACM Web Conference 2024, WWW ’24, page 1302–1305, New York, NY, USA. Association for Computing Machinery. find
  129. Zheyuan Zhang, Daniel Zhang-Li, Jifan Yu, Linlu Gong, Jinchang Zhou, Zhanxin Hao, Jianxiao Jiang, Jie Cao, Huiqin Liu, Zhiyuan Liu, and 1 others. 2024b. Simulating classroom education with llm-empowered agents. arXiv preprint arXiv:2406.19226.
  130. Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, and Yu Qiao. 2024. Beyond one-preference-fits-all alignment: Multi-objective direct preference optimization. In Findings of the Association for Computational Linguistics: ACL 2024, pages 10586–10613, Bangkok, Thailand. Associati… find
  131. Shucheng Zhu, Bingjie Du, Jishun Zhao, Ying Liu, and Pengyuan Liu. 2024. Do PLMs and annotators share the same gender bias? definition, dataset, and framework of contextualized gender bias. In Proceedings of the 5th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 20–32, Bangko… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/yong-2025-state-multilingual-llm-safety-research/