Elamur Research⌕
☰
home / papers / 2026 / SEA-SafeguardBench: Culturally Grounded Safety Benchmark for Southeast Asian Languages

SEA-SafeguardBench: Culturally Grounded Safety Benchmark for Southeast Asian Languages

Paper · 2026 · Panuthep Tasawong, Jian Gang Ngui, Alham Fikri Aji et al.

Summary

Safeguard models that screen LLM inputs and outputs for harm are mostly tested in English. Multilingual safety benchmarks usually rely on machine translation, which can distort low-resource languages and miss local cultural risks. Southeast Asia has especially little coverage. SEA-SafeguardBench addresses this gap with a human-verified safety benchmark for seven SEA languages: Indonesian, Malay, Burmese, Thai, Tamil, Tagalog and Vietnamese. Every item is paired with an English counterpart.

The benchmark has three parts:

  • General subset: items drawn from existing English safety sets, machine-translated, then post-edited by bilingual annotators.
  • In-the-wild subset: safe and unsafe prompts written by native speakers around cultural topics from their own countries.
  • Content-generation subset: requests for culturally harmful material such as fake news, with LLM-generated responses that annotators labeled and translated.

The authors tested 20 models. Safeguards and LLMs that perform well on English do noticeably worse on SEA languages and on culture-specific harm. They frequently misjudge harmful prompts and responses, and they struggle to separate sensitive-but-safe content from content that is truly unsafe. The resource provides a native, culturally grounded way to check safety claims for an underrepresented region.

Key points

  • Scale: 21,640 samples in total (13,830 prompts and 7,810 responses) across eight languages (7 SEA plus English), built around 1,338 cultural topics.
  • Subset sizes (public set):
    • General: 4,800 prompts and 4,800 responses, built from 200 items each from JailbreakBench, Aegis2 and WildGuardMix, with Google NMT drafts edited by humans.
    • In-the-Wild: 6,020 prompts.
    • Content Generation: 3,010 prompts and 3,010 responses, with responses generated by GPT-4o.
  • Authors' conclusions: fixed-threshold safety classification gives suboptimal results, and better outcomes require improving both safeguard models and aligned LLMs together.
  • Availability and licensing: the benchmark is part of the SEA-HELM leaderboard. Licenses differ by subset: WildGuard terms for General, CC-BY for In-the-Wild, and ChatGPT terms for Content Generation.
  • Annotation and coverage: 50 native-speaker annotators were recruited, and only those who passed a qualification test were kept. Annotators were paid 18 USD/hour. Brunei, Laos, Cambodia and East Timor are not covered.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksSocial Sciences · 1,772 worksSafety Research · 543 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksBias, fairness & safety · 517 works

Cite

Cite as Panuthep Tasawong, Jian Gang Ngui, Alham Fikri Aji, Trevor Cohn and Peerat Limkonchotiwat. 2026. SEA-SafeguardBench: Culturally Grounded Safety Benchmark for Southeast Asian Languages. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.194

@inproceedings{tasawong2026sea,
  title = {{SEA-SafeguardBench: Culturally Grounded Safety Benchmark for Southeast Asian Languages}},
  author = {Panuthep Tasawong, Jian Gang Ngui, Alham Fikri Aji, Trevor Cohn and Peerat Limkonchotiwat},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.194},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.194}
}

References

62 references, 25 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Arkadeep Acharya, Brijraj Singh, and Naoyuki Onoe. 2023. Llm based generation of item-description for recommendation system. In Proceedings of the 17th ACM conference on recommender systems, pages 1204–1207. find
  2. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  3. Siavash Ameli, Siyuan Zhuang, Ion Stoica, and Michael W. Mahoney. 2025. A statistical framework for ranking LLM-based chatbots. In The Thirteenth International Conference on Learning Representations. find
  4. Navid Ayoobi, Sadat Shahriar, and Arjun Mukherjee. 2023. The looming threat of fake and llm-generated linkedin profiles: Challenges and opportunities for detection and prevention. In Proceedings of the 34th ACM conference on hypertext and social media, pages 1–10. find
  5. Azure. 2025. Azure ai content safety documentation. find
  6. Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, and 12 others. 202… find
  7. Samuel Cahyawijaya, Holy Lovenia, Joel Ruben Antony Moniz, Tack Hwa Wong, Mohammad Rifqi Farhansyah, Thant Thiri Maung, Frederikus Hudi, David Anugraha, Muhammad Ravi Shulthan Habibi, Muhammad Reza Qorib, Amit Agarwal, Joseph Marvin Imperial, Hitesh Laxmichand Patel, Vicky Feliren, Bahrul Ilmi Nasut… find
  8. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. Pr…
  9. Gabriel Chua, Leanne Tan, Ziyu Ge, and Roy Ka-Wei Lee. 2025. Rabakbench: Scaling human annotations to construct localized multilingual safety benchmarks for low-resource languages. Preprint, arXiv:2507.05980. — [arXiv:2507.05980](https://arxiv.org/abs/2507.05980) find
  10. Jacob Cohen. 1960. A coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20:37 – 46. find
  11. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2025. Or-bench: An over-refusal benchmark for large language models. Preprint, arXiv:2405.20947.
  12. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual jailbreak challenges in large language models. Preprint, arXiv:2310.06474.
  13. Joseph L. Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological Bulletin, 76:378–382. find
  14. Google Gemma Team. 2024. Gemma. find
  15. Google Gemma Team. 2025. Gemma 3. find
  16. Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. 2024. Aegis: Online adaptive ai content safety moderation with ensemble of llm experts. Preprint, arXiv:2404.05993. — [arXiv:2404.05993](https://arxiv.org/abs/2404.05993) find
  17. Shaona Ghosh, Prasoon Varshney, Makesh Narsimhan Sreedhar, Aishwarya Padmakumar, Traian Rebedea, Jibin Rajan Varghese, and Christopher Parisien. 2025. Aegis2.0: A diverse ai safety dataset and risks taxonomy for alignment of llm guardrails. Preprint, arXiv:2501.09004.
  18. Amelia Glaese, Nat McAleese, Maja Tr˛ebacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, and 15… — Improving alignment of dialogue agents via targeted human judgements
  19. Google Google Cloud. 2025. Model armor overview. find
  20. Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, and Alexandra Birch. 2022. Survey of low-resource machine translation. Computational Linguistics, 48(3):673–732.
  21. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. In Advances in Neural Information Processing Systems, volume 37, pages 8093–8131. C…
  22. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, and 1 others. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
  23. Paul Jaccard. Etude comparative de la distribution florale dans une portion des alpes et des jura. Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito Agili Lopo, William Nixon, Börje F. Karlsson, James Jaya, and 42 others. 2024. SEACrowd: A multilingual multimodal data hu…
  24. Jiaming Ji, Mickel Liu, Juntao Dai, Xuehai Pan, Chi Zhang, Ce Bian, Chi Zhang, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. Beavertails: Towards improved safety alignment of llm via a human-preference dataset. Preprint, arXiv:2307.04657.
  25. Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, Scott Johnston, Sheer El-Showk, Andy Jones, Nelson Elhage, Tristan Hume, Anna Chen, Yuntao Bai, Sam Bowman, Stanislav Fort, and 17 others. 2022.… find
  26. Mahdi Farrokhi Maleki and Richard Zhao. 2024. Procedural content generation in games: A survey with insights on emerging llm integration. In Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, volume 20, pages 167–178. find
  27. Klaus Krippendorff. 2011. Computing krippendorff’s alpha-reliability. find
  28. Christopher D. Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge University Press. find
  29. Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, and Maarten Sap. 2025. Polyguard: A multilingual safety moderation tool for 17 languages. Preprint, arXiv:2504.04377. — [arXiv:2504.04377](https://arxiv.org/abs/2504.04377) find
  30. Todor Markov, Chong Zhang, Sandhini Agarwal, Tyna Eloundou, Teddy Lee, Steven Adler, Angela Jiang, and Lilian Weng. 2023. A holistic approach to undesired content detection in the real world. Preprint, arXiv:2208.03274.
  31. Philippe Laban, Wojciech Kryscinski, Divyansh Agarwal, Alexander Fabbri, Caiming Xiong, Shafiq Joty, and Chien-Sheng Wu. 2023. SummEdits: Measuring LLM ability at factual reasoning through the lens of summarization. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Proce… find
  32. Raphael Merx, Adérito José Guterres Correia, Hanna Suominen, and Ekaterina Vylomova. 2025. Low-resource machine translation: what for? who for? an observational study on a dedicated tetun language translation service. In Proceedings of the Eighth Workshop on Technologies for Machine Translation of L… find
  33. LakeraAI. 2025. Lakeraguard. find
  34. Dongyuan Li, Ying Zhang, Zhen Wang, Shiyin Tan, Satoshi Kosugi, and Manabu Okumura. 2024. Active learning for abstractive text summarization via LLM-determined curriculum and certainty gain maximization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8959–8971, Miami… find
  35. Hao Li and Xiaogeng Liu. 2025. Injecguard: Benchmarking and mitigating over-defense in prompt injection guardrail models. Preprint, arXiv:2410.22770. — [arXiv:2410.22770](https://arxiv.org/abs/2410.22770) find
  36. Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. 2023. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. Preprint, arXiv:2310.17389.
  37. AI @ Meta Llama Team. 2024. The llama 3 herd of models. Preprint, arXiv:2407.21783.
  38. Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad João Monteiro, Pierre-André Noël, Étienne Marcotte, Sai Rajeswar Mudumba, Valen… find
  39. Raymond Ng, Thanh Ngan Nguyen, Yuli Huang, Ngee Chia Tai, Wai Yi Leong, Wei Qi Leong, Xianbin Yong, Jian Gang Ngui, Yosephine Susanto, Nicholas Cheng, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Adithya Venkatadri Hulagadri, Kok Wai Teng, Yeo Yeow Tong, Bryan Siow, Wei Yi Teo, Wayne Lau, Choon… — SEA-LION: Southeast Asian Languages in One Network
  40. Ri Chi Ng, Nirmalendu Prakash, Ming Shan Hee, Kenny Tsu Wei Choo, and Roy Ka-wei Lee. 2024. SGHateCheck: Functional tests for detecting hate speech in low-resource languages of Singapore. In Proceedings of the 8th Workshop on Online Abuse and Harms (WOAH 2024), pages 312–327, Mexico City, Mexico. As… find
  41. OpenAI. 2024. Upgrading the moderation api with our new multi- modal moderation model. find
  42. OpenAI. 2025. Introducing gpt-oss. find
  43. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train…
  44. Renhao Pei, Yihong Liu, Peiqin Lin, François Yvon, and Hinrich Schuetze. 2025. Understanding in-context machine translation for low-resource languages: A case study on Manchu. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 87… find
  45. Paul Röttger, Hannah Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, and Dirk Hovy. 2024. XSTest: A test suite for identifying exaggerated safety behaviours in large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational L…
  46. Wenliang Shan, Michael Fu, Rui Yang, and Chakkrit Tantithamthavorn. 2025. Sealguard: Safeguarding the multilingual conversations in southeast asian languages for llm software systems. Preprint, arXiv:2507.08898. — [arXiv:2507.08898](https://arxiv.org/abs/2507.08898) find
  47. Jiayang Song, Yuheng Huang, Zhehua Zhou, and Lei Ma. 2025. Multilingual blending: Large language model safety alignment evaluation with language mixture. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 3433– 3449, Albuquerque, New Mexico. Association for Computational… find
  48. Yosephine Susanto, Adithya Venkatadri Hulagadri, Jann Railey Montalan, Jian Gang Ngui, Xianbin Yong, Wei Qi Leong, Hamsawardhini Rengarajan, Peerat Limkonchotiwat, Yifan Mai, and William Chandra Tjhi. 2025. SEA-HELM: Southeast Asian holistic evaluation of language models. In Findings of the Associat… find
  49. Leanne Tan, Gabriel Chua, Ziyu Ge, and Roy Ka-Wei Lee. 2025. Lionguard 2: Building lightweight, data-efficient & localised multilingual content moderators. Preprint, arXiv:2507.15339. find
  50. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  51. Bibek Upadhayay, Vahid Behzadan, and Ph. D. 2025. X-guard: Multilingual guard agent for content moderation. Preprint, arXiv:2504.08848. find
  52. Bertie Vidgen, Nino Scherrer, Hannah Rose Kirk, Rebecca Qian, Anand Kannappan, Scott A. Hale, and Paul Röttger. 2024. Simplesafetytests: a test suite for identifying critical safety risks in large language models. Preprint, arXiv:2311.08370.
  53. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024a. Multilingual e5 text embeddings: A technical report. Preprint, arXiv:2402.05672.
  54. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2024b. All languages matter: On the multilingual safety of large language models. Preprint, arXiv:2310.00905. — [arXiv:2310.00905](https://arxiv.org/abs/2310.00905) find
  55. Xunguang Wang, Zhenlan Ji, Wenxuan Wang, Zongjie Li, Daoyuan Wu, and Shuai Wang. 2025. Sok: Evaluating jailbreak guardrails for large language models. Preprint, arXiv:2506.10597. — [arXiv:2506.10597](https://arxiv.org/abs/2506.10597) find
  56. Genta Indra Winata, Frederikus Hudi, Patrick Amadeus Irawan, David Anugraha, Rifki Afina Putri, Wang Yutong, Adam Nohejl, Ubaidillah Ariq Prathama, Nedjma Ousidhoum, Afifa Amriani, Anar Rzayev, Anirban Das, Ashmari Pramodya, Aulia Adila, Bryan Wilie, Candy Olivia Mawalim, Cheng Ching Lam, Daud Abola… — WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
  57. Tinghao Xie, Xiangyu Qi, Yi Zeng, Yangsibo Huang, Udari Madhushani Sehwag, Kaixuan Huang, Luxi He, Boyi Wei, Dacheng Li, Ying Sheng, Ruoxi Jia, Bo Li, Kai Li, Danqi Chen, Peter Henderson, and Prateek Mittal. 2025. Sorry-bench: Systematically evaluating large language model safety refusal. Preprint,… find
  58. Wenjun Zeng, Yuchi Liu, Ryan Mullins, Ludovic Peran, Joe Fernandez, Hamza Harkous, Karthik Narasimhan, Drew Proud, Piyush Kumar, Bhaktipriya Radharapu, Olivia Sturman, and Oscar Wahltinez. 2024. Shieldgemma: Generative ai content moderation based on gemma. Preprint, arXiv:2407.21772. — [arXiv:2407.21772](https://arxiv.org/abs/2407.21772) find
  59. Haiquan Zhao, Chenhan Yuan, Fei Huang, Xiaomeng Hu, Yichang Zhang, An Yang, Bowen Yu, Dayiheng Liu, Jingren Zhou, Junyang Lin, and 1 others. 2025a. Qwen3guard technical report. arXiv preprint arXiv:2510.14276.
  60. Weixiang Zhao, Yulin Hu, Yang Deng, Tongtong Wu, Wenxuan Zhang, Jiahe Guo, An Zhang, Yanyan Zhao, Bing Qin, Tat-Seng Chua, and Ting Liu. 2025b. MPO: Multilingual safety alignment via reward gap optimization. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (…
  61. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: A…
  62. Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. Toolqa: A dataset for LLM question answering with external tools. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, Decembe…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tasawong-2026-sea-safeguardbench-culturally-grounded-safety/