Elamur Research⌕
☰
home / papers / 2026 / Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese

Paper · 2026 · Wajdi Zaghouani, Kholoud Khalil Aldous and Yicheng Gao

Summary

Safety alignment built mainly on English data often fails in Chinese. There, users hide harmful requests with Pinyin spellings, homophones, characters split into radicals, underground slang (hēihuà), or hedged, curiosity-style wording. Existing Chinese and multilingual benchmarks mostly contain direct or machine-translated prompts, so they miss these tactics.

The authors introduce ChiSafe-PAS, a set of 1,897 adversarial Chinese prompts across four domains: self-harm and violence, drugs and illicit trade, fraud, and political satire. A native speaker familiar with Chinese online culture wrote every prompt by hand. Each annotated item gets:

  • a target model behaviour (REFUSE, SAFE-REDIRECT or RESPOND),
  • a label from a nine-type obfuscation taxonomy,
  • a risk rating,
  • a written rationale.

The prompts come from community knowledge rather than scraped web text or LLM generation. This should keep the benchmark independent of the systems it evaluates and limit data contamination. The paper delivers the resource only; model benchmarking is left for later work. The dataset matters because it probes both harmful compliance and needless refusal in a language with far thinner safety coverage than English.

Key points

  • 1,897 prompts in total; 1,544 (81.4%) have complete gold annotation. In the self-harm domain, 147 annotated items form the Pilot Annotation Set (PAS), and 353 remain in the Pending Annotation Wave (PAW).
  • The three-way label scheme separates harmful instruction requests (REFUSE), distress or satire (SAFE-REDIRECT) and benign decoys (RESPOND). In practice, the drug domain is labelled entirely REFUSE and satire entirely SAFE-REDIRECT.
  • The nine-category obfuscation taxonomy covers Pinyin substitution, homophone and look-alike characters, chāizì decomposition, hēihuà slang, hedging and similar tactics. Its large "Other" category (85 self-harm items alone) shows the taxonomy is still incomplete.
  • Limitations include a single prompt author, a three-person verification process with no formal inter-annotator agreement, and no evaluation yet of models such as Qwen or ChatGLM.
  • Access is research-only via a request form under CC BY-NC, and adversarial fine-tuning is prohibited. Prompts seeking harmful procedures are withheld, and the annotation guidelines are published.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Northwestern University in QatarDoha · Qatar39 works · 148 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksSocial Sciences · 1,954 worksSafety Research · 626 works

🧭 Topics 4Research subjects and languages the work is about.Language models & pretraining · 3,894 worksEnglish · 2,379 worksEvaluation & benchmarks · 1,341 worksBias, fairness & safety · 587 works

Cite

Cite as Wajdi Zaghouani, Kholoud Khalil Aldous and Yicheng Gao. 2026. Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese. Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/37ko7p43tuys

@inproceedings{zaghouani2026beyond,
  title = {{Beyond English and Evasion: A Human-Annotated Multi-Domain Benchmark for High-Stakes LLM Safety Evaluation in Chinese}},
  author = {Wajdi Zaghouani, Kholoud Khalil Aldous and Yicheng Gao},
  booktitle = {Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026)},
  year = {2026},
  doi = {10.63317/37ko7p43tuys},
  url = {https://doi.org/10.63317/37ko7p43tuys}
}

References

30 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609.
  2. Valerio Basile, Tommaso Caselli, Alexandra Balahur, and Lun-Wei Ku. 2022. Bias, subjectivity and perspectives in natural language processing. find
  3. Federico Cabitza, Andrea Campagner, and Valerio Basile. 2023. Toward a perspectivist turn in ground truthing for predictive computing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6860–6868.
  4. Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna.lmsys. org (accessed 14 April 2023), 2(3):6. ↗
  5. Justin Cui, Wei-Lin Chiang, Ion Stoica, and Cho-Jui Hsieh. 2025. Or-bench: An over-refusal benchmark for large language models.
  6. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual jailbreak challenges in large language models.
  7. Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 320–335…
  8. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858.
  9. Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli. 2023. Chatgpt outperforms crowd workers for text-annotation tasks. Proceedings of the National Academy of Sciences, 120(30):e2305016120.
  10. Chaya Hiruncharoenvate, Zhiyuan Lin, and Eric Gilbert. 2021. Algorithmically bypassing censorship on sina weibo with nondeterministic homophone substitutions. Proceedings of the International AAAI Conference on Web and Social Media, 9:150–158. find
  11. Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. 2024. Gpt-4o system card. arXiv preprint arXiv:2410.21276.
  12. Zainab Iftikhar, Amy Xiao, Sean Ransom, Jeff Huang, and Harini Suresh. 2025. How llm counselors violate ethical standards in mental health practice: A practitioner-informed framework. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, volume 8, pages 1311– 1323. find
  13. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674.
  14. Heng Ji and Kevin Knight. 2018. Creative language encoding under censorship. In Proceedings of the First Workshop on Natural Language Processing for Internet Freedom, pages 23–33, Santa Fe, New Mexico, USA. Association for Computational Linguistics. find
  15. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249.
  16. Maja Pavlovic and Massimo Poesio. 2024. The effectiveness of llms as annotators: A comparative overview and empirical analysis of direct representation. In Proceedings of the 3rd Workshop on Perspectivist Approaches to NLP (NLPerspectives)@ LREC-COLING 2024, pages 100– 110.
  17. Barbara Plank. 2022. The ”problem” of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 conference on empirical methods in natural language processing, pages 10671–10682.
  18. Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety assessment of chinese large language models. arXiv preprint arXiv:2304.10436.
  19. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024. Do-notanswer: Evaluating safeguards in llms. In Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911. find
  20. Alexander Wei, Nika Haghtalab, and Jacob Steinhardt. 2023. Jailbroken: How does LLM safety training fail? In Advances in Neural Information Processing Systems, volume 36, pages 80079– 80110. Curran Associates, Inc.
  21. Yunze Xiao, Yujia Hu, Kenny Tsu Wei Choo, and Roy Ka-Wei Lee. 2024. ToxiCloakCN: Evaluating robustness of offensive language detection in Chinese with cloaking perturbations. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 6012–6025, Miami, Florida, U… find
  22. Shujian Yang, Shiyao Cui, Chuanrui Hu, Haicheng Wang, Tianwei Zhang, Minlie Huang, Jialiang Lu, and Han Qiu. 2025. Exploring multimodal challenges in toxic Chinese detection: Taxonomy, benchmark, and findings. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14382–14396,… find
  23. WeiMing Ye and Luming Zhao. 2023. ”i know it’s sensitive”: Internet censorship, recoding, and the sensitive word culture in china. 51:100666. find
  24. Zheng Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach, and Julia Kreutzer. 2025. The state of multilingual LLM safety research: From measuring the language gap to mitigating it. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15845–15860, Suzhou,…
  25. Youliang Yuan, Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Pinjia He, Shuming Shi, and Zhaopeng Tu. 2023. Gpt-4 is too smart to be safe: Stealthy chat with llms via cipher. arXiv preprint arXiv:2308.06463.
  26. Wajdi Zaghouani, Kholoud Khalil Aldous, and Fejzullaj Isra. 2026a. Albanianllmsafety: A safety evaluation dataset for large language models in albanian. In Proceedings of LREC 2026.
  27. Wajdi Zaghouani, Shimaa Amer Ibrahim, Aruzhan Muratbek, Olzhasbek Zhakenov, and Adiya Akhmetzhanova. 2026b. Kz-safetyprompts: A kazakh safety evaluation prompt dataset for large language models. In Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI and DCLRL at LREC 2026.
  28. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. Safetybench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap…
  29. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems, 36:46595–46623.
  30. Zhenhong Zhou, Shilinlu Yan, Chuanpu Liu, Qiankun Li, Kun Wang, and Zhigang Zeng. 2026. Cssbench: Evaluating the safety of lightweight llms against chinese-specific adversarial patterns. arXiv preprint arXiv:2601.00588. — [arXiv:2601.00588](https://arxiv.org/abs/2601.00588) find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2026-beyond-english-evasion-human-annotated/