Elamur Research⌕
☰
home / papers / 2026 / AI Safety Lost in Translation: Evaluating the Effectiveness of English-Italian Cross-Lingual LLM Safety Alignment

AI Safety Lost in Translation: Evaluating the Effectiveness of English-Italian Cross-Lingual LLM Safety Alignment

Summary

Work on LLM safety alignment is mostly done in English, so it is unclear whether these protections carry over to other languages. This paper uses English–Italian as a test case. It asks whether small open models are less safe in Italian, and whether prompt engineering or supervised fine-tuning (SFT) can close the gap. SFT is tested with English-only data and with bilingual data. The authors evaluate Qwen3 (1.7B), Llama 3.2 (3B) and Gemma 3 (1B, 4B) on six risk categories of the CFSafety benchmark. They fine-tune on the Gretel Safety Alignment dataset and build Italian versions of both resources through machine translation plus manual spot-checks. Statistical testing is used throughout. Every model scored lower on safety in Italian. Safety-oriented prompting gave the most reliable improvement. SFT was erratic: English-only training sometimes failed to transfer to Italian and even made some models worse, and bilingual SFT repeatedly trailed the other methods. The takeaway is that English alignment cannot be assumed to protect speakers of other languages, so each language needs its own safety evaluation.

Key points

  • Models: four open models up to 4B parameters (the hardware limit): Qwen3 1.7B, Llama 3.2 3B, Gemma 3 1B and Gemma 3 4B.
  • Evaluation data: CFSafety, with about 25,000 prompts (over 10,000 in English) across 10 risk groups. Only 6 groups were retained: social bias; criminal or unethical material; insulting language and sensitive subjects; leakage of private data; misleading content; and persuasion via scenario embedding.
  • Training data: the Gretel Safety Alignment dataset, with about 5,600 English scenarios paired with safe responses in five risk categories. These categories were mapped conceptually onto the selected CFSafety groups.
  • Italian translation: done via the Google Translate API. A fluent co-author manually checked 200 randomly sampled items out of 8,238.
  • Failure cases: the appendix shows examples where SFT backfired. In one, Gemma3 1B refused a drug-smuggling prompt before training (safety score 5). After bilingual SFT it gave evasion advice in both languages (score 0.01).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/24nruqbycv2a

@inproceedings{wu2026ai,
  title = {{AI Safety Lost in Translation: Evaluating the Effectiveness of English-Italian Cross-Lingual LLM Safety Alignment}},
  author = {Wu, Alessio and Brandao, Martim},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/24nruqbycv2a},
  url = {https://doi.org/10.63317/24nruqbycv2a}
}

References

46 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. 2024. Building Datasets to Enable Safer AI Responses. https://gretel.ai/blog/gretel-open-synthetic-safety-dataset.2024. Comparing LLM Alignment Techniques. https://apxml.com/courses/llm-alignment-safety/chapter-3-advanced-alignment-algorithms/comparative-analysis-alignment-techniques.2024. Gemma 2 is now available to researchers and developers. https://blog.google/technology/developers/google-gemma-2/.2024. Generation_config.json · meta-llama/Llama-3.2-3B-Instruct at main. https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct/blob/main/generation_config.json.2024. Gretelai/gretel-safety-alignment-en-v1 · Datasets at Hugging Face. https://huggingface.co/datasets/gretelai/gretel-safety-alignment-en-v1/viewer/default/train.2024. Llama 3.2: Revolutionizing edge AI and vision with open, customizable models. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/.2024. Meta-llama/Llama-3.2-3B-Instruct · Hugging Face. https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct.2024. A Survey on Multilingual Large Language Models: Corpora, Alignment, and Bias. https://arxiv.org/html/2404.00929v1.2024. TinyLLM Leaderboard. https://ubiqui touslearning.github.io/TinyLLMLead erBoard/#/slm. 2025a. Gemma 3. https://deepmind.googl e/models/gemma/gemma-3/. 2025b. Gemma 3: How to Run & Fine-tune | Unsloth Documentation. https://docs.unsloth.ai/basics/tutorials-how-to-fine-tune-and-run-llms/gemma-3-how-to-run-and-fine-tune.2025c. Gemma 3 model card. https://ai.goo gle.dev/gemma/docs/core/model_card _3. 2025. Google/gemma-3-1b-it · Hugging Face. https://huggingface.co/google/gemma-3-1b-it.2025. Making LLMs even more accessible with bitsandbytes, 4-bit quantization and QLoRA. https://huggingface.co/blog/4bit-transformers-bitsandbytes.2025. Qwen/Qwen3-1.7B · Hugging Face. https://huggingface.co/Qwen/Qwen3-1.7B.2025. Supervised Fine-Tuning vs. RLHF: How to Choose the Right Approach to Train Your LLM | Invisible Technologies. https://www.invisible.co/blog/supervised-fine-tuning-vs-rlhf-how-to-choose-the-right-approach-to-train-your-llm. ↗
  2. Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, Tom Henighan, Andy Jones, Nicholas Joseph, Ben Mann, Nova Das-Sarma, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Jackson Kernion, Kamal Ndousse, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, and Jared Kaplan. 2021. A General Language Assistant as a Laboratory for Alignment. ↗
  3. Burak Erinç Çetin, Yıldırım Özen, Elif Naz Demiryıl-maz, Kaan Engür, and Cagri Toraman. 2025. OpenEthics: A Comprehensive Ethical Evaluation of Open-Source Generative Large Language Models. ↗
  4. Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. ↗
  5. Nadezhda Chirkova and Vassilina Nikoulina. 2024. Zero-shot cross-lingual transfer in instruction tuning of large language models. find
  6. Yu Ying Chiu, Liwei Jiang, Bill Yuchen Lin, Chan Young Park, Shuyue Stella Li, Sahithya Ravi, Mehar Bhatia, Maria Antoniak, Yulia Tsvetkov, Vered Shwartz, et al. 2024. Culturalbench: a robust, diverse and challenging benchmark on measuring (the lack of) cultural knowledge of llms. find
  7. Yue Deng, Wenxuan Zhang, Sinno Jialin Pan, and Lidong Bing. 2024. Multilingual Jailbreak Challenges in Large Language Models. ↗ our card
  8. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient Fine-tuning of Quantized LLMs. find
  9. Emilio Ferrara. 2023. Fairness And Bias in Artificial Intelligence: A Brief Survey of Sources, Impacts, And Mitigation Strategies. Sci, 6(1):3. ↗
  10. Robert French. 1993. Catastrophic interference in connectionist networks: Can It Be predicted, can It be prevented? In Advances in Neural Information Processing Systems, volume 6. Morgan-Kaufmann. find
  11. Shaz Furniturewala, Surgan Jandial, Abhinav Java, Pragyan Banerjee, Simra Shahid, Sumit Bhatia, and Kokil Jaidka. 2024. “Thinking” Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 213–227, Miami, Florida, USA. Association for Computational Linguistics. ↗
  12. Zishan Guo, Renren Jin, Chuang Liu, Yufei Huang, Dan Shi, Supryadi, Linhao Yu, Yan Liu, Jiaxuan Li, Bojian Xiong, and Deyi Xiong. 2023. Evaluating Large Language Models: A Comprehensive Survey. ↗ our card
  13. Daniil Gurgurov, Tanja Bäumel, and Tatiana Anikina. 2024. Multilingual large language models and curse of multilinguality. arXiv preprint arXiv:2406.10602. ↗
  14. Yutan Huang, Chetan Arora, Wen Cheng Houng, Tanjila Kanij, Anuradha Madulgalla, and John Grundy. 2025. Ethical Concerns of Generative AI and Mitigation Strategies: A Systematic Mapping Study. ↗
  15. Damjan Kalajdzievski. 2024. Scaling Laws for Forgetting When Fine-Tuning Large Language Models. ↗
  16. Ashutosh Kumar, Shiv Vignesh Murthy, Sagarika Singh, and Swathy Ragupathy. 2024. The Ethics of Interaction: Mitigating Security Threats in LLMs. ↗
  17. Jie Li, Yi Liu, Chongyang Liu, Ling Shi, Xiaoning Ren, Yaowen Zheng, Yang Liu, and Yinxing Xue. 2024. A Cross-Language Investigation into Jailbreak Attacks in Large Language Models. ↗
  18. Zhihao Liu and Chenhui Hu. 2024. CFSafety: Comprehensive Fine-grained Safety Assessment for LLMs. ↗
  19. Yuxiao Lu, Arunesh Sinha, and Pradeep Varakantham. 2024a. Semantic Loss Guided Data Efficient Supervised Fine Tuning for Safe Responses in LLMs. ↗
  20. Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu. 2024b. Small Language Models: Survey, Measurements, and Insights. ↗
  21. Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, and Santiago Zanella-Béguelin. 2023. Analyzing Leakage of Personally Identifiable Information in Language Models. find
  22. Kaifeng Lyu, Haoyu Zhao, Xinran Gu, Dingli Yu, Anirudh Goyal, and Sanjeev Arora. 2025. Keeping LLMs Aligned After Fine-tuning: The Crucial Role of Prompt Templates. find
  23. Moin Nadeem, Anna Bethke, and Siva Reddy. 2020. StereoSet: Measuring stereotypical bias in pretrained language models. find our card
  24. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. 2021. BBQ: A Hand-Built Bias Benchmark for Question Answering. https://arxiv.org/abs/2110.08193v2. ↗ our card
  25. Benji Peng, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Junyu Liu, and Qian Niu. 2024. Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks. find
  26. Abhinav Sukumar Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. 2025. Normad: A framework for measuring the cultural adaptability of large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2373–2403. find
  27. Paul Röttger, Fabio Pernisi, Bertie Vidgen, and Dirk Hovy. 2025. SafetyPrompts: A Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety. ↗
  28. Taylor Sorensen, Liwei Jiang, Jena D Hwang, Sydney Levine, Valentina Pyatkin, Peter West, Nouha Dziri, Ximing Lu, Kavel Rao, Chandra Bhagavatula, et al. 2024. Value kaleidoscope: Engaging ai with pluralistic human values, rights, and duties. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19937–19947. find
  29. Yingshui Tan, Yilei Jiang, Yanshi Li, Jiaheng Liu, Xingyuan Bu, Wenbo Su, Xiangyu Yue, Xiaoyong Zhu, and Bo Zheng. 2025. Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean-bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penchev, Gaël Liu, Francesco Visin, Kathleen Kenealy, Lucas Beyer, Xiaohai Zhai, Anton Tsitsulin, Robert Busa-Fekete, Alex Feng, Noveen Sachdeva, Benjamin Coleman, Yi Gao, Basil Mustafa, Iain Barr, Emilio Parisotto, David Tian, Matan Eyal, Colin Cherry, Jan-Thorsten Peter, Danila Sinopalnikov, Surya Bhupatiraju, Rishabh Agarwal, Mehran Kazemi, Dan Malkin, Ravin Kumar, David Vilar, Idan Brusilovsky, Jiaming Luo, Andreas Steiner, Abe Friesen, Abhanshu Sharma, Abheesht Sharma, Adi Mayrav Gilady, Adrian Goedeckemeyer, Alaa Saade, Alex Feng, Alexander Kolesnikov, Alexei Bendebury, Alvin Abdagic, Amit Vadi, András György, André Susano Pinto, Anil Das, Ankur Bapna, Antoine Miech, Antoine Yang, Antonia Paterson, Ashish Shenoy, Ayan Chakrabarti, Bilal Piot, Bo Wu, Bobak Shahriari, Bryce Petrini, Charlie Chen, Charline Le Lan, Christopher A. Choquette-Choo, C. J. Carey, Cormac Brick, Daniel Deutsch, Danielle Eisenbud, Dee Cattle, Derek Cheng, Dimitris Paparas, Divyashree Shivakumar Sreepathihalli, Doug Reid, Dustin Tran, Dustin Zelle, Eric Noland, Erwin Huizenga, Eugene Kharitonov, Frederick Liu, Gagik Amirkhanyan, Glenn Cameron, Hadi Hashemi, Hanna Klimczak-Plucińska, Harman Singh, Harsh Mehta, Harshal Tushar Lehri, Hussein Hazimeh, Ian Ballantyne, Idan Szpektor, Ivan Nardini, Jean Pouget-Abadie, Jetha Chan, Joe Stanton, John Wieting, Jonathan Lai, Jordi Orbay, Joseph Fernandez, Josh Newlan, Ju-yeong Ji, Jyotinder Singh, Kat Black, Kathy Yu, Kevin Hui, Kiran Vodrahalli, Klaus Greff, Linhai Qiu, Marcella Valentine, Marina Coelho, Marvin Ritter, Matt Hoffman, Matthew Watson, Mayank Chaturvedi, Michael Moynihan, Min Ma, Nabila Babar, Natasha Noy, Nathan Byrd, Nick Roy, Nikola Momchev, Nilay Chauhan, Noveen Sachdeva, Oskar Bunyan, Pankil Botarda, Paul Caron, Paul Kishan Rubenstein, Phil Culliton, Philipp Schmid, Pier Giuseppe Sessa, Pingmei Xu, Piotr Stanczyk, Pouya Tafti, Rakesh Shivanna, Renjie Wu, Renke Pan, Reza Rokni, Rob Willoughby, Rohith Vallu, Ryan Mullins, Sammy Jerome, Sara Smoot, Sertan Girgin, Shariq Iqbal, Shashir Reddy, Shruti Sheth, Siim Põder, Sijal Bhatnagar, Sindhu Raghuram Panyam, Sivan Eiger, Susan Zhang, Tianqi Liu, Trevor Yacovone, Tyler Liechty, Uday Kalra, Utku Evci, Vedant Misra, Vincent Roseberry, Vlad Feinberg, Vlad Kolesnikov, Woohyun Han, Woosuk Kwon, Xi Chen, Yinlam Chow, Yuvein Zhu, Zichuan Wei, Zoltan Egyed, Victor Cotruta, Minh Giang, Phoebe Kirk, Anand Rao, Kat Black, Nabila Babar, Jessica Lo, Erica Moreira, Luiz Gustavo Martins, Omar Sanseviero, Lucas Gonzalez, Zach Gleicher, Tris Warkentin, Vahab Mirrokni, Evan Senter, Eli Collins, Joelle Barral, Zoubin Ghahramani, Raia Hadsell, Yossi Matias, D. Sculley, Slav Petrov, Noah Fiedel, Noam Shazeer, Oriol Vinyals, Jeff Dean, Demis Hassabis, Koray Kavukcuoglu, Clement Farabet, Elena Buchatskaya, Jean-Baptiste Alayrac, Rohan Anil, Dmitry, Lepikhin, Sebastian Borgeaud, Olivier Bachem, Armand Joulin, Alek Andreev, Cassidy Hardin, Robert Dadashi, and Léonard Hussenot. 2025. Gemma 3 Technical Report. ↗
  30. Qwen Team. 2024. Qwen2.5: A Party of Foundation Models! https://qwenlm.github.io/blog/qwen2.5/. ↗
  31. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models. find
  32. Fengxiang Wang, Ranjie Duan, Peng Xiao, Xiaojun Jia, YueFeng Chen, Chongwen Wang, Jialing Tao, Hang Su, Jun Zhu, and Hui Xue. 2024a. MRJ-Agent: An Effective Jailbreak Agent for Multi-Round Dialogue. ↗
  33. Shuhe Wang, Shengyu Zhang, Jie Zhang, Runyi Hu, Xiaoya Li, Tianwei Zhang, Jiwei Li, Fei Wu, Guoyin Wang, and Eduard Hovy. 2024b. Reinforcement Learning Enhanced LLMs: A Survey. ↗
  34. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2024c. All Languages Matter: On the Multilingual Safety of Large Language Models. ↗
  35. Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment. find
  36. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, Zac Kenton, Sasha Brown, Will Hawkins, Tom Stepleton, Courtney Biles, Abeba Birhane, Julia Haas, Laura Rimell, Lisa Anne Hendricks, William Isaac, Sean Legassick, Geoffrey Irving, and Iason Gabriel. 2021. Ethical and social risks of harm from Language Models. ↗ our card
  37. Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022. An Explanation of In-context Learning as Implicit Bayesian Inference. ↗ our card
  38. Jingxin Xu, Guoshun Nan, Sheng Guan, Sicong Leng, Yilian Liu, Zixiao Wang, Yuyang Ma, Zhili Zhou, Yanzhao Hou, and Xiaofeng Tao. 2025. Refining Positive and Toxic Samples for Dual Safety Self-Alignment of LLMs with Minimal Human Interventions. ↗
  39. Zhao Xu, Fan Liu, and Hao Liu. 2024. Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMs. find
  40. Xinyi Yang, Runzhe Zhan, Derek F. Wong, Shu Yang, Junchao Wu, and Lidia S. Chao. 2025. Rethinking Prompt-based Debiasing in Large Language Models. ↗
  41. Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen H. Bach, and Julia Kreutzer. 2025. The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It. ↗
  42. Yu Zhang, Shutong Qiao, Jiaqi Zhang, Tzu-Heng Lin, Chen Gao, and Yong Li. 2025. A Survey of Large Language Model Empowered Agents for Recommendation and Search: Towards Next-Generation Information Retrieval. ↗
  43. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. Safety-Bench: Evaluating the Safety of Large Language Models. find
  44. Chujie Zheng, Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. 2024. On Prompt-Driven Safeguarding for Large Language Models. ↗
  45. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2024. ROSE Doesn’t Do That: Boosting the Safety of Instruction-Tuned Large Language Models with Reverse Prompt Contrastive Decoding. In Findings of the Association for Computational Linguistics: ACL 2024, pages 13721–13736, Bangkok, Thailand. Association for Computational Linguistics. find
  46. Hanqing Zhou, Diana Inkpen, and Burak Kantarci. 2024. Evaluating and Mitigating Gender Bias in Generative Large Language Models. INTERNA-TIONAL JOURNAL OF COMPUTERS COMMU-NICATIONS & CONTROL, 19(6). ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/wu-2026-ai-safety-lost-translation-evaluating/