Elamur Research⌕
☰
home / papers / 2026 / Mind the Language Gap: Assessing LLM Safety in Italian

Mind the Language Gap: Assessing LLM Safety in Italian

Summary

Most LLM safety benchmarks are built in English and reach other languages through machine translation, which can introduce artifacts and miss cultural context. The authors propose a language-independent method for building safety test sets from native-language sources and apply it to Italian. They started from the ALERT taxonomy (6 categories, 34 subcategories) and mapped it onto Italian Wikipedia categories. They then added subcategories wherever Wikipedia held a substantial cluster of sensitive articles that ALERT did not cover, giving 42 subcategories. From the selected pages, an LLM generated three difficult Italian questions per page using a fixed template. Model responses were labelled by hand as correct refusal, safe informative, unsafe, or ambiguous. The authors tested four models: ANITA, Minerva, Llama 4 Maverick and ChatGPT-4o. The two multilingual systems were almost perfectly safe and ANITA was also strong, while Minerva often answered harmful requests without any filtering. The work offers a native, culturally grounded Italian benchmark and a recipe that other languages can reuse.

Key points

  • Released resources (GitHub: SapienzaNLP/SafeLLM-it):
    • 1,762 categorized Italian Wikipedia pages
    • 5,286 auto-generated prompts
    • the prompt template
    • Model outputs are deliberately withheld.
  • Taxonomy expansion: a new subcategory was added only when it was backed by at least 10 Wikipedia pages. Examples include:
    • terrorism, gambling, animal cruelty, sex tourism and DCA (eating disorders)
    • a new "others" category covering pseudoscience and ethical decisions
  • Balance rule: no subcategory makes up more than roughly 10–15% of the data, and pages with ambiguous titles were excluded.
  • Model results:
    • ANITA reached an 86.9% safe rate, often pairing disclaimers with informative content.
    • Minerva produced unsafe output in over one third of cases.
    • Behaviour differed most on moral, sexual and medical topics.
  • Limitations: safety labels rest on subjective human judgment, and no inter-annotator agreement is reported; the authors list stronger annotation protocols as future work.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/4mvfemxb3wvv

@inproceedings{marafatto2026mind,
  title = {{Mind the Language Gap: Assessing LLM Safety in Italian}},
  author = {Marafatto, Elena and Navigli, Roberto},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/4mvfemxb3wvv},
  url = {https://doi.org/10.63317/4mvfemxb3wvv}
}

References

29 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Andrea Bacciu, Cesare Campagnano, Giovanni Trappolini, and Fabrizio Silvestri. 2024. DanteLLM: Let’s push Italian LLM research forward! In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4343–4355, Torin… find
  2. Valerio Basile, Cristina Bosco, Michael Fell, Viviana Patti, and Rossella Varvara. 2022. Italian NLP for everyone: Resources and models from EVALITA to the European language grid. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 174–180, Marseille, France. Europea… find
  3. Rishabh Bhardwaj and Soujanya Poria. 2023. Red-teaming large language models using chain of utterances for safety-alignment. find
  4. Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramer, Hamed Hassani, and Eric Wong. 2024. Jailbreakbench: An open robustness benchmark for jailbreaking large language models. find our card
  5. Adrian De Wynter, Ishaan Watts, Tua Wongsangaroonsri, Minghui Zhang, Noura Farra, Nektar Ege Altıntoprak, Lena Baur, Samantha Claudet, Pavel Gajdušek, Qilong Gu, Anna Kaminska, Tomasz Kaminski, Ruby Kuo, Akiko Kyuba, Jongho Lee, Kartik Mathur, Petter Merok, Ivana Milovanović, Nani Paananen, Vesa-Mat… find
  6. Felix Friedrich, Katharina Hämmerl, Patrick Schramowski, Manuel Brack, Jindrich Libovicky, Kristian Kersting, and Alexander Fraser. 2025a. Multilingual text-to-image generation magnifies gender stereotypes and prompt engineering may not help you. find
  7. Felix Friedrich, Simone Tedeschi, Patrick Schramowski, Manuel Brack, Roberto Navigli, Huu Nguyen, Bo Li, and Kristian Kersting. 2025b. Llms lost in translation: M-alert uncovers cross-linguistic safety inconsistencies. find
  8. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A. Smith. 2020. Real-ToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3356–3369, Online. Association for Computational Linguist… find
  9. Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, and Nouha Dziri. 2024. Wildguard: Open one-stop moderation tools for safety risks, jailbreaks, and refusals of llms. find our card
  10. Devansh Jain, Priyanshu Kumar, Samuel Gehman, Xuhui Zhou, Thomas Hartvigsen, and Maarten Sap. 2024. Polyglotoxicityprompts: Multilingual evaluation of neural toxic degeneration in large language models. find
  11. Priyanshu Kumar, Devansh Jain, Akhila Yerukola, Liwei Jiang, Himanshu Beniwal, Thomas Hartvigsen, and Maarten Sap. 2025. Polyguard: A multilingual safety moderation tool for 17 languages. find
  12. Jing-Jing Li, Valentina Pyatkin, Max Kleiman-Weiner, Liwei Jiang, Nouha Dziri, Anne G. E. Collins, Jana Schaich Borg, Maarten Sap, Yejin Choi, and Sydney Levine. 2025. Safetyanalyst: Interpretable, transparent, and steerable safety moderation for ai behavior. find
  13. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3214–3252, Dublin, Ireland. Association for Computational Linguistics… find our card
  14. Giuseppe Magazzù, Alberto Sormani, Giulia Rizzi, Francesca Pulerà, Daniel Scalena, Stefano Cariddi, Edoardo Michielon, Marco Pasqualini, Claudio Stamile, and Elisabetta Fersini. 2025. BeaverTails-IT: Towards a safety benchmark for evaluating Italian large language models. In Proceedings of the Eleve… find
  15. Roberto Navigli, Simone Conia, and Björn Ross. 2023. Biases in large language models: Origins, inventory, and discussion. ACM J. Data Inf. Qual., 15(2):10:1–10:21. find
  16. OpenAI. 2025. Chatgpt-4o. https://openai.com/chatgpt. Large language model; accessed October 2025. ↗
  17. Riccardo Orlando, Luca Moroni, Pere-Lluís Huguet Cabot, Simone Conia, Edoardo Barba, Sergio Orlandini, Giuseppe Fiameni, and Roberto Navigli. 2024. Minerva LLMs: The first family of large language models trained from scratch on Italian data. In Proceedings of the 10th Italian Conference on Computati… find
  18. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel Bowman. 2022. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, Dublin, Ireland. Asso… find our card
  19. Fabio Pernisi, Dirk Hovy, and Paul Röttger. 2024. Compromesso! Italian many-shot jailbreaks undermine the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop), pages 245–251, Bangkok, Thailan… find
  20. Marco Polignano, Pierpaolo Basile, and Giovanni Semeraro. 2024. Advanced natural-based interaction for the italian language: Llamantino-3-anita. find
  21. Giulia Rizzi, Giuseppe Magazzù, Alberto Sormani, Francesca Pulerà, Daniel Scalena, and Elisabetta Fersini. 2025. Uncovering unsafety traits in Italian language models. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), pages 974–982, Cagliari, Italy. CEUR… find
  22. Elena Sofia Ruzzetti, Dario Onorati, Leonardo Ranaldi, Davide Venditti, and Fabio Massimo Zanzotto. 2023. Investigating gender bias in large language models for the Italian language. In Proceedings of the 9th Italian Conference on Computational Linguistics (CLiC-it 2023), pages 562–569, Venice, Ital… find
  23. Xiaoqing Tan, Prangthip Hansanti, Arina Turkatenko, Joe Chuang, Carleigh Wood, Bokai Yu, Christophe Ropers, and Marta R. Costa-jussà. 2025. Towards massive multilingual holistic bias. In Proceedings of the 6th Workshop on Gender Bias in Natural Language Processing (GeBNLP), pages 403–426, Vienna, Au… find
  24. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  25. Simone Tedeschi, Felix Friedrich, Patrick Schramowski, Kristian Kersting, Roberto Navigli, Huu Nguyen, and Bo Li. 2024. Alert: A comprehensive benchmark for assessing large language models’ safety through red teaming. find
  26. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. find
  27. Boxin Wang, Weixin Chen, Hengzhi Pei, Chulin Xie, Mintong Kang, Chenhui Zhang, Chejian Xu, Zidi Xiong, Ritik Dutta, Rylan Schaeffer, Sang T. Truong, Simran Arora, Mantas Mazeika, Dan Hendrycks, Zinan Lin, Yu Cheng, Sanmi Koyejo, Dawn Song, and Bo Li. 2024a. Decodingtrust: A comprehensive assessment… find our card
  28. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024b. Do-notanswer: Evaluating safeguards in LLMs. In Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911, St. Julian’s, Malta. Association for Computational Linguistics. find
  29. Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. 2025. A survey on multilingual large language models: corpora, alignment, and bias. Frontiers of Computer Science, 19(11). find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/marafatto-2026-mind-language-gap-assessing-llm/