Elamur Research⌕
☰
home / papers / 2026 / AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albanian

AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albanian

Summary

LLM safety benchmarks are concentrated on English and a few other well-resourced languages. Albanian, an isolated Indo-European branch with roughly 7.5 million speakers, has had essentially no dedicated safety resources. Multilingual suites either skip it or rely on machine-translated English prompts that miss local harm patterns. The authors address this with AlbanianLLMSafety, which they describe as the first dedicated Albanian safety benchmark. It contains 2,951 short prompts written in Standard Albanian that imitate the kind of brief requests real users send to chat assistants. Each prompt carries one of 11 harm-category labels and an English reference translation. The project manager reviewed every entry to confirm it fits exactly one category, and duplicates and unclear items were removed. The paper reports no model evaluations. Its contribution is the resource itself and a suggested protocol: prompt a model, label each answer safe or unsafe, and compute refusal rates overall and per category. The dataset is meant to support evaluation, safety fine-tuning, red-teaming, cross-lingual transfer studies and guardrail training for Albanian.

Key points

  • Size and coverage: 2,951 prompts across 11 categories, about 268 per category on average.
  • Categories: self-harm, violence, child exploitation, vulgar language, sexual content, racist content, radicalization, regulated goods/illegal acts, education, family and health.
  • Prompt length: very short, with a mean of 8.7 tokens and a range of 2 to 21 tokens.
  • Translations: English versions come from Google Translate and were not post-edited, so idiomatic expressions may be mistranslated.
  • Limitations and access:
    • Only Standard Albanian is covered; Gheg/Tosk dialects, colloquial register and slang are absent, so real-world risk may be underestimated.
    • The data is shared on request for safety research.
    • The work was funded by QNRF grant NPRP14C-0916-210015.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/42i5uiy6yyqq

@inproceedings{zaghouani2026albanianllmsafety,
  title = {{AlbanianLLMSafety: A Safety Evaluation Dataset for Large Language Models in Albanian}},
  author = {Zaghouani, Wajdi and Aldous, Kholoud Khalil and Fejzullaj, Isra},
  booktitle = {Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI, and DCLRL: Towards Inclusivity and Equality: Language Resources and Technologies for Under-Resourced and Endangered Languages},
  year = {2026},
  doi = {10.63317/42i5uiy6yyqq},
  url = {https://doi.org/10.63317/42i5uiy6yyqq}
}

References

32 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. 2022. Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. find our card
  2. Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Röttger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. 2023. Safety-tuned LLaMAs: Lessons from improving the safety of Large Language Models that follow instructions. arXiv preprint arXiv:2309.07875. find our card
  3. Harm category boundaries involve inherent subjectivity, and some prompts may be relevant to multiple categories. 6.4. find
  4. Assoc Anila Çepani and Assoc Adelina Çerpja. 2026. Digital morphology of the Albanian language. Journal of Responsible Technology, page 100151. find
  5. Data Availability The dataset is released for non commercial research purposes. Users must comply with platform terms of service, avoid attempts at deanonymization, and refrain from surveillance or discriminatory applications. The dataset can be Çağrı Çöltekin. 2020. A corpus of Turkish offensive la… find our card
  6. Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, et al. 2022. Red teaming language models to reduce harms: Methods, scaling behaviors, and lessons learned. arXiv preprint arXiv:2209.07858. find our card
  7. HarmBench: A standardized evaluation framework for automated red teaming and robust refusal. arXiv preprint arXiv:2402.04249. find our card
  8. Hala Mulki, Hatem Haddad, Chedi Bechikh Ali, and Halima Alshabani. 2019. L-HSAB: A Levantine Twitter dataset for hate speech and abusive language. In Proceedings of the third workshop on abusive language online, pages 111–118. find
  9. Samuel Gehman, Suchin Gururangan, Maarten Sap, Yejin Choi, and Noah A Smith. 2020. RealToxicityPrompts: Evaluating neural toxic degeneration in language models. In Findings of the association for computational linguistics: EMNLP 2020, pages 3356–3369. find our card
  10. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744. find
  11. Thomas Hartvigsen, Saadia Gabriel, Hamid Palangi, Maarten Sap, Dipankar Ray, and Ece Kamar. 2022. ToxiGen: A large-scale machine-generated dataset for adversarial and implicit hate speech detection. In Proceedings of the 60th annual meeting of the association for computational linguistics (volume 1:… find our card
  12. Ethan Perez, Saffron Huang, Francis Song, Trevor Cai, Roman Ring, John Aslanides, Amelia Glaese, Nat McAleese, and Geoffrey Irving. 2022. Red teaming language models with language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3419–3448. find our card
  13. Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. 2023. Llama Guard: LLM-based Input-Output safeguard for Human-AI conversations. arXiv preprint arXiv:2312.06674. find our card
  14. Hao Sun, Zhexin Zhang, Jiawen Deng, Jiale Cheng, and Minlie Huang. 2023. Safety assessment of Chinese large language models. arXiv preprint arXiv:2304.10436. find
  15. Jiaming Ji, Mickel Liu, Josef Dai, Xuehai Pan, Chi Zhang, Ce Bian, Boyuan Chen, Ruiyang Sun, Yizhou Wang, and Yaodong Yang. 2023. BeaverTails: Towards improved safety alignment of LLM via a human-preference dataset. Advances in Neural Information Processing Systems, 36:24678–24704. find our card
  16. Bertie Vidgen and Leon Derczynski. 2020. Directions in abusive language training data, a systematic review: Garbage in, garbage out. PLOS One, 15(12):e0243300. find
  17. Wenxuan Wang, Zhaopeng Tu, Chang Chen, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael Lyu. 2024a. All languages matter: On the multilingual safety of LLMs. In Findings of the Association for Computational Linguistics: ACL 2024, pages 5865–5877, Bangkok, Thailand. Association for Computatio… find
  18. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 6282–6293. find our card
  19. Yuxia Wang, Haonan Li, Xudong Han, Preslav Nakov, and Timothy Baldwin. 2024b. Do-notanswer: Evaluating safeguards in LLMs. In Findings of the Association for Computational Linguistics: EACL 2024, pages 896–911. find
  20. Aida Kurani and Anisa Trifoni. 2014. Syntactic similarities and differences between Albanian and English. European Scientific Journal, 14:47–62. find
  21. Sharon Levy, Emily Allaway, Melanie Subbiah, Lydia Chilton, Desmond Patton, Kathleen McKeown, and William Yang Wang. 2022. SafeText: A benchmark for exploring physical safety in language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 2407–242… find
  22. Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021. Ethical and social risks of harm from language models. arXiv preprint arXiv:2112.04359. find our card
  23. Wikipedia contributors. 2025. Albanian lan-https://en.wikipedia.org/ guage. wiki/Albanian_language. find
  24. Wikimedia Foundation. Accessed February 2026. find
  25. Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li, et al. 2024. find
  26. Yicheng Xiao, Houda Bouamor, and Wajdi Zaghouani. 2024. Chinese offensive language detection: Current status and future directions. arXiv preprint arXiv:2403.18314. — [arXiv:2403.18314](https://arxiv.org/abs/2403.18314) find
  27. Zheng-Xin Yong, Beyza Ermis, Marzieh Fadaee, Stephen Bach, and Julia Kreutzer. 2025. The state of multilingual LLM safety research: From measuring the language gap to mitigating it. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 15856–15871. find
  28. Zheng-Xin Yong, Cristina Menghini, and Stephen H Bach. 2023. Low-Resource languages jailbreak GPT-4. arXiv preprint arXiv:2310.02446. find our card
  29. Wajdi Zaghouani, Kholoud Khalil Aldous, and Yicheng Gao. 2026a. Beyond english and evasion: A human-annotated multi-domain benchmark for high-stakes llm safety evaluation in chinese. In Proceedings of the RESOURCEFUL 2026 Workshop at LREC 2026. find
  30. Wajdi Zaghouani, Shimaa Amer Ibrahim, Aruzhan Muratbek, Olzhasbek Zhakenov, and Adiya Akhmetzhanova. 2026b. Kz-safetyprompts: A kazakh safety evaluation prompt dataset for large language models. In Proceedings of the SIGUL 2026 Joint Workshop with ELE, EURALI and DCLRL at LREC 2026. find our card
  31. Wajdi Zaghouani, Hamdy Mubarak, and Md Rafiul Biswas. 2024. So hateful! building a multi-label hate speech annotated arabic dataset. In Proceedings of the Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024). find
  32. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2024. SafetyBench: Evaluating the safety of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pap… find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zaghouani-2026-albanianllmsafety-safety-evaluation-dataset-large/