Elamur Research⌕
☰
home / papers / 2026 / Can LLMs Translate Italy's Language Varieties?

Can LLMs Translate Italy's Language Varieties?

Summary

Italy has dozens of regional and minority language varieties, and more than 30 are endangered according to UNESCO. These varieties have little written data and almost no language technology, while Italian is well served. This paper measures how accurately small and mid-sized LLMs translate between Italian and six local varieties: Sardinian and Sicilian from the islands, plus the northern Venetian, Lombard, Ligurian, and Friulan. The authors use the FLORES+ and OLDI-Seed benchmarks and score outputs with CHRF.

They compare three instruct models and two reasoning models, served through Ollama. Each is prompted with zero, one, or three examples, using instructions in either English or Italian. They also apply continued pretraining, full fine-tuning, and QLoRA to the tiny Qwen2.5-0.5B-Instruct.

Overall quality is low. The best model reaches only about 40 CHRF, and reasoning models do no better than instruct models. None of the adaptation setups beat the zero-shot baseline in general.

The study gives a reproducible baseline for these languages. It argues that progress depends on better, community-built data rather than on prompting alone.

Key points

  • Models tested: Mistral NeMo 12B, EuroLLM-9B-Instruct, Qwen2.5-14B-Instruct, DeepSeek-R1-14B, and Phi-4-Reasoning, each in both translation directions (Italian ↔ variety).
  • EuroLLM performed best:
    • With English prompts it fell from 39.6 to 37.5 CHRF as more shots were added.
    • With Italian prompts it stayed around 40 but did not improve.
  • Effect of few-shot examples on other models:
    • Mistral dropped from about 39 to about 36 CHRF.
    • Qwen2.5-14B gained, from about 37 to about 39.
    • Phi-4-Reasoning was the weakest model, averaging about 20 CHRF.
  • Hardest directions: Lombard was the hardest target language and Sardinian the hardest source.
  • Adaptation results (Qwen2.5-0.5B, trained with LitGPT):
    • Monolingual continued pretraining came closest to the zero-shot baseline but surpassed it only for Friulan.
    • Full fine-tuning helped only Lombard.
    • QLoRA was generally the worst setup.
    • The estimated footprint of all experiments was 98.82 kgCO2eq.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loresmt-1.5

@inproceedings{signoroni2026can,
  title = {{Can LLMs Translate Italy's Language Varieties?}},
  author = {Signoroni, Edoardo and Rychlý, Pavel},
  booktitle = {Proceedings for the Ninth Workshop on Technologies for Machine Translation of Low Resource Languages (LoResMT 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loresmt-1.5},
  url = {https://doi.org/10.18653/v1/2026.loresmt-1.5}
}

References

32 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Christopher R. Haberland, Jean Maillard, and Stefano Lusito. 2024. Italian-Ligurian machine translation in its cultural context. In Proceedings of the 3rd Annual Meeting of the Special Interest Group on Under-resourced Languages @ LREC-COLING 2024, pages 168–176, Torino, Italia. ELRA and ICCL. find
  2. Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, and 8 oth… find our card
  3. Jonathan Hus and Antonios Anastasopoulos. 2024. Back to school: Translation using grammar books. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 20207–20219, Miami, Florida, USA. Association for Computational Linguistics. find
  4. Seth Aycock, David Stap, Di Wu, Christof Monz, and Khalil Sima’an. 2025. Can LLMs really learn to translate a low-resource language from one grammar book? In The Thirteenth International Conference on Learning Representations. find
  5. ISTAT. 2017. L’uso della lingua italiana, dei dialetti e di altre lingue in italia. https://www.istat.it/it/ archivio/207961. Accessed: 2025-08-07. ↗
  6. Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, and 2… find
  7. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp… find our card
  8. Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700. find our card
  9. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll… find our card
  10. Martin Maiden and Mair Parry. 1997. The Dialects of Italy. Routledge. find
  11. Rodolfo Delmonte, Antonella Bristot, Sara Tonelli, and Emanuele Pianta. 2009. English/veneto resource poor machine translation with stilven. In Proceedings of ISMTCL, pages 82–89, Besançon, France. Presses Universitaires de Franche-Comté. find
  12. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurollm: Multilingual language m… find our card
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  14. Raphaël Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo, and Ekaterina Vylomova. 2024. Low-resource machine translation through retrieval-augmented LLM prompting: A study on the Mambai language. In Proceedings of the 2nd Workshop on Resources and Technologies for Indigenous, Endangered an… find
  15. Gianfranco Fronteddu, Hèctor Alòs i Font, and Francis M. Tyers. 2017. Una eina per a una llengua en procés d’estandardització: El traductor automàtic català-sard. Linguamática, 9(2):3–20. find
  16. Mistral-AI. 2024. Mistral nemo. https://mistral.ai/news/mistral-nemo. Accessed: 2025-08-07. ↗
  17. Christopher Moseley. 2010. Atlas of the World’s Languages in Danger. UNESCO Publishing. find
  18. Ping Guo, Yubing Ren, Yue Hu, Yunpeng Li, Jiarui Zhang, Xingsheng Zhang, and Heyan Huang. 2024. Teaching large language models to translate on low-resource languages with textbook prompting. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources an… find
  19. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  20. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  21. Eryk Wdowiak. 2022. A recipe for low-resource nmt. In Intelligent Computing (SAI 2022), pages 739–746, Cham, Switzerland. Springer International Publishing. find
  22. Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, and 25 others. 2025. Qwen2.5 technical report. Preprint, arXiv:2412.15115. find
  23. Alan Ramponi. 2024. Language varieties of Italy: Technology challenges and opportunities. Transactions of the Association for Computational Linguistics, 12:19– 38. find
  24. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  25. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. ChatGPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pages 392–418, Singapore. Association for Computational Linguistics. find our card
  26. Peng Shu, Junhao Chen, Zhengliang Liu, Hui Wang, Zihao Wu, Tianyang Zhong, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Constance Owl, Xiaoming Zhai, Ninghao Liu, Claudio Saunt, and Tianming Liu. 2024. Transcending language boundaries: Harnessing llms for low-resource language translation… — [arXiv:2411.11295](https://arxiv.org/abs/2411.11295) find
  27. Edoardo Signoroni. 2022. Piötòst ché niènt, mèi piötòst-a manually revised lombard-italian parallel corpus. find
  28. RASLAN 2022 Recent Advances in Slavonic Natural Language Processing, page 105. find
  29. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024. A benchmark for learning to translate a new language from one grammar book. Preprint, arXiv:2309.16575. find
  30. Jörg Tiedemann. 2009. News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces, volume V, pages 237–248. find
  31. Francis M. Tyers, Hèctor Alòs i Font, Gianfranco Fronteddu, and Adrià Martín-Mor. 2017. Rule-based machine translation for the italian–sardinian language pair. The Prague Bulletin of Mathematical Linguistics, 108:221–232. find
  32. Stefaan van der Jeught. 2016. The protection of linguistic minorities in italy: A clean break with the past. Journal on Ethnopolitics and Minority Issues in Europe. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/signoroni-2026-can-llms-translate-italy-s/