Elamur Research⌕
☰
home / papers / 2026 / Towards the First NLP Benchmark for Ladin - an Extremely Low-Resource Language

Towards the First NLP Benchmark for Ladin - an Extremely Low-Resource Language

Paper · 2026 · Ulin Nuha and Adam Jatowt

Summary

Ladin is an endangered Rhaeto-Romance language of Northern Italy with about 30,000 speakers. It has five quite different dialects and almost no NLP resources beyond a small Italian–Ladin machine translation benchmark. The authors focus on the Val Badia variety, which has the most accessible data. They first compare translation strategies on existing parallel data: few-shot prompting of LLMs such as GPT-4 versus fine-tuning models such as NLLB. They then use translation to convert labeled Italian corpora into Ladin, covering sentiment analysis and multiple-choice question answering. To control quality, Ladin outputs are filtered, back-translated into Italian, and filtered a second time. Adding this synthetic parallel data to the original training pairs makes a fine-tuned NLLB model better than the earlier benchmark in both translation directions on BLEU, ROUGE and chrF++. That earlier benchmark used RBMT-based augmentation. The paper also releases the first Ladin SA and MCQA datasets with baseline scores. These give a starting point for work on Ladin and other extremely low-resource languages.

Key points

  • Translation data: The baseline MT data comes from Frontull and Moser (2024).
    • Training set AD Ita_Lad: 18,139 short Italian–Ladin pairs, about 5 words per sentence.
    • Test subsets: Legal (424), Historical-Legal (833) and Literary (1,563).
  • Sentiment source data: 41,077 Italian Tripadvisor reviews (Desole, 2020).
    • Reviews longer than the third quartile were dropped, leaving 30,712.
    • GPT-4 corrected grammatical errors before translation.
  • Question answering source data: an Italian MCQA set of over 5,000 hand-written questions (Rinaldi et al., 2024). Items with 2 or 6 answer options were removed.
  • Synthetic data quality: Semantic similarity between Italian and Ladin is 86.61 in the synthetic SD Ita_Lad and 85.75 in human-translated data. This suggests the synthetic pairs are close in quality to human translations.
  • Task benchmarks: Few-shot LLaMA 3.1 70B scored best overall, ahead of m-DistilBERT, XLM-RoBERTa and mT5-small.
    • On Ladin SA it reached 93.99 balanced accuracy and 96.07 F1.
    • Ladin MCQA stayed hard at about 44 accuracy, versus about 55 for Italian.
    • Code is available at github.com/ulinnuhaha/NLP_Ladin.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Toyota Technological InstituteNagoya · Japan9 works · 3,330 citations🎓Universität InnsbruckInnsbruck · Austria8 works · 737 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksLow-resource languages · 267 works

Cite

Cite as Ulin Nuha and Adam Jatowt. 2026. Towards the First NLP Benchmark for Ladin - an Extremely Low-Resource Language. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.55

@inproceedings{nuha2026first,
  title = {{Towards the First NLP Benchmark for Ladin - an Extremely Low-Resource Language}},
  author = {Ulin Nuha and Adam Jatowt},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.55},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.55}
}

References

47 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jesujoba O. Alabi, David Ifeoluwa Adelani, Marius Mosbach, and Dietrich Klakow. 2022. Adapting pre-trained language models to african languages via multilingual adaptive fine-tuning. In Proceedings of the 29th International Conference on Computational Linguistics, page 4336–4349.
  2. Sina Bagheri Nezhad, Ameeta Agrawal, and Rhitabrat Pokharel. 2025. Beyond data quantity: Key factors driving performance in multilingual language models. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, page 225–239. find
  3. Steven Bird. 2020. Decolonising speech and language technology. In Proceedings of the 28th International Conference on Computational Linguistics, page 3504–3519.
  4. Samuel Cahyawijaya, Holy Lovenia, and Pascale Fung. 2024. Llms are few-shot in-context low-resource language learners. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, volume 1, page 405–433.
  5. Shweta Chauhan, Shefali Saxena, and Philemon Daniel. 2022. Improved unsupervised neural machine translation with semantically weighted back translation for morphologically rich and low resource languages. Neural Processing Letters, 54:1707–1726. find
  6. Qi Cheng, Liqiong Chen, Zhixing Hu, Juan Tang, Qiang Xu, and Binbin Ning. 2024. A novel prompting method for few-shot ner via llms. Natural Language Processing Journal, 8(100099). find
  7. Google Cloud. 2025. Evaluate models. Accessed September 5, 2025. find
  8. Anthony Thomas Connor. 2023. Ladin Perspectives on Language and Identity in the Central Dolomites of Northern Italy. Ph.d. thesis, The University of Sheffield, Faculty of Arts and Humanities, School of Languages and Cultures. find
  9. Alessandro Desole. 2020. Sentiment analysis on tripadvisor reviews. Accessed: 2025-01-23. find
  10. Silvia Erardi, Ronald L. Gardner, and Sara Comploi. 2022. Anglicisms in ladin: Loanwords and local perceptions. Forum Italicum, 56(2):272—-306. find
  11. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2022. Language-agnostic bert sentence embedding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, volume 1, pages 878 – 891.
  12. Samuel Frontull and Georg Moser. 2024. Rule-based, neural and llm back-translation: Comparative insights from a variant of ladin. In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024), page 128–138. find
  13. Andrew Gambardella, Yusuke Iwasawa, and Yutaka Matsuo. 2024. Language models do hard arithmetic tasks easily and hardly do easy arithmetic tasks. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 2, page 85–91. find
  14. Sofía García. 2024. Enhaced apertium system: Translation into low-resource languages of spain spanish–asturian. In Proceedings of the Ninth Conference on Machine Translation, page 878–884. find
  15. Ali Hatami, Shubhanker Banerjee, Mihael Arcan, Paul Buitelaar, and John Philip McCrae. 2024. English-to-low-resource translation: A multimodal approach for hindi, malayalam, bengali, and hausa. In Proceedings of the Ninth Conference on Machine Translation, page 815–822. find
  16. Jyotsana Khatri and Pushpak Bhattacharyya. 2020. Filtering back-translated data in unsupervised neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics, page 4334–4339. find
  17. Seong Hoon Lim, Taejun Yun, Jinhyeon Kim, Jihun Choi, and Taeuk Kim. 2024. Analysis of multi-source language training in cross-lingual transfer. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, page 712–725. find
  18. Wuying Liu, Wei Li, and Lin Wang. 2023. Multiloop incremental bootstrapping for lowresource machine translation. In Proceedings of Machine Translation Summit XIX, page 1–11. find
  19. Arle Lommel, Serge Gladkoff, Alan Melby, Sue Ellen Wright, Ingemar Strandvik, Katerina Gasova, Angelika Vaasa, Andy Benzo, Romina Marazzato Sparano, Monica Foresi, Johani Innis, Lifeng Han, and Goran Nenadic. 2024. The multi-range theory of translation quality measurement: Mqm scoring models and sta… find
  20. Arle Richard Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35.
  21. Kaiwen Lu, Yating Yang, Fengyi Yang, Rui Dong, Bo Ma, Aihetamujiang Aihemaiti, Abibilla Atawulla, Lei Wang, and Xi Zhou. 2025. Low-resource language expansion and translation capacity enhancement for llm: A study on the uyghur. In Proceedings of the 31st International Conference on Computational Lin… find
  22. Luca Melchior. 2023. Raeto-romance: Romansh, ladin, friulian. Linguistics. find
  23. Hojjat Mokhtarabadi, Ziba Zamani, Abbas Maazallahi, and Mohammad Hossein Manshaei. 2025. Empowering persian llms for instruction following: A novel dataset and training approach. In Proceedings of the First Workshop on Language Models for Low-Resource Languages, page 31–67. find
  24. Ana Alexandra Morim da Silva, Nikit Srivastava, Tatiana Moteu Ngoli, Michael Röder, Diego Moussallem, and Axel-Cyrille Ngonga Ngomo. 2024. find
  25. Benchmarking low-resource machine translation systems. In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024), page 175–185. find
  26. Rudra Murthy, Anoop Kunchukuttan, and Pushpak Bhattacharyya. 2019. Addressing word-order divergence in multilingual neural machine translation for extremely low resource languages. In Proceedings of NAACL-HLT 2019, page 3868–3873. find
  27. Xuan-Phi Nguyen, Sharifah Mahani Aljunied, Shafiq Joty, and Lidong Bing. 2024. Democratizing llms for low-resource languages by leveraging their english dominant abilities with linguistically-diverse prompts. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics,…
  28. Trinh Pham, Khoi M. Le, and Luu Anh Tuan. 2024. Unibridge: A unified approach to cross-lingual transfer learning for low-resource languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics, volume 1, page 3168–3184. find
  29. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191.
  30. Taido Purason, Hele-Andra Kuulmets, and Mark Fishel. 2024. Llms for extremely low-resource finno-ugric languages. arXiv preprint arXiv:2410.18902. Cs.CL, version 1. find
  31. Pawan Kumar Rajpoot, Nagraj N Bhat, and Ashish Shrivastava. 2024. Multimodal machine translation for low-resource indic languages: A chain-of-thought approach using large language models. In Proceedings of the Ninth Conference on Machine Translation, page 833–838. find
  32. Surangika Ranathunga, En-Shiun Annie Lee, Marjana Prifti Skenduli, Ravi Shekhar, Mehreen Alam, and Rishemjit Kaur. 2023. Neural machine translation for low-resource languages: A survey. ACM Computing Surveys, 55(11):1–37. find
  33. Matteo Rinaldi, Jacopo Gili, Maria Francis, Mattia Gof-fetti, Viviana Patti, and Malvina Nissim. 2024. Multiple choice questions on multiple topics in italian: A calamita challenge. In CLiC-it 2024: Tenth Italian Conference on Computational Linguistics. find
  34. Peng Shu, Junhao Chen, Zhengliang Liu, Hui Wang, Zihao Wu, Tianyang Zhong, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Constance Owl, Xiaoming Zhai, Ninghao Liu, Claudio Saunt, and Tianming Liu. 2024. Transcending language boundaries: Harnessing llms for low-resource language translation… — [arXiv:2411.11295](https://arxiv.org/abs/2411.11295) find
  35. Tong Su, Xin Peng, Sarubi Thillainathan, David Guzmán, Surangika Ranathunga, and En-Shiun Annie Lee. 2024. Unlocking parameter-efficient fine-tuning for low-resource language translation. In Findings of the Association for Computational Linguistics: NAACL 2024, page 4217–4225. find
  36. Felipe Sánchez-Martínez, Juan Antonio Pérez-Ortiz, Aarón Galiano-Jiménez, and Antoni Oliver. 2024. Findings of the wmt 2024 shared task translation into low-resource languages of spain: Blending rule-based and neural systems. In Proceedings of the Ninth Conference on Machine Translation, page 684–69… find
  37. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  38. Khanh-Tung Tran, Barry O’Sullivan, and Hoang D. Nguyen. 2024. Irish-based large language model with extreme low-resource settings in machine translation. In Proceedings of the Seventh Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT 2024), page 193–202. find
  39. Sami Ul Haq, Rudali Huidrom, and Sheila Castilho. 2024. Dcu adapt at wmt24: English to low-resource multi-modal translation task. In Proceedings of the Ninth Conference on Machine Translation, page 810–814. find
  40. UNESCO. 2010. Atlas of the World’s Languages in Danger. UNESCO Publishing, Paris, France. find
  41. Gorka Urbizu, Inaki San Vicente, Xabier Saralegi, Rodrigo Agerri, and Aitor Soroa. 2022. Basqueglue: A natural language understanding benchmark for basque. In Proceedings of the 13th Conference on Language Resources and Evaluation (LREC 2022), page 1603–1612. find
  42. Giovanni Valer, Nicolò Penzo, and Jacopo Staiano. 2024. Nesciun lengaz lascià endò: Machine translation for fassa ladin. In Proceedings of CLiC-it 2024: Tenth Italian Conference on Computational Linguistics. find
  43. Yuxia Wang, Minghan Wang, Muhammad Arslan Manzoor, Fei Liu, Georgi Georgiev, Rocktim Jyoti Das, and Preslav Nakov. 2024. Factuality of large language models: A survey. In Proceedings of the 2024
  44. Conference on Empirical Methods in Natural Language Processing, page 19519–19529. find
  45. Zheng Xin Yong, Cristina Menghini, and Stephen Bach. 2024. Lexc-gen: Generating data for extremely low-resource languages with large language models and bilingual lexicons. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13990–14009. find
  46. Liang Zhang, Katherine Jijo, Spurthi Setty, Eden Chung, Fatima Javid, Natan Vidra, and Tommy Clifford. 2024. Enhancing large language model performance to answer questions and extract information more accurately. arXiv preprint 2402.01722. Cs.CL, version 1. — [arXiv:2402.01722](https://arxiv.org/abs/2402.01722) find
  47. Xuan Zhang, Navid Rajabi, Kevin Duh, and Philipp Koehn. 2023. Machine translation with large language models: Prompting, few-shot learning, and fine-tuning with qlora. In Proceedings of the Eighth Conference on Machine Translation, page 468–481. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nuha-2026-first-nlp-benchmark-ladin-extremely/