Elamur Research⌕
☰
home / papers / 2026 / Anchoring the Judge: Curriculum-Based Adaptation and Reference-Anchored MQM for LLM-Based Machine Translation of an Unseen Low-Resource Language - A Case of Nupe

Anchoring the Judge: Curriculum-Based Adaptation and Reference-Anchored MQM for LLM-Based Machine Translation of an Unseen Low-Resource Language - A Case of Nupe

Summary

The paper asks whether a decoder-only LLM can learn to translate a language it never saw in pre-training. The test case is Nupe, a Niger–Congo language with millions of speakers that is almost absent from benchmarks and web corpora. The authors propose NupeMT-QLoRA, which fine-tunes LLaMA-2 7B with QLoRA in two curriculum stages. The first stage uses a large, noisy Nupe–English parallel corpus. The second uses a smaller, cleaned bidirectional set in which each example carries a tag for translation direction.

The authors also show that popular automatic judges such as GEMBA and xCOMET agree poorly with human ratings on Nupe. They attribute this to fluency bias: the judges reward fluent output even when its meaning is wrong. As an alternative they introduce Ref-Anchor-MQM. In this protocol, the judge first lists the Key Semantic Units of a human reference and then scores the translation against them.

The adapted model clearly outperforms NLLB-200, and the anchored metric tracks human judgments much more closely. Together they offer a practical pipeline for training and evaluating MT systems for languages with no prior representation in LLMs.

Key points

  • Curriculum data:
    • Stage 1 uses about 34k noisy sentence pairs (about 820k tokens), drawn from roughly 20k pairs of natural Nupe texts, 6.6k Bible and Qur'an pairs, and about 7.4k pairs from other curated sources.
    • Stage 2 uses about 12k clean, direction-tagged bidirectional pairs.
  • Translation quality: chrF++ rises from 22.73 (NLLB-200) to 41.10 (NupeMT-QLoRA).
  • Existing judges: On 106 sampled test sentences, GEMBA and xCOMET correlate weakly with human Direct Assessment (Kendall's τ ≈ 0.18–0.21) and show low inter-rater reliability (Fleiss' κ ≈ 0.23–0.27).
  • Proposed metric: Ref-Anchor-MQM reaches τ = 0.71 against human judgments. Its main limitation is that it depends on human reference translations.
  • Resources: Replication data and code are on Mendeley Data (k7dtv7k2hy). The adapters and Ref-Anchor-MQM code are promised on GitHub (futminna-ai-lab). The study is limited to one model (LLaMA-2 7B) and one language pair.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.loreslm-1.19

@inproceedings{umar2026anchoring,
  title = {{Anchoring the Judge: Curriculum-Based Adaptation and Reference-Anchored MQM for LLM-Based Machine Translation of an Unseen Low-Resource Language - A Case of Nupe}},
  author = {Umar, Umar Baba and Bashir, Sulaimon Adebayo and Mohammed, Abdulmalik Danlami},
  booktitle = {Proceedings of the Second Workshop on Language Models for Low-Resource Languages (LoResLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.loreslm-1.19},
  url = {https://doi.org/10.18653/v1/2026.loreslm-1.19}
}

References

33 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Aleman, Diogo Almeida, and ... 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774. ↗ our card
  2. David Ifeoluwa Adelani, Jesujoba O. Alabi, Angela Fan, Julia Kreutzer, Samuel Olowobo, Ituen Okayo, Bernard Opoku, Chiamaka Chukwuneke, and 1 others. 2022. A few-shot learning approach for low-resource african languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 904–917. find
  3. Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, José Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, and Pierre Colombo. 2024. Tower: An open multilingual large language model for translation-related tasks. Computing Research Repository, arXiv:2402.17733. ↗ our card
  4. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proceedings of the 26th International Conference on Machine Learning (ICML 2009), pages 41–48. ACM. find our card
  5. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, and 12 others. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33, pages 1877– 1901. find
  6. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRA: Efficient finetuning of quantized LLMs. arXiv preprint arXiv:2305.14314. ↗
  7. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024a. QLoRA: Efficient fine-tuning of quantized LLMs. Computing Research Repository, arXiv:2305.14314. ↗
  8. Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2024b. QLoRA: Efficient fine-tuning of quantized LLMs. Computing Research Repository, arXiv:2305.14314. ↗
  9. Nuno M. Guerreiro, Matt Post, and André F. T. Martins. 2024. xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:1001–1018. ↗ our card
  10. Md Mehedi Hasan, Shuo Ma, Mohammad S. Khan, and 1 others. 2024. Unlocking parameter-efficient fine-tuning for low-resource language translation. In Findings of the Association for Computational Linguistics: NAACL 2024, pages 4152–4166, Mexico City, Mexico. Association for Computational Linguistics. find
  11. Edward J. Hu, Yallen Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR). find
  12. Ziyang Hui, Chong Feng, and Tianfu Zhang. 2022. Review-based curriculum learning for neural machine translation. In Machine Translation: 18th China Conference, CCMT 2022, Lhasa, China, August 4–6, 2022, Revised Selected Papers, pages 24– 36. Springer. find
  13. Nisar Hussain, Amna Qasim, Gull Mehak, Muhammad Zain, Momina Hafeez, and Grigori Sidorov. 2025. Fine-tuning large language models with qlora for offensive language detection in roman urdu-english code-mixed text. arXiv preprint. Available at arXiv:2510.03683. ↗
  14. Omkar Khade, Shruti Jagdale, Abhishek Phaltankar, Gauri Takalikar, and Raviraj Joshi. 2024. Challenges in adapting multilingual llms to low-resource languages using lora peft tuning. arXiv preprint. Available at arXiv:2411.18571. ↗
  15. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. ↗
  16. Zhanibek Kozhirbayev and Zhandos Yessenbayev. 2025. Fine-tuning methods and dataset structures for multilingual neural machine translation: A kazakh–english–russian case study in the it domain. Electronics, 14(15):3126. ↗
  17. S. Lankford, H. Afli, and A. Way. 2025. adaptmllm: Fine-tuning multilingual language models on low-resource languages with integrated llm playgrounds. Information, 14(12):638. find
  18. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality metrics. In Proceedings of the LT-Innovate Summit. ↗
  19. LonWeb. 2017. The night watch (french-english parallel text). http://www.lonweb.org/daisy/ ds-frenchnightwatch.htm. Accessed January 2026. ↗
  20. Quentin Meeus, Marie-Francine Moens, and Hugo Van hamme. 2023. Bidirectional representations for low-resource spoken language understanding. Applied Sciences, 13(20):11291. find
  21. Aislu Mukhambetkalieva. 2024. New reference-based metrics for MT evaluation: MetricX, GEMBA-MQM, and CometKiwi-XL. Intento Blog. find
  22. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, and 13 others. 2022. No language left behind: Scaling human-centered machine translation to 200 languages. Computing Research Repository, arXiv:2207.04672. ↗
  23. Femi Oladipupo. 2025. A beginner’s guide to fine-tuning AI models on african datasets. Medium. find
  24. Leiyu Pan, Yongqi Leng, and Deyi Xiong. 2024. Can large language models learn translation robustness from noisy-source in-context demonstrations? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2798–2808, Torino, Italia. ELRA and ICCL. find
  25. Li Pan, Yufeng Leng, and Deyi Xiong. 2025. The noise sensitivity of LLMs in zero-shot translation tasks. Findings of the Association for Computational Linguistics: EMNLP 2025. find
  26. Renhao Pei, Yihong Liu, Peiqin Lin, François Yvon, and Hinrich Schütze. 2025. Understanding in-context machine translation for low-resource languages: A case study on Manchu. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8767–8788, Vienna, Austria. Association for Computational Linguistics. find
  27. Taido Purason, Hele-Andra Kuulmets, and Mark Fishel. 2024. Llms for extremely low-resource finno-ugric languages. Preprint, arXiv:2410.18902. Md. ↗
  28. Atiqur Rahman, Sabrina Islam, and Mushfiqul Haque Omi. 2025. Llm-based evaluation of low-resource machine translation: A reference-less dialect guided approach with a refined sylheti-english benchmark. Preprint, arXiv:2505.12273. ↗
  29. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luísa Coheur, José G. C. de Souza, and André F. T. Martins. 2023. Scaling up cometkiwi: Unbabel-IST 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 841–848, Singapore. Association for Computational Linguistics. find our card
  30. Archchana Sindhujan, Diptesh Kanojia, Constantin Orăsan, and Shenbin Qian. 2025. When llms struggle: Reference-less translation evaluation for low-resource languages. In Proceedings of the First Workshop on Language Models for Low-Resource Languages (LoResLM 2025), pages 437–459, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. ↗
  31. Abhinav Singh, Nishant Singh, and Shreyas Vatsal. 2024. Robustness of LLMs to perturbations in text. Computing Research Repository, arXiv:2407.08989. 19 others. 2023b. Llama 2: Open foundation and fine-tuned chat models. Computing Research Repository, arXiv:2307.09288. ↗
  32. Chen Xu, Bojie Hu, Yufan Jiang, Kai Feng, Zeyang Wang, Shen Huang, Qi Ju, Tong Xiao, and Jingbo Zhu. 2024a. Dynamic curriculum learning for low-resource neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics (COLING 2024), pages 3977– 3989. find
  33. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen shifts, Benjamin Van Durme, and Yong-Yeol Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of LLM performance in machine translation. Computing Research Repository, arXiv:2401.08417. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/umar-2026-anchoring-judge-curriculum-based-adaptation/