Elamur Research⌕
☰
home / papers / 2024 / Findings of WMT 2024 Shared Task on Low-Resource Indic Languages Translation

Findings of WMT 2024 Shared Task on Low-Resource Indic Languages Translation

Summary

This paper reports the results of the WMT 2024 shared task on machine translation for low-resource languages of northeastern India. These languages have few digital resources, little institutional support and complex scripts or morphology. The task follows the 2023 edition and keeps its four pairs: English with Assamese, Mizo, Khasi and Manipuri, in both directions. Training data came from the IndicNE-Corp1.0 parallel and monolingual collection. The organisers built a new test set that is harder than the previous year's. For Khasi, the Sohra (Cherra) dialect was used as the standard variety.

Submissions were scored with five automatic metrics (BLEU, TER, RIBES, METEOR, ChrF) and with human judgements of Adequacy, Fluency and Overall Quality. The most successful approach fine-tuned a large pretrained multilingual model and enlarged the training data with back-translated monolingual text. Scores stayed low overall, which shows these pairs remain hard. The task matters because it offers a shared benchmark for under-supported languages and a basis for growing toward more regional languages.

Key points

  • Eleven teams submitted English–Khasi systems, including DLUT-NLP, HW-TSC, SPRING-IITM, JUNLP and MTNLP-IIITH. Teams could send primary and contrastive runs.
  • SPRING-IITM ranked first for English–Mizo and English–Khasi in both directions. Its method was back translation plus fine-tuning NLLB 3.3B, with special tokens and a custom corpus for Khasi.
  • HW-TSC was strongest for Manipuri→English. A contrastive run that applied LoRA fine-tuning to a pretrained translation model improved its results further.
  • The best English–Khasi runs reached only about 0.12 / 0.10 BLEU (0–1 scale) and about 0.45 / 0.43 ChrF. Their human Overall Quality ratings were close to 5.
  • Planned future pairs are English with Nyishi, Bodo, Mising and Kokborok, along with larger corpora.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.wmt-1.54

@inproceedings{pakray2024findings,
  title = {{Findings of WMT 2024 Shared Task on Low-Resource Indic Languages Translation}},
  author = {Pakray, Partha and Pal, Santanu and Vetagiri, Advaitha and Krishna, Reddi and Maji, Arnab Kumar and Dash, Sandeep and Laitonjam, Lenin and Sarah, Lyngdoh and Manna, Riyanka},
  booktitle = {Proceedings of the Ninth Conference on Machine Translation},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.54},
  url = {https://doi.org/10.18653/v1/2024.wmt-1.54}
}

References

27 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. James F. Allen. 2003. Natural language processing, page 1218–1222. John Wiley and Sons Ltd., GBR. find
  2. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Ass… find our card
  3. Chenfei Ju, Junpeng Liu, Kaiyu Huang, and Degen Huang. 2024. Dlut-nlp machine translation systems for wmt24 low-resource indic language translation. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024,… find
  4. Vanlalmuansangi Khenglawt, Sahinur Rahman Laskar, Santanu Pal, Partha Pakray, and Ajoy Kumar Khan. 2022. Language resource building and English-to-mizo neural machine translation encountering tonal words. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Confe… find
  5. Lenin Laitonjam and Sanasam Ranbir Singh. 2021. Manipuri-English machine translation using comparable corpus. In Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), pages 78–88, Virtual. Association for Machine Translation in the Americas. find
  6. Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji Darsh Kaushik, Partha Pakray, and Sivaji Bandyopadhyay. 2021. EnKhCorp1.0: An English– Khasi corpus. In Proceedings of the 4th Workshop on Technologies for MT of Low Resource Languages (LoResMT2021), pages 89–95, Virtual. Association for Machine… find
  7. Yash Bhaskar and Parameswari Krishnamurthy. 2024. Yes-mt’s submission to the low-resource indic language translation shared task in wmt 2024. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miami,… find
  8. Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji, Partha Pakray, and Sivaji Bandyopadhyay. 2020. EnAsCorp1.0: English-Assamese corpus. In Proceedings of the 3rd Workshop on Technologies for MT of Low Resource Languages, pages 62–68, Suzhou, China. Association for Computational Linguistics. find
  9. Neha Gajakos, Prashanth Nayak, Rejwanul Haque, and Andy Way. 2024. Adapt-mt submissions to wmt 2024 low-resource indic language translation task. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Mia… find
  10. Sahinur Rahman Laskar, Riyanka Manna, Partha Pakray, and Sivaji Bandyopadhyay. 2022. A domain specific parallel corpus and enhanced english-assamese neural machine translation. Computación y Sistemas, 26(4):1669–1687. find
  11. Loitongbam Gyanendro Singh, Lenin Laitonjam, and Sanasam Ranbir Singh. 2016. Automatic syllabification for Manipuri language. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 349–357, Osaka, Japan. find
  12. The COLING 2016 Organizing Committee. find
  13. Barry Haddow and Faheem Kirefu. 2020. Pmindia – a collection of parallel corpora of languages of india. find
  14. Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, and Hajime Tsukada. 2010. Automatic evaluation of translation quality for distant language pairs. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing, pages 944– 952, Cambridge, MA. Association for Compu… find our card
  15. Abhinav P M, Ketaki Shetye, and Parameswari Krishnamurthy. 2024. Mtnlp-iiith: Machine translation for low-resource indic languages. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miami, Florida, U… find
  16. Santanu Pal, Partha Pakray, Sahinur Rahman Laskar, Lenin Laitonjam, Vanlalmuansangi Khenglawt, Sunita Warjri, Pankaj Kundan Dadure, and Sandeep Kumar Dash. 2023. Findings of the WMT 2023 shared task on low-resource Indic language translation. In Proceedings of the Eighth Conference on Machine Transl… find our card
  17. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  18. Pranamya Patil, Raghavendra HR, Aditya Raghuwanshi, and Kushal Verma. 2024. Srib-nmt’s submission to the indic mt shared task in wmt 2024. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miami, Flo… find
  19. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  20. Bin Wei, Jiawei Zhen, Zongyao Li, Zhanglin Wu, Jiaxin Guo, Daimeng Wei, Zhiqiang Rao, Shaojun Li, Yuanchang Luo, Hengchao Shang, Jinlong Yang, Yuhao Xie, and Hao Yang. 2024. Machine translation advancements of low-resource indian languages by transfer learning. In Proceedings of the Low-Resource Ind… find
  21. Saumitra Yadav, Ananya Mukherjee, and Manish Shrivastava. 2024. A3-108 controlling token generation in low resource machine translation systems. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miam… find
  22. Matthew Theodore Roquea, Carlos Rafael Catalana, Dan John Velascoa, Manuel Antonio Rufinoa, and Jan Christian Blaise Cruza. 2024. Samsung r&d institute philippines @ wmt 2024 indic mt task. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translati… find
  23. Pramit Sahoo, Maharaj Brahma, and Maunendra Sankar Desarkar. 2024. Nlip-lab-iith low-resource mt system for wmt24 indic mt shared task. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miami, Florid… find
  24. Hamees Sayed, Advait Joglekar, and Srinivasan Umesh. 2024. Spring lab iitm’s submission to low resource indic language translation shared task. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miami… find
  25. Loitongbam Gyanendro Singh and Sanasam Ranbir Singh. 2017. Word polarity detection using syllable features for manipuri language. In 2017 International Conference on Asian Language Processing (IALP), pages 206–209. find
  26. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachu… find our card
  27. Noel Murphy Sören Dréano, Derek Molloy. 2024. Exploration of the cyclegn framework for low-resource languages. In Proceedings of the Low-Resource Indic Language Translation Shared Task, 9th Conference on Machine Translation (WMT24), EMNLP, November 15-16, 2024, Miami, Florida, USA. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/pakray-2024-findings-wmt-2024-shared-task/