Elamur Research⌕
☰
home / papers / 2024 / Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language

Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language

Paper · 2024 · Raphael Merx, Aso Mahmudi, Katrina Langford et al.

Summary

Mambai, an Austronesian language of Timor-Leste with about 200,000 native speakers, has almost no digital NLP resources. OPUS holds only 36 Tatoeba sentences, and the language is missing from FLORES-200. The authors test whether general-purpose LLMs can translate English into Mambai when given carefully chosen in-context material. They digitised Geoffrey Hull's 2001 language manual, which covers the southern (Ainaro) variety. Using OCR, they extracted two bilingual dictionaries and an aligned parallel sentence corpus. A native speaker also translated 50 new English sentences to serve as an independent test set. For each input, the prompt combines relevant dictionary entries with example pairs retrieved by TF-IDF and by semantic embeddings. The study compares LlaMa 2 70b, Mixtral 8x7B and GPT-4. Adding dictionary entries and mixing both retrieval methods clearly helps, and GPT-4 reaches moderate BLEU on held-out manual sentences. However, quality drops sharply on the native-speaker set. The work releases a first Mambai corpus. It also warns that testing on data from the same source as the prompt examples can badly overstate low-resource MT quality.

Key points

  • Data pipeline: The manual was digitised with ScanTailor and ABBYY FineReader 15, using Indonesian as the fallback OCR language. Dictionaries were then parsed into JSON with python-docx and regex: 1,790 English→Mambai entries and 1,592 Mambai→English entries.
  • Parallel corpus: Sentence pairs were aligned with Hunalign, which combines Gale-Church length cues with dictionary-based lexical similarity. After filtering at a score threshold of 0.2, 1,187 of 1,275 candidate pairs remained. The sentences are short, averaging about 5–6 words.
  • Experimental split: 119 manual sentences were held out for testing and 1,068 formed the retrieval pool. The extra native-speaker set has 50 sentences averaging 9 words, spread across many everyday domains.
  • Results: Best scores on manual sentences were about 21.2 BLEU in the abstract and up to 23.5 BLEU in the introduction (GPT-4, 10-shot, mixed retrieval). On the native-speaker set, BLEU stayed below 5 in every configuration, with a maximum of 4.4.
  • Release and limitations: Code and the native-speaker corpus are on GitHub (raphaelmerx/mambai), while the manual data is available on request. Evaluation relied only on automatic metrics, and the lack of a standard orthography likely lowered BLEU.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 3Research subjects and languages the work is about.Machine translation · 1,087 worksInformation retrieval & search · 473 worksLow-resource languages · 267 works

Cite

Cite as Raphael Merx, Aso Mahmudi, Katrina Langford, Leo Alberto de Araujo and Ekaterina Vylomova. 2024. Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language. arXiv:2404.04809.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.04809 · arXiv 2404.04809

@misc{merx2024low,
  title = {{Low-Resource Machine Translation through Retrieval-Augmented LLM Prompting: A Study on the Mambai Language}},
  author = {Raphael Merx, Aso Mahmudi, Katrina Langford, Leo Alberto {de Araujo} and Ekaterina Vylomova},
  year = {2024},
  doi = {10.48550/arxiv.2404.04809},
  eprint = {2404.04809},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.04809}
}

References

54 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. David Adelani, Jesujoba Alabi, Angela Fan, Julia Kreutzer, Xiaoyu Shen, Machel Reid, Dana Ruiter, Dietrich Klakow, Peter Nabende, Ernie Chang, Tajuddeen Gwadabe, Freshia Sackey, Bonaventure F. P. Dossou, Chris Emezue, Colin Leong, Michael Beukman, Shamsuddeen Muhammad, Guyo Jarso, Oreen Yousuf, Andr… — A Few Thousand Translations Go a Long Way! Leveraging Pre-trained Models for African News Translation
  2. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In-context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics.
  3. Duarte M. Alves, José Pombal, Nuno M. Guerreiro, Pedro H. Martins, João Alves, Amin Farajian, Ben Peters, Ricardo Rei, Patrick Fernandes, Sweta Agrawal, Pierre Colombo, José G. C. de Souza, and André F. T. Martins. 2024. Tower: An open multilingual large language model for translation-related tasks.
  4. Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Melvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. 2019. Massively multilingual neural machine translation in the wild: Findings and challenges.
  5. Berlie. 2008. Notes on east timor: Languages and education. Asian Journal of Social Science, 36(3-4):629–637. find
  6. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emi… — PaLM: Scaling Language Modeling with Pathways
  7. Grzegorz Chrupała. 2023. Putting natural in natural language processing. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7820–7827, Toronto, Canada. Association for Computational Linguistics. find
  8. Seamless Communication, Loïc Barrault, Yu-An Chung, Mariano Cora Meglioli, David Dale, Ning Dong, Paul-Ambroise Duquenne, Hady Elsahar, Hongyu Gong, Kevin Heffernan, John Hoffman, Christopher Klaiber, Pengwei Li, Daniel Licht, Jean Maillard, Alice Rakotoarison, Kaushik Ram Sadagopan, Guillaume Wenze… — SeamlessM4T: Massively Multilingual & Multimodal Machine Translation
  9. Xiangyu Duan, Baijun Ji, Hao Jia, Min Tan, Min Zhang, Boxing Chen, Weihua Luo, and Yue Zhang. 2020. Bilingual dictionary based neural machine translation without using parallel sentences. find
  10. Long Duong. 2017. Natural language processing for resource-poor languages. University of Melbourne. find
  11. Sergey Edunov, Myle Ott, Michael Auli, and David Grangier. 2018. Understanding back-translation at scale. find
  12. Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky, Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. 2020. Beyond english-centric multiling… find
  13. Helem Andressa de Oliveira Fogaça. 2013. Estudo fonético e fonológico do Mambae de Same: uma língua de Timor-Leste. Ph.D. thesis, University of Brasilia. find
  14. William A. Gale and Kenneth W. Church. 1993. A program for aligning sentences in bilingual corpora. Computational Linguistics, 19(1):75–102.
  15. Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. find
  16. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based phrase-level prompting of large language models for machine translation.
  17. Government of Timor-Leste. 2002. Constitution of the democratic republic of timor-leste. find
  18. Kirsty Sword Gusmão. 2023. The Key to Quality Inclusive Education in Timor-Leste’s Third Decade as an Independent Nation. [Accessed 27-02-2024]. find
  19. Aswanth Kumar, Ratish Puduppully, Raj Dabre, and Anoop Kunchukuttan. 2023. Ctqscorer: Combining multiple features for in-context example selection for machine translation. find
  20. Barry Haddow, Rachel Bawden, Antonio Valerio Miceli Barone, Jindřich Helcl, and Alexandra Birch. 2022. Survey of low-resource machine translation. Computational Linguistics, 48(3):673–732.
  21. Guillaume Lample, Alexis Conneau, Ludovic Denoyer, and Marc’Aurelio Ranzato. 2018. Unsupervised machine translation using monolingual corpora only.
  22. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? a comprehensive evaluation. find
  23. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  24. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  25. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. GlotLID: Language identification for low-resource languages. In The 2023 Conference on Empirical Methods in Natural Language Processing.
  26. Yafu Li, Yongjing Yin, Jing Li, and Yue Zhang. 2022. Prompt-driven neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2579–2590, Dublin, Ireland. Association for Computational Linguistics. find
  27. Rahul Mehta and Vasudeva Varma. 2023. LLM-RM at SemEval-2023 task 2: Multilingual complex NER using XLM-RoBERTa. In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 453–456, Toronto, Canada. Association for Computational Linguistics. find
  28. Yasmin Moslem, Rejwanul Haque, John D. Kelleher, and Andy Way. 2023. Adaptive machine translation with large language models.
  29. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of ChatGPT for machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5622–5633, Singapore. Association for Computational…
  30. Vineel Pratap, Andros Tjandra, Bowen Shi, Paden Tomasello, Arun Babu, Sayani Kundu, Ali Elkahky, Zhaoheng Ni, Apoorv Vyas, Maryam Fazel-Zarandi, Alexei Baevski, Yossi Adi, Xiaohui Zhang, Wei-Ning Hsu, Alexis Conneau, and Michael Auli. 2023. Scaling speech technology to 1,000+ languages. arXiv.
  31. Damien Kingsbury. 2010. National identity in timor-leste: challenges and opportunities. South East Asia Research, 18(1):133–159. find
  32. Ehud Reiter. 2018. A Structured Review of the Validity of BLEU. Computational Linguistics, 44(3):393–401.
  33. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin P… find
  34. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. Chat-GPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pages 392– 418, Singapore. Association for Computational Linguistics. find
  35. Rico Sennrich and Martin Volk. 2011. Iterative, MT-based sentence alignment of parallel texts. In Proceedings of the 18th Nordic Conference of Computational Linguistics (NODALIDA 2011), pages 175–182, Riga, Latvia. Northern European Association for Language Technology (NEALT). find
  36. Xiaofei Sun, Xiaoya Li, Jiwei Li, Fei Wu, Shangwei Guo, Tianwei Zhang, and Guoyin Wang. 2023. Text classification via large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8990–9005, Singapore. Association for Computational Linguistics.
  37. Garrett Tanzer, Mirac Suzgun, Eline Visser, Dan Jurafsky, and Luke Melas-Kyriazi. 2024. A benchmark for learning to translate a new language from one grammar book. In The Twelfth International Conference on Learning Representations. find
  38. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  39. Brian Thompson and Philipp Koehn. 2019. Vecalign: Improved sentence alignment in linear time and space. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). Association for…
  40. Jörg Tiedemann. 2009. News from OPUS - A Collection of Multilingual Parallel Corpora with Tools and Interfaces, volume V, pages 237–248. John Benjamins, Amsterdam/Philadelphia. find
  41. Timor-Leste General Directorate of Statistics. 2015. 2015 population and housing census. find
  42. Dániel Varga, Péter Halácsy, András Kornai, Viktor Nagy, László Németh, and Viktor Trón. 2007. Parallel corpora for medium density languages, page 247–258. John Benjamins Publishing Company. find
  43. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for translation: Assessing strategies and performance. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15406–1542… find
  44. Dr. Stephen L. Walter. 2016. The embli endline evaluation study. find
  45. Shuo Wang, Zhaopeng Tu, Zhixing Tan, Wenxuan Wang, Maosong Sun, and Yang Liu. 2021. Language models are good translators. find
  46. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024a. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations. find
  47. Haoran Xu, Amr Sharaf, Yunmo Chen, Weiting Tan, Lingfeng Shen, Benjamin Van Durme, Kenton Murray, and Young Jin Kim. 2024b. Contrastive preference optimization: Pushing the boundaries of llm performance in machine translation.
  48. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 41092–41110. PMLR.
  49. Haopeng Zhang, Xiao Liu, and Jiawei Zhang. 2023b. SummIt: Iterative text summarization via ChatGPT. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10644–10657, Singapore. Association for Computational Linguistics. find
  50. Timor-Leste General Directorate of Statistics. 2022. 2022 population and housing census. find
  51. Language Resource References Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundat…
  52. Gowda, Thamme and Zhang, Zhao and Mattmann, Chris and May, Jonathan. 2021. Many-to-English Machine Translation Tools, Data, and Pretrained Models. Association for Computational Linguistics. find
  53. Hull, Geoffrey. 2001. Mambai Language Manual: Ainaro Dialect. Sebastião Aparício da Silva Project. find
  54. NLLB Team and Marta R. Costa-jussà and James Cross and Onur Çelebi and Maha Elbayad and Kenneth Heafield and Kevin Heffernan and Elahe Kalbassi and Janice Lam and Daniel Licht and Jean Maillard and Anna Sun and Skyler Wang and Guillaume Wenzek and Al Youngblood and Bapi Akula and Loic Barrault and G… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/merx-2024-low-resource-machine-translation-through/