Elamur Research⌕
☰
home / papers / 2024 / From LLM to NMT: Advancing Low-Resource Machine Translation with Claude

From LLM to NMT: Advancing Low-Resource Machine Translation with Claude

Paper · 2024 · Maxim Enis and Mark Hopkins

Summary

Earlier studies found that LLMs such as GPT-4 translate well between high-resource languages but trail dedicated NMT systems like NLLB-54B on low-resource ones. Enis and Hopkins revisit this with Claude 3 Opus. Because Claude appears to have seen FLORES-200 during training, they build new test sets dated after the model's training cutoff. These consist of sentence pairs mined from aligned BBC News articles in 36 languages paired with English, using shared images as an alignment cue, plus Maltese–English Common Voice data. Measured with spBLEU and chrF++, Claude beats NLLB-54B and Google Translate on a sizeable share of pairs, mostly when translating into English. Its quality also degrades less on scarce-data languages than that of eight other LLMs. When translating out of English, specialized systems still generally win. The authors then distill Claude into a compact model. Claude translates whole Yoruba news documents in one prompt, and NLLB-1.3B is fine-tuned on the resulting corpus. The student matches or beats NLLB-54B and Google Translate on Yoruba–English. This shows that LLM gains can be moved cheaply into deployable MT systems.

Key points

  • Languages: The evaluation covers English plus 36 languages, split into 15 high-resource, 17 low-resource and 4 very low-resource. Every pair is tested in both directions on 100-sentence samples from FLORES-200 devtest, mined BBC News, and Maltese speech data from Common Voice (IWSLT 2024).
  • Results on unseen data:
    • Overall, Claude 3 Opus outperforms NLLB-54B on 44% of pairs and Google Translate on 22%.
    • Into English, it beats NLLB-54B on 55.6% of pairs.
    • Out of English, it beats NLLB-54B on only 33.3% of pairs.
  • Document-level prompting: Batching all sentences of a document into one prompt lowers API cost and improves quality. On Yoruba–English BBC News, claude_doc reaches 26.17 spBLEU / 47.06 chrF++, compared with 21.15 / 43.78 for sentence-level prompting.
  • Distillation data:
    • NLLB-54B forward-translated 53,193 Yoruba sentences to train a base model, combined with MENYO-20k.
    • Claude translated 431 documents (7,996 sentences) for $46.06.
  • Distillation result: claude_distill scores 22.61 spBLEU / 44.15 chrF++. This beats NLLB-54B (22.51 / 43.26), Google Translate (21.09 / 41.99) and google_distill (20.81 / 42.29), which shows that the quality of synthetic data matters.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Williams CollegeWilliamstown · United States3 works · 462 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksLow-resource languages · 267 works

Cite

Cite as Maxim Enis and Mark Hopkins. 2024. From LLM to NMT: Advancing Low-Resource Machine Translation with Claude. arXiv:2404.13813.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2404.13813 · arXiv 2404.13813

@misc{enis2024llm,
  title = {{From LLM to NMT: Advancing Low-Resource Machine Translation with Claude}},
  author = {Maxim Enis and Mark Hopkins},
  year = {2024},
  doi = {10.48550/arxiv.2404.13813},
  eprint = {2404.13813},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2404.13813}
}

References

34 references, 19 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Kevin Heffernan, Onur Çelebi, and Holger Schwenk. 2022. Bitext mining using distilled sentence representations for low-resource languages. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2101–2112, Abu Dhabi, United Arab Emirates. Association for Computational Linguis…
  3. David Adelani, Dana Ruiter, Jesujoba Alabi, Damilola Adebonojo, Adesina Ayeni, Mofe Adeyemi, Ayodele Esther Awokoya, and Cristina España-Bonet. 2021. The effect of domain and diacritics in Yoruba– English neural machine translation. In Proceedings of Machine Translation Summit XVIII: Research Track,… find
  4. Carlos Daniel Hernandez Mena, Albert Gatt, Andrea DeMarco, Claudia Borg, Lonneke van der Plas, Amanda Muscat, and Ian Padovani. 2020. MASRI-HEADSET: A Maltese corpus for speech recognition. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6381–6388, Marseille, France…
  5. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
  6. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network.
  7. Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. 2023. Quantifying memorization across neural language models. find
  8. Jang Hyun Cho and Bharath Hariharan. 2019. On the efficacy of knowledge distillation. CoRR, abs/1910.01348. find
  9. Raj Dabre and Atsushi Fujita. 2020. Combining sequence distillation and transfer learning for efficient low-resource neural machine translation models. In Proceedings of the Fifth Conference on Machine Translation, pages 492–502, Online. Association for Computational Linguistics. find
  10. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255. find
  11. David M. Eberhard, Gary F. Simons, and Charles D. Fennig. 2024. Yoruba. In David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors, Ethnologue: Languages of the World, 27 edition. SIL International, Dallas, Texas. find
  12. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. find
  13. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. Two new evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. find
  14. Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, E… — Training Compute-Optimal Large Language Models
  15. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. Is chatgpt a good translator? yes with gpt-4 as the engine.
  16. Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models.
  17. Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. In Proceedings of the Eighth Conference on Machine Translation, pages 419–451, Singapore. Association for Computational Linguistics.
  18. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation.
  19. Yeskendir Koishekenov, Alexandre Berard, and Vassilina Nikoulina. 2023. Memory-efficient NLLB-200: Language-specific expert pruning of a massively multilingual machine translation model. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers… find
  20. Jiahuan Li, Shanbo Cheng, Shujian Huang, and Jiajun Chen. 2024. Mt-patcher: Selective and extendable knowledge distillation from large language models for machine translation. find
  21. Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. find
  22. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.… — Crosslingual Generalization through Multitask Finetuning
  23. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational…
  24. Maja Popović. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation, pages 612–618, Copenhagen, Denmark. Association for Computational Linguistics.
  25. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
  26. Nathaniel Robinson, Perez Ogayo, David R. Mortensen, and Graham Neubig. 2023. ChatGPT MT: Competitive for high- (but not low-) resource languages. In Proceedings of the Eighth Conference on Machine Translation, pages 392–418, Singapore. Association for Computational Linguistics.
  27. Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre. 2023. NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singap…
  28. David Stap and Ali Araabi. 2023. ChatGPT is not a good indigenous translator. In Proceedings of the Workshop on Natural Language Processing for Indigenous Languages of the Americas (AmericasNLP), pages 163–167, Toronto, Canada. Association for Computational Linguistics. find
  29. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hof… find
  30. Barak Turovsky. 2016. Ten years of google translate. https://blog.google/products/translate/ ten-years-of-google-translate/. Accessed: 2024-04-11. ↗
  31. Abraham Wald. 1943. Tests of statistical hypotheses concerning several parameters when the number of observations is large. Transactions of the American Mathematical society, 54(3):426–482. find
  32. Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shuming Shi, and Zhaopeng Tu. 2023. Document-level machine translation with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 16646–16661, Singapore. Association for Co…
  33. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study.
  34. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Multilingual machine translation with large language models: Empirical results and analysis.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/enis-2024-llm-nmt-advancing-low-resource/