Elamur Research
☰
home / papers / 2023 / Adaptive Machine Translation with Large Language Models

Adaptive Machine Translation with Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2301.13294

Summary

The paper explores whether large language models can perform adaptive machine translation on the fly, without retraining. This means following a client's approved terminology and style as each new sentence is translated. The authors use GPT-3.5 (text-davinci-003) and in-context learning. For each source sentence, they place similar previously translated pairs (fuzzy matches) from a translation memory into the prompt. These matches are retrieved by embedding similarity. Experiments use the TICO-19 dataset in five directions: EN-AR, EN-ZH, EN-FR, EN-RW and EN-ES.

Prompts built from fuzzy matches consistently beat both zero-shot prompts and prompts with random example pairs. Adding more matches keeps improving quality, though the gains shrink. For high-resource targets, GPT-3.5 can match or beat strong encoder-decoder systems. The authors also test three related uses:

  • LLM post-editing of conventional MT output,
  • terminology extraction,
  • glossary-constrained translation.

They also compare GPT-3.5 briefly with the open models BLOOM and BLOOMZ. The work suggests that translation-memory workflows can use off-the-shelf LLMs. Different pipelines may suit different languages, depending on how well the model supports each one.

Key points

  • Data and setup: The context dataset is TICO-19 (3,070 unique segments). Fuzzy matches are retrieved with Sentence-Transformers paraphrase mining using all-MiniLM-L6-v2, with 1–10 matches per prompt. EN-AR is capped at 7 matches because of token limits.
  • Gains from fuzzy matches (spBLEU):
    • EN-AR: 27.6 zero-shot vs. 41.81 with 7 matches.
    • EN-ZH: 32.41 zero-shot vs. 49.11 with 10 matches.
    • EN-RW: 2.82 zero-shot vs. 17.87 with 10 matches.
  • Weaker languages: Results are weaker for EN-RW (low-resource) and EN-AR (tokenizer issues). However, GPT-3.5 post-editing of OPUS EN-AR output outperformed both OPUS and the Google Translation API.
  • Terminology constraints: Adding glossary terms helps most in zero-shot mode. For example, five terms lift EN-RW spBLEU from 2.82 to 7.26.
  • Open-model comparison: With 2 fuzzy matches, GPT-3.5 generally beats BLOOM and BLOOMZ. The exception is EN-AR, where BLOOM scores 43.19 spBLEU against GPT-3.5's 38.41. The scripts are released at github.com/ymoslem/Adaptive-MT-LLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics: Machine translation

Cited by

143 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 140 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2301.13294 · arXiv 2301.13294

@misc{moslem2023adaptive,
  title = {{Adaptive Machine Translation with Large Language Models}},
  author = {Moslem, Yasmin and Haque, Rejwanul and Kelleher, John D. and Way, Andy},
  year = {2023},
  doi = {10.48550/arxiv.2301.13294},
  eprint = {2301.13294},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2301.13294}
}

References

34 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, Sweta, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In-context Examples Selection for Machine Translation. arXiv [cs.CL], December. find our card
  2. Anastasopoulos, Antonios, Alessandro Cattelan, Zi-Yi Dou, Marcello Federico, Christian Federmann, Dmitriy Genzel, Franscisco Guzmán, et al. 2020. TICO-19: the Translation Initiative for COvid-19. In Proceedings of the 1st Workshop on NLP for COVID-19 (Part 2) at EMNLP 2020, Online, December. find
  3. BigScience Workshop, Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, et al. 2022. BLOOM: A 176B-Parameter Open-Access Multilingual Language Model. arXiv [cs.CL], November. find
  4. Brown, Tom B, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, et al. 2020. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems (NeurIPS 2020), volume 33, pages 1877–1901. find
  5. Bulte, Bram and Arda Tezcan. 2019. Neural Fuzzy Repair: Integrating Fuzzy Matches into Neural Machine Translation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1800–1809, Florence, Italy, July. find
  6. Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, et al. 2022. PaLM: Scaling Language Modeling with Pathways. arXiv [cs.CL], April. find
  7. Coughlin, Deborah. 2003. Correlating automated and human assessments of machine translation quality. In Proceedings of Machine Translation Summit IX: Papers, New Orleans, USA. find
  8. Dinu, Georgiana, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training Neural Machine Translation to Apply Terminology Constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3063– 3068, Florence, Italy, July. find our card
  9. Dong, Qingxiu, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2022. A Survey on In-context Learning. arXiv [cs.CL], December. find our card
  10. Etchegoyhen, Thierry, David Ponce, Harritxu Gete, and Victor Ruiz. 2021. Online Learning over Time in Adaptive Neural Machine Translation. In Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021), pages 411–420, Held Online, September. This work i… find
  11. Haque, Rejwanul, Yasmin Moslem, and Andy Way. 2020. Terminology-Aware Sentence Mining for NMT Domain Adaptation: ADAPT’s Submission to the Adap-MT 2020 English-to-Hindi AI Translation Shared Task. In Proceedings of the 17th International Conference on Natural Language Processing (ICON): Adap-MT 2020… find
  12. Hokamp, Chris and Qun Liu. 2017. Lexically Constrained Decoding for Sequence Generation Using Grid Beam Search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1546, Vancouver, Canada, July. find our card
  13. Hosseini, Kasra, Federico Nanni, and Mariona Coll Ardanuy. 2020. DeezyMatch: A Flexible Deep Learning Approach to Fuzzy String Matching. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 62–69, Online, October. find
  14. Hu, Junjie, Mengzhou Xia, Graham Neubig, and Jaime Carbonell. 2019. Domain Adaptation of Neural Machine Translation by Lexicon Induction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2989–3001, Florence, Italy, July. find
  15. Klein, Guillaume, Dakun Zhang, Clément Chouteau, Josep Crego, and Jean Senellart. 2020. Efficient and high-quality neural machine translation with OpenNMT. In Proceedings of the Fourth Workshop on Neural Generation and Translation, pages 211–217, Stroudsburg, PA, USA, July. find
  16. Knowles, Rebecca, John Ortega, and Philipp Koehn. 2018. A Comparison of Machine Translation Paradigms for Use in Black-Box Fuzzy-Match Repair. In Proceedings of the AMTA 2018 Workshop on Translation Quality Estimation and Automatic Post-Editing, pages 249–255, Boston, MA, March. find
  17. Kudo, Taku and John Richardson. 2018. SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 66–71, Brussels, Belgium, Novem… find our card
  18. Michon, Elise, Josep Crego, and Jean Senellart. 2020. Integrating Domain Terminology into Neural Machine Translation. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3925–3937, Barcelona, Spain (Online), December. International Committee on Computational Lingu… find
  19. Moslem, Yasmin, Rejwanul Haque, John Kelleher, and Andy Way. 2022. Domain-Specific Text Generation for Machine Translation. In Proceedings of the 15th biennial conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 14–30, Orlando, USA, September. find
  20. Muennighoff, Niklas, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, et al. 2022. Crosslingual Generalization through Multitask Finetuning. arXiv [cs.CL], November. find our card
  21. NLLB Team, Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, et al. 2022. No Language Left Behind: Scaling Human-Centered Machine Translation. arXiv [cs.CL], July. find
  22. Ouyang, Long, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, et al. 2022. Training language models to follow instructions with human feedback. arXiv [cs.CL], March. find
  23. Peris, Álvaro and Francisco Casacuberta. 2019. Online learning for effort reduction in interactive neural machine translation. Comput. Speech Lang., 58:98–126, November. find
  24. Pham, Minh Quang, Jitao Xu, Josep Crego, François Yvon, and Jean Senellart. 2020. Priming Neural Machine Translation. In Proceedings of the Fifth Conference on Machine Translation, pages 516–527, Online, November. find
  25. Post, Matt and David Vilar. 2018. Fast Lexically Constrained Decoding with Dynamic Beam Allocation for Neural Machine Translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)… find our card
  26. Reimers, Nils and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,… find
  27. Tiedemann, Jörg. 2020. The Tatoeba Translation Challenge – Realistic Data Sets for Low Resource and Multilingual MT. In Proceedings of the Fifth Conference on Machine Translation, pages 1174–1182, Online, November. find our card
  28. Touvron, Hugo, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, et al. 2023. LLaMA: Open and Efficient Foundation Language Models. arXiv [cs.CL], February. find
  29. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems (NIPS 2017), volume 30. find
  30. Vilar, David, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompting PaLM for Translation: Assessing Strategies and Performance. arXiv [cs.CL], November. find
  31. Wang, Shuo, Zhaopeng Tu, Zhixing Tan, Wenxuan Wang, Maosong Sun, and Yang Liu. 2021. Language Models are Good Translators. ArXiv. find
  32. Wuebker, Joern, Patrick Simianer, and John DeNero. 2018. Compact Personalized Models for Neural Machine Translation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 881–886, Brussels, Belgium. find
  33. Xu, Jitao, Josep Crego, and Jean Senellart. 2020. Boosting Neural Machine Translation with Similar Translations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1580–1590, Online, July. find
  34. Zhang, Biao, Barry Haddow, and Alexandra Birch. 2023. Prompting Large Language Model for Machine Translation: A Case Study. arXiv [cs.CL], January. find our card