Elamur Research⌕
☰
home / papers / 2024 / Iterative Translation Refinement with Large Language Models

Iterative Translation Refinement with Large Language Models

Summary

Machine translation systems, including LLMs and automatic post-editing models, usually produce output in a single pass. Human translators, by contrast, revise drafts several times. This paper tests whether GPT-3.5 can mimic that habit. The method is zero-shot prompting: the model receives the source sentence and the current translation, returns an improved version, and the loop repeats for up to four rounds. The starting translation can come from any system, including WMT submissions, online engines or human references. Experiments use 200-sentence samples from WMT21 and WMT22 test sets in seven directions, one of which is Ukrainian→Czech.

The refined outputs differ substantially from the initial translations, so BLEU and chrF++ drop. COMET scores, however, stay level or improve, and native-speaker judges found the refined text more fluent and natural than both the first GPT drafts and human references, with no loss in overall quality. Ablations show that keeping the source in the prompt and starting from a sensible draft are both essential. The study also exposes how string-based metrics fail to capture naturalness and reduced translationese.

Key points

  • Five prompts compared:
    • Translate: plain translation.
    • Refine: improve a given translation.
    • Refine Contrast: the previous output is labelled "bad".
    • Refine Random: a random sentence is used as the first "bad" draft.
    • Paraphrase: rewrite the translation without seeing the source.
  • Setup:
    • Model: gpt-3.5-turbo, whose training data has a cutoff of Sep 2021.
    • Data: random 200-sentence subsets of WMT21/22 test sets.
    • Directions: en↔de, en↔zh, de→fr, en→ja, uk→cs.
    • Each refinement call is made without conversation history.
  • Metrics:
    • Automatic: BLEU and chrF++ (sacrebleu), plus COMET DA and COMET QE.
    • The best iteration was selected by COMET QE.
  • Main result: Refine generally beats the initial GPT translation on COMET, with the clearest gains for into-English directions. The authors attribute this to the abundance of English pre-training data.
  • Ablations:
    • Paraphrase, which has no source anchor, lowers every metric through semantic drift.
    • Refine Random also performs poorly.
    • Refinement applied to WMT21 de→en system outputs and human translations behaves similarly.
    • The authors identify the cost of multiple API calls as a key limitation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

59 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 53 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{chen2024iterative,
  title = {{Iterative Translation Refinement with Large Language Models}},
  author = {Chen, Pinzhen and Guo, Zhicheng and Haddow, Barry and Heafield, Kenneth},
  booktitle = {Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1)},
  year = {2024},
  url = {https://aclanthology.org/2024.eamt-1.17/}
}

References

50 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Agrawal, Sweta, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In-context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023. find our card
  2. Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In 3rd International Conference on Learning Representations. find our card
  3. Baker, Mona, 1996. Corpus-based Translation Studies: The Challenges that Lie Ahead. Benjamins Translation Library. John Benjamins Publishing Company. find
  4. Bizzoni, Yuri, Tom S Juzek, Cristina España-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. 2020. How human is machine translationese? comparing human and machine translations of text and speech. In Proceedings of the 17th International Conference on Spoken Language Translation. find
  5. Brown, Tom, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems. find
  6. Chatterjee, Rajen, Matteo Negri, Raphael Rubino, and Marco Turchi. 2018. Findings of the WMT 2018 shared task on automatic post-editing. In Proceedings of the Third Conference on Machine Translation. find our card
  7. Chen, Pinzhen, Jindřich Helcl, Ulrich Germann, Laurie Burchell, Nikolay Bogoychev, Antonio Valerio Miceli Barone, Jonas Waldendorf, Alexandra Birch, and Kenneth Heafield. 2021. The University of Edinburgh’s English-German and English-Hausa submissions to the WMT21 news translation task. In Proceedi… find
  8. Chen, Kehai, Masao Utiyama, Eiichiro Sumita, Rui Wang, and Min Zhang. 2022. Synchronous refinement for neural machine translation. In Findings of the Association for Computational Linguistics: ACL 2022. find
  9. Chollampatt, Shamil, Raymond Hendy Susanto, Liling Tan, and Ewa Szymanska. 2020. Can automatic post-editing improve NMT? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. find
  10. Chowdhery, Aakanksha, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022. PaLM: Scaling language modeling with pathways. arXiv preprint. find
  11. Dutta Chowdhury, Koel, Rricha Jalota, Cristina España-Bonet, and Josef Genabith. 2022. Towards debiasing translation artifacts. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. find
  12. Farhad, Akhbardeh, Arkhangorodsky Arkady, Biesialska Magdalena, Bojar Ondřej, Chatterjee Rajen, Chaudhary Vishrav, Marta R Costa-jussa, España-Bonet Cristina, Fan Angela, Federmann Christian, et al. 2021. Findings of the 2021 conference on machine translation (WMT21). In Proceedings of the Sixth C… find our card
  13. Freitag, Markus, Isaac Caswell, and Scott Roy. 2019. APE at scale and its implications on MT evaluation biases. In Proceedings of the Fourth Conference on Machine Translation. find our card
  14. Freitag, Markus, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Confer… find our card
  15. Gellerstam, Martin. 1986. Translationese in Swedish novels translated from English. In Translation studies in Scandinavia: Proceedings from the Scandinavian Symposium on Translation Theory II. CWK Gleerup. find
  16. Gu, Jiatao, Changhan Wang, and Junbo Zhao. 2019. Levenshtein transformer. In Advances in Neural Information Processing Systems. find
  17. Hendy, Amr, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? a comprehensive evaluation. arXiv preprint. find
  18. Ive, Julia, Lucia Specia, Sara Szoc, Tom Vanallemeersch, Joachim Van den Bogaert, Eduardo Farah, Christine Maroti, Artur Ventura, and Maxim Khalilov. 2020. A post-editing dataset in the legal domain: Do we underestimate neural machine translation quality? In Proceedings of the Twelfth Language Resou… find
  19. Kocmi, Tom, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, et al. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation. find
  20. Koppel, Moshe and Noam Ordan. 2011. Translationese and its dialects. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. find
  21. Lee, Jason, Elman Mansimov, and Kyunghyun Cho. 2018. Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. find our card
  22. Lembersky, Gennadi, Noam Ordan, and Shuly Wintner. 2012. Language Models for Machine Translation: Original vs. Translated Texts . Computational Linguistics. find
  23. Lu, Qingyu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023. Error analysis prompting enables human-like translation evaluation in large language models: A case study on ChatGPT. arXiv preprint. find our card
  24. Niehues, Jan, Eunah Cho, Thanh-Le Ha, and Alex Waibel. 2016. Pre-translation for neural machine translation. In Proceedings of the 26th International Conference on Computational Linguistics. find
  25. Novak, Roman, Michael Auli, and David Grangier. 2016. Iterative refinement for machine translation. arXiv preprint. find
  26. Jiao, Wenxiang, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is ChatGPT a good translator? Yes with GPT-4 as the engine. arXiv preprint. find our card
  27. Ouyang, Long, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems. find
  28. Junczys-Dowmunt, Marcin and Roman Grundkiewicz. 2018. MS-UEdin submission to the WMT2018 APE shared task: Dual-source transformer for automatic post-editing. In Proceedings of the Third Conference on Machine Translation. find
  29. Pal, Santanu, Hongfei Xu, Nico Herbig, Sudip Kumar Naskar, Antonio Krüger, and Josef van Genabith. 2020. The transference architecture for automatic post-editing. In Proceedings of the 28th International Conference on Computational Linguistics. find
  30. Kaplan, Jared, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. Scaling laws for neural language models. arXiv preprint. find
  31. Papineni, Kishore, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics. find our card
  32. Knight, Kevin and Ishwar Chander. 1994. Automated postediting of documents. In Proceedings of the Twelfth AAAI National Conference on Artificial Intelligence. find
  33. Popović, Maja. 2017. chrF++: words helping character n-grams. In Proceedings of the Second Conference on Machine Translation. find our card
  34. Kocmi, Tom and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint. find
  35. Radford, Alec, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. openai.com. find
  36. Raunak, Vikas, Arul Menezes, Matt Post, and Hany Hassan. 2023a. Do GPTs produce less literal translations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. find
  37. Raunak, Vikas, Amr Sharaf, Hany Hassan Awadallah, and Arul Menezes. 2023b. Leveraging GPT-4 for automatic translation post-editing. arXiv preprint. find
  38. Rei, Ricardo, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. find our card
  39. Simard, Michel, Cyril Goutte, and Pierre Isabelle. 2007. Statistical phrase-based post-editing. In Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics. find
  40. Xu, Weijia and Marine Carpuat. 2021. EDITOR: An edit-based transformer with repositioning for neural machine translation with soft lexical constraints. Transactions of the Association for Computational Linguistics. find
  41. Xu, Wenda, Danqing Wang, Liangming Pan, Zhenqiao Song, Markus Freitag, William Yang Wang, and Lei Li. 2023. INSTRUCTSCORE: Towards explainable text generation evaluation with automatic feedback. arXiv preprint. find
  42. Zhang, Biao, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In Proceedings of the 40th International Conference on Machine Learning. find our card
  43. Teich, Elke. 2003. Cross-Linguistic Variation in System and Text: A Methodology for the Investigation of Translations and Comparable Texts. De Gruyter Mouton. find
  44. Toral, Antonio. 2019. Post-editese: An exacerbated translationese. In Proceedings of Machine Translation Summit XVII. find
  45. Tran, Chau, Shruti Bhosale, James Cross, Philipp Koehn, Sergey Edunov, and Angela Fan. 2021. Facebook AI’s WMT21 news translation task submission. In Proceedings of the Sixth Conference on Machine Translation. find our card
  46. Vaswani, Ashish, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems. find
  47. Vilar, David, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2023. Prompting PaLM for translation: Assessing strategies and performance. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics. find
  48. Wang, Longyue, Mu Li, Fangxu Liu, Shuming Shi, Zhaopeng Tu, Xing Wang, Shuangzhi Wu, Jiali Zeng, and Wen Zhang. 2021. Tencent translation system for the WMT21 news translation task. In Proceedings of the Sixth Conference on Machine Translation. find
  49. Wei, Daimeng, Zongyao Li, Zhanglin Wu, Zhengzhe Yu, Xiaoyu Chen, Hengchao Shang, Jiaxin Guo, Minghan Wang, Lizhi Lei, Min Zhang, et al. 2021. HW-TSC’s participation in the WMT 2021 news translation shared task. In Proceedings of the Sixth Conference on Machine Translation. find
  50. Wei, Jason, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/chen-2024-iterative-translation-refinement-large-language/