Elamur Research
☰
home / papers / 2023 / Towards Making the Most of ChatGPT for Machine Translation

Towards Making the Most of ChatGPT for Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2023.findings-emnlp.373.pdf

Summary

Earlier evaluations found that ChatGPT matches commercial engines on high-resource language pairs but falls behind on low-resource and distant pairs. Those studies relied on basic prompts and default settings. This paper asks how much of the gap comes from that setup rather than from the model itself. The authors test gpt-3.5-turbo-0301 against Google Translator on Flores-200 and on domain test sets from WMT19 (News, Biomedical) and WMT22 E-Commerce. COMET-20 is the main metric, with BLEU and ChrF also reported.

They vary three factors:

  • Decoding temperature.
  • Task-Specific Prompts (TSP): a system role framing the model as a translation engine.
  • Domain-Specific Prompts (DSP): prompts that state the text's domain.

Lower temperatures give better translations. TSP brings small but mostly consistent gains, largest on harder directions. Correct domain hints help, while wrong ones hurt noticeably. Few-shot in-context learning also improves output. Chain-of-thought prompting, however, pushes the model toward literal word-by-word translation and degrades quality. The work offers practical recipes for using LLMs in MT and flags hallucination risks when neither side is English.

Key points

  • Temperatures from 0 to 1 were tested on EN⇒RO, EN⇒ZH and EN⇒DE. Moving from 0 to 1 cost Chinese about 4.3 COMET and 3.7 BLEU, while German was barely affected. T = 0 was adopted as the default.
  • Data: Flores-200 (1,012 sentences, 204 languages), WMT19 News, WMT19 Biomedical and WMT22 E-Commerce. COMET-20 (wmt20-COMET-da) was the primary metric, with BLEU and ChrF computed via SacreBLEU.
  • TSP improved ChatGPT in most Flores-200 directions, with several statistically significant gains. Google Translator still led on many pairs, especially non-English-centric ones such as ZH⇒RO.
  • In non-English-centric directions, ChatGPT tends to hallucinate. The proposed prompts only partly mitigate this.
  • Zero-shot CoT lowered EN⇒ZH by 8.8 COMET. 1-shot CoT did better than zero-shot CoT but still trailed plain zero-shot prompting. Code is available at github.com/Romainpkq/ChatGPT4MT.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.373

@inproceedings{peng2023making,
  title = {{Towards Making the Most of ChatGPT for Machine Translation}},
  author = {Peng, Keqin and Ding, Liang and Zhong, Qihuang and Shen, Li and Liu, Xuebo and Zhang, Min and Ouyang, Yuanxin and Tao, Dacheng},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.373},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.373}
}

References

55 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Loïc Barrault, Ondrej Bojar, Marta R. Costa-jussà, et al. 2019. Findings of the 2019 conference on machine translation (WMT19). In WMT. find our card
  2. Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, et al. 2019. Findings of the WMT 2019 biomedical translation shared task: Evaluation for MEDLINE abstracts and biomedical terminologies. In WMT. find
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020a. Language models are few-shot learners. NeurIPS. find
  4. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, et al. 2020b. Language models are few-shot learners. In NeurIPS. find
  5. Marta R Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, et al. 2022. No language left behind: Scaling human-centered machine translation. arXiv preprint. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  7. Liang Ding, Longyue Wang, Xuebo Liu, Derek F. Wong, Dacheng Tao, and Zhaopeng Tu. 2021. Progressive multi-granularity training for non-autoregressive translation. In Findings of ACL. find
  8. Liang Ding, Longyue Wang, and Dacheng Tao. 2020. Self-attention with cross-lingual position representation. In ACL. find
  9. Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023. A survey on in-context learning. arXiv preprint. find our card
  10. Jinhua Du and Andy Way. 2017. Pre-reordering for neural machine translation: Helpful or harmful? Prague Bulletin of Mathematical Linguistics. find
  11. Jiangtao Feng, Lingpeng Kong, Po-Sen Huang, Chong Wang, Da Huang, Jiayuan Mao, Kan Qiao, and Dengyong Zhou. 2018. Neural phrase-to-phrase machine translation. arXiv preprint. find
  12. Markus Freitag, Ricardo Rei, Nitika Mathur, et al. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In WMT. find our card
  13. Yuan Gao, Ruili Wang, and Feng Hou. 2023. How to design translation prompts for chatgpt: An empirical study. arXiv e-prints. find
  14. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, et al. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. TACL. find
  15. Zhiwei He, Tian Liang, Wenxiang Jiao, Zhuosheng Zhang, Yujiu Yang, Rui Wang, Zhaopeng Tu, Shuming Shi, and Xing Wang. 2023. Exploring human-like translation strategy with large language models. arXiv preprint. find our card
  16. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, et al. 2023. How good are gpt models at machine translation? a comprehensive evaluation. arXiv preprint. find
  17. Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of diverse decoding methods from conditional language models. In ACL. find
  18. Wenxiang Jiao, Wenxuan Wang, Jen-tse Huang, Xing Wang, and Zhaopeng Tu. 2023. Is chatgpt a good translator? a preliminary study. arXiv preprint. find
  19. Ronald M Kaplan, Klaus Netter, Jurgen Wedekind, and Annie Zaenen. 1989. Translation by structural correspondences. In EACL. find our card
  20. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popovic… find
  21. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint. find
  22. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. find
  23. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In WMT. find
  24. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. In NeurIPS. find our card
  25. Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In ACL. find our card
  26. Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2022. What makes good in-context examples for GPT-3? In DeeLIO. find our card
  27. Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. find our card
  28. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint. find
  29. Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. 2023a. Chain-of-dictionary prompting elicits translation in large language models. arXiv preprint. find
  30. Qingyu Lu, Baopu Qiu, Liang Ding, Liping Xie, and Dacheng Tao. 2023b. Error analysis prompting enables human-like translation evaluation in large language models: A case study on chatgpt. arXiv preprint. find
  31. Makoto Nagao. 1984. A framework of a mechanical translation between japanese and english by analogy principle. Artificial and human intelligence. find
  32. Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, et al. 2022a. Training language models to follow instructions with human feedback. arXiv preprint. find
  33. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022b. Training language models to follow instructions with human feedback. NeurIPS. find
  34. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In ACL. find our card
  35. Keqin Peng, Liang Ding, Qihuang Zhong, Yuanxin Ouyang, Wenge Rong, Zhang Xiong, and Dacheng Tao. 2023. Token-level self-evolution training for sequence-to-sequence learning. In ACL. find
  36. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In WMT. find our card
  37. Matt Post. 2018. A call for clarity in reporting BLEU scores. In WMT. find our card
  38. Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language processing task solver? arXiv preprint. find our card
  39. Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog. find
  40. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR. find
  41. Richard Zens, Franz Josef Och, and Hermann Ney. 2002. Phrase-based statistical machine translation. In KI. find
  42. Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint. find our card
  43. Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2023. Can chatgpt understand too? a comparative study on chatgpt and fine-tuned bert. arXiv preprint. find
  44. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In EMNLP. find our card
  45. Qihuang Zhong, Liang Ding, Yibing Zhan, Yu Qiao, Yonggang Wen, Li Shen, Juhua Liu, Baosheng Yu, Bo Du, Yixin Chen, et al. 2022. Toward efficient language model pretraining and downstream adaptation via self-evolution: A case study on superglue. arXiv preprint. find
  46. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompting palm for translation: Assessing strategies and performance. arXiv preprint. find
  47. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2022. Large language models are human-level prompt engineers. arXiv preprint. find our card
  48. Jiaan Wang, Yunlong Liang, Fandong Meng, Zhixu Li, Jianfeng Qu, and Jie Zhou. 2023a. Cross-lingual summarization via chatgpt. arXiv preprint. find
  49. Qingyue Wang, Liang Ding, Yanan Cao, Zhiliang Tian, Shi Wang, Dacheng Tao, and Li Guo. 2023b. Recursively summarizing enables long-term dialogue memory in large language models. arXiv preprint. find
  50. Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a. Finetuned language models are zero-shot learners. In ICLR. find
  51. Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. 2022b. Emergent abilities of large language models. arXiv preprint. find our card
  52. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. 2022c. Chain of thought prompting elicits reasoning in large language models. arXiv preprint. find our card
  53. Changtong Zan, Liang Ding, Li Shen, Yu Cao, Weifeng Liu, and Dacheng Tao. 2022a. Bridging cross-lingual gaps during leveraging the multilingual sequence-to-sequence pretraining for text generation. arXiv preprint. find
  54. Changtong Zan, Liang Ding, Li Shen, Yibin Lei, Yibing Zhan, Weifeng Liu, and Dacheng Tao. 2023. Unlikelihood tuning on negative samples amazingly improves zero-shot translation. arXiv preprint. find
  55. Changtong Zan, Keqin Peng, Liang Ding, Baopu Qiu, Boan Liu, Shwai He, Qingyu Lu, Zheng Zhang, Chuang Liu, Weifeng Liu, Yibing Zhan, and Dacheng Tao. 2022b. Vega-MT: The JD explore academy machine translation system for WMT22. In WMT. find