Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2301.08745
Summary
This 2023 Tencent AI Lab report is one of the first systematic checks of how well ChatGPT translates compared with commercial machine translation (MT) services. The authors asked ChatGPT itself to suggest translation prompts, reduced its suggestions to three templates, and kept the best one. They then compared ChatGPT with Google Translate, DeepL Translate and Tencent TranSmart. The tests covered 12 translation directions among German, English, Romanian and Chinese on Flores-101, plus three robustness sets: biomedical text, Reddit comments and speech transcripts. Because the early web interface was slow, each test set was cut to 50 random sentences. Scoring used BLEU, with ChrF++ and TER in some cases. With the GPT-3.5 engine, ChatGPT matched commercial systems on high-resource European pairs. It fell well behind on low-resource languages and on pairs from different language families. Translating through a high-resource "pivot" language helped with distant pairs. Once GPT-4 became the engine, quality reached commercial level even for distant pairs, and human raters found it made the fewest errors. The study suggests that general-purpose LLMs can now serve as practical translators, though weaker languages still need attention.
Key points
- Baselines were Google Translate, DeepL and Tencent TranSmart. The test data were Flores-101 for the multilingual tests, plus WMT19 Biomedical and WMT20 Robustness set2 (Reddit, Ja⇒En) and set3 (Common Voice, De⇒En).
- On Flores-101 Zh⇒En, ChatGPT with the best prompt (TP3) scored 24.73 BLEU, compared with 31.66 for Google.
- The resource gap matters most when translating into the weaker language. On En⇒Ro, ChatGPT's BLEU was about 46% below Google's, but on Ro⇒En it was only about 10% lower.
- ChatGPT did worse than the commercial systems on biomedical and Reddit text, but did well on spoken-language input.
- With GPT-4 as the engine, ChatGPT scored 28.50 BLEU on Zh⇒En, still below Google. Even so, human raters ranked its output best on 32 of 50 sentences. GPT-3.5 produced more hallucinations and mistranslations.
Cited by
- Assessing Cross-Cultural Alignment between ChatGPT and Human Societies: An Empirical Study
- AugGPT: Leveraging ChatGPT for Text Data Augmentation
- Multilingual Jailbreak Challenges in Large Language Models
- Evaluating Large Language Models: A Comprehensive Survey
- AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators
- Exploring Human-Like Translation Strategy with Large Language Models
- Not All Languages Are Created Equal in LLMs: Improving Multilingual Capability by Cross-Lingual-Thought Prompting
- Large Language Models Effectively Leverage Document-level Context for Literary Translation, but Critical Errors Persist
- ChatGPT Beyond English: Towards a Comprehensive Evaluation of Large Language Models in Multilingual Learning
- Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedback
- Counting the Bugs in ChatGPT's Wugs: A Multilingual Investigation into the Morphological Capabilities of a Large Language Model
- Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate
- Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
- ChatGPT as a Factual Inconsistency Evaluator for Text Summarization
- NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark
- "Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models
- Is ChatGPT a Good NLG Evaluator? A Preliminary Study
- Document-Level Machine Translation with Large Language Models
- ChatIE: Zero-Shot Information Extraction via Chatting with ChatGPT
- Contrastive Preference Optimization: Pushing the Boundaries of LLM Performance in Machine Translation
- Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyond
- Benchmarking Machine Translation with Cultural Awareness
- Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
- A Survey of Large Language Models
- Multilingual Machine Translation with Large Language Models: Empirical Results and Analysis
Cite
@misc{jiao2023is,
title = {{Is ChatGPT A Good Translator? Yes With GPT-4 As The Engine}},
author = {Jiao, Wenxiang and Wang, Wenxuan and Huang, Jen-tse and Wang, Xing and Shi, Shuming and Tu, Zhaopeng},
year = {2023},
doi = {10.48550/arxiv.2301.08745},
eprint = {2301.08745},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2301.08745}
}References
- Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In-context examples selection for machine translation. arXiv. find our card
- Alham Fikri Aji, Nikolay Bogoychev, Kenneth Heafield, and Rico Sennrich. 2020. In neural machine translation, what does transfer learning transfer? In ACL. find
- Rachel Bawden, Kevin Bretonnel Cohen, Cristian Grozea, Antonio Jimeno Yepes, Madeleine Kittner, Martin Krallinger, Nancy Mah, Aurelie Neveol, Mariana Neves, Felipe Soares, et al. 2019. Findings of the WMT 2019 biomedical translation shared task: Evaluation for medline abstracts and biomedical termin… find
- Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, et al. 2016. Findings of the 2016 conference on machine translation. In WMT. find our card
- Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS. find
- Yong Cheng, Yang Liu, Qian Yang, Maosong Sun, and Wei Xu. 2016. Neural machine translation with pivot languages. arXiv. find
- Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Mandeep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, et al. 2021. Beyond english-centric multilingual machine translation. JMLR, 22(107):1–48. find
- Akhbardeh Farhad, Arkhangorodsky Arkady, Biesialska Magdalena, Bojar Ondřej, Chatterjee Rajen, Chaudhary Vishrav, Marta R Costa-jussa, España-Bonet Cristina, Fan Angela, Federmann Christian, et al. 2021. Findings of the 2021 conference on machine translation (WMT21). In WMT. find our card
- Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2021. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. arXiv. find
- Zhiwei He, Xing Wang, Zhaopeng Tu, Shuming Shi, and Rui Wang. 2022. Tencent AI Lab-Shanghai Jiao Tong University low-resource translation system for the WMT22 translation task. In WMT. find
- Wenxiang Jiao, Zhaopeng Tu, Jiarui Li, Wenxuan Wang, Jen-tse Huang, and Shuming Shi. 2022a. Tencent’s multilingual machine translation system for WMT22 large-scale african languages. In WMT. find
- Wenxiang Jiao, Xing Wang, Shilin He, Zhaopeng Tu, Irwin King, and Michael R Lyu. 2022b. Exploiting inactive examples for natural language generation with data rejuvenation. IEEE/ACM TASLP. find
- Wenxiang Jiao, Xing Wang, Zhaopeng Tu, Shuming Shi, Michael Lyu, and Irwin King. 2021. Self-training sampling with monolingual data uncertainty for neural machine translation. In ACL. find
- Melvin Johnson, Mike Schuster, Quoc V. Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda B. Viégas, Martin Wattenberg, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2017. Google’s multilingual neural machine translation system: Enabling zero-shot translation. TACL. find
- Zhaocong Li, Xuebo Liu, Derek F Wong, Lidia S Chao, and Min Zhang. 2022. ConsistTL: Modeling consistency in transfer learning for low-resource neural machine translation. In EMNLP. find
- Paul Michel and Graham Neubig. 2018. MTNT: A testbed for machine translation of noisy text. In EMNLP. find
- OpenAI. 2023. GPT-4 technical report. arXiv. find our card
- Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instructions with human feedback. arXiv. find
- Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In ACL. find our card
- Maja Popović. 2017. ChrF++: Words helping character n-grams. In WMT. find our card
- Matt Post. 2018. A call for clarity in reporting bleu scores. In WMT. find our card
- Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In AMTA. find our card
- Lucia Specia, Zhenhao Li, Juan Pino, Vishrav Chaudhary, Francisco Guzmán, Graham Neubig, Nadir Durrani, Yonatan Belinkov, Philipp Koehn, Hassan Sajjad, et al. 2020. Findings of the WMT 2020 shared task on machine translation robustness. In WMT. find
- Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing Wang, Shuming Shi, Zhaopeng Tu, and Michael Lyu. 2022a. Understanding and improving sequence-to-sequence pretraining for neural machine translation. In ACL. find
- Wenxuan Wang, Wenxiang Jiao, Shuo Wang, Zhaopeng Tu, and Michael R Lyu. 2022b. Understanding and mitigating the uncertainty in zero-shot translation. arXiv. find
- Xing Wang, Zhaopeng Tu, and Shuming Shi. 2021. Tencent ai lab machine translation systems for the WMT21 biomedical translation task. In WMT. find
- Barret Zoph, Deniz Yuret, Jonathan May, and Kevin Knight. 2016. Transfer learning for low-resource neural machine translation. In EMNLP. find