Elamur Research
☰
home / papers / 2024 / Exploring Human-Like Translation Strategy with Large Language Models

Exploring Human-Like Translation Strategy with Large Language Models

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.tacl-1.13.pdf

Summary

Typical machine translation maps a source sentence directly to a target sentence. Professional translators usually prepare first: they identify key terms, work out the subject, and look at similar examples. This paper asks whether large language models can follow a comparable workflow. The authors propose MAPS (Multi-Aspect Prompting and Selection), which has three stages:

  1. Knowledge mining. The LLM extracts keyword pairs, a topic, and a relevant demonstration for the source sentence.
  2. Knowledge integration. Each type of knowledge is used separately as context, producing several translation candidates.
  3. Knowledge selection. Reference-free quality estimation picks the best candidate and discards knowledge that is noisy or unhelpful.

The method was tested on 11 WMT22 translation directions, covering high-, medium- and low-resource pairs, with three LLMs. On COMET and BLEURT it outperformed zero-shot, 5-shot and reranking baselines. Human evaluation through preference studies and MQM showed fewer hallucination, ambiguity, mistranslation, style, untranslated-text and omission errors. MAPS needs no glossary, dictionary or example pool prepared in advance, so it is a general-purpose, domain-agnostic way to improve LLM translation.

Key points

  • Models: text-davinci-003 (closed, accessed via API), plus Alpaca 7B (LLaMA-based) and Vicuna 7B (LLaMA-2-based), both run on a single V100 32GB GPU.
  • Complementary knowledge: keywords, topics and demonstrations each help, and the best results come from using all three together.
  • Selection: both the external COMET-QE model and the LLM itself work as the selector; the LLM is prompted with single-choice questions. Human evaluations were added to check that the gains do not come from overfitting to the automatic metrics.
  • Cost: run serially, MAPS takes about 14× the Baseline's inference time. Parallel processing cuts this by roughly 66%. Generating all knowledge in one JSON call speeds up mining from 3.7 to 2.3 s per sentence, but it causes some invalid outputs (2.0% for Zh-En) and slightly lower scores.
  • Code: available at github.com/zwhe99/MAPS-mt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

110 papers of our library cite it (of the 140673 whose bibliographies we read); 2 of them have a page here, 108 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00642

@article{he2024exploring,
  title = {{Exploring Human-Like Translation Strategy with Large Language Models}},
  author = {He, Zhiwei and Liang, Tian and Jiao, Wenxiang and Zhang, Zhuosheng and Yang, Yujiu and Wang, Rui and Tu, Zhaopeng and Shi, Shuming and Wang, Xing},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00642},
  url = {https://doi.org/10.1162/tacl_a_00642}
}

References

66 references, 22 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2023. In-context examples selection for machine translation. In Findings of the Association for Computational Linguistics: ACL 2023, pages 8857–8873, Toronto, Canada. Association for Computational Linguistics. http… find our card
  2. Mona Baker. 2018. In Other Words: A Coursebook on Translation. Routledge. https:// doi.org/10.4324/9781315619187 — [doi:10.4324/9781315619187](https://doi.org/10.4324/9781315619187) find
  3. Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. find
  4. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, Andre F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. https://doi.org/… find our card
  5. Rachel Bawden and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: The case of bloom. ArXiv preprint, abs/2303.01911. https://doi.org/10.48550/arXiv.2303 .01911 ↗
  6. Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: H… find
  7. Lynne Bowker. 2002. Computer-aided Translation Technology: A Practical Introduction. find
  8. Katja Filippova. 2020. Controlled hallucinations: Learning to generate faithfully from Yehoshua Bar-Hillel. 1960. A demonstration of the nonfeasibility of fully automatic high quality translation. Advances in Computers, 1:158–163. find
  9. Nuno M. Guerreiro, Elena Voita, and André Martins. 2023b. Looking for a needle in a haystack: A comprehensive study of hallucinations in neural machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1059–1075, Dubro… find
  10. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. https://doi.org/10.1162 /… find our card
  11. Basil Hatim and Jeremy Munday. 2004. Translation: An Advanced Resource Book. Psychology Press. https://doi.org/10.4324 /9780203501887 find
  12. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Confere… find our card
  13. Jie He, Tao Wang, Deyi Xiong, and Qun Liu. 2020. The box is in the pen: Evaluating commonsense reasoning in neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3662–3672, Online. Association for Computational Linguistics. https://doi.org/10.186… find
  14. Zhiwei He, Xing Wang, Rui Wang, Shuming Shi, and Zhaopeng Tu. 2022. Bridging the data gap between training and inference for unsupervised neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6611–6623, D… find
  15. Xavier Garcia, Yamini Bansal, Colin Cherry, George Foster, Maxim Krikun, Fangxiaoyu Feng, Melvin Johnson, and Orhan Firat. 2023. The unreasonable effectiveness of few-shot learning for machine translation. ArXiv preprint, abs/2302.01398. https://doi.org /10.48550/arXiv.2302.01398 — [doi:10.48550/arxiv.2302.01398](https://doi.org/10.48550/arxiv.2302.01398) ↗
  16. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based phrase-level prompting of large language models for machine translation. ArXiv preprint, abs/2302 .07856. https://doi.org/10.48550/arXiv .2302.07856 ↗
  17. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are gpt models at machine translation? A comprehensive evaluation. ArXiv preprint, abs/ 2302.09210. https://doi.org/10.48550 /arXiv.2302.… find
  18. Daniel Gile. 2009. Basic Concepts and Models for Interpreter and Translator Training. Benjamins Translation Library. John Benjamins, Amsterdam. https://doi.org/10.1075 /btl.8 find
  19. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12). https://doi.org/10.1145/3571730 ↗ our card
  20. Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023a. Hallucinations in large multilingual translation models. ArXiv preprint, abs/2303 .16104. https://doi.org/10.1162/tacl a 00615 ↗ our card
  21. Wenxiang Jiao, Jen tse Huang, Wenxuan Wang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023a. Parrot: Translating during chat using large language models. In ArXiv, arXiv:2304 .02426. mation Processing Systems, volume 35, pages 22199–22213. find
  22. Wenxiang Jiao, Wenxuan Wang, Jen tse Huang, Xing Wang, and Zhaopeng Tu. 2023b. Is chatgpt a good translator? A preliminary study. In ArXiv preprint, abs/2301.08745. find our card
  23. Junlong Li, Zhuosheng Zhang, and Hai Zhao. 2022. Self-prompting large language models for open-domain qa. ArXiv preprint, abs/2212 .08635. https://doi.org/10.48550/arXiv .2212.08635 — [doi:10.48550/arxiv](https://doi.org/10.48550/arxiv) ↗
  24. Marzena Karpinska and Mohit Iyyer. 2023. Large language models effectively leverage document-level context for literary translation, but critical errors persist. ArXiv preprint, abs/ 2304.03245. https://doi.org/10.18653 /v1/2023.wmt-1.41 find our card
  25. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  26. Hyuhng Joon Kim, Hyunsoo Cho, Junyeob Kim, Taeuk Kim, Kang Min Yoo, and Sang-goo Lee. 2022. Self-generated in-context learning: Leveraging auto-regressive language models as a demonstration generator. ArXiv preprint, abs/2206.08082. — [arXiv:2206.08082](https://arxiv.org/abs/2206.08082) find
  27. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popovi… find
  28. Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Haoran Yang, Wai Lam, and Furu Wei. 2023. Chain-of-dictionary prompting elicits translation in large language models. ArXiv preprint, abs/2305.06575. https://doi.org/10 .48550/arXiv.2305.06575 find
  29. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, pages 478–494, Online.… find
  30. Chenyang Lyu, Jitao Xu, and Longyue Wang. 2023. New trends in machine translation using large language models: Case examples with chatgpt. ArXiv preprint, abs/2305.01181. https://doi.org/10.18653/v1/2023 .emnlp-main.1036 — [doi:10.18653/v1/2023](https://doi.org/10.18653/v1/2023) ↗
  31. Elliott Macklovitch. 1995. The future of MT is now and Bar-Hillel was (almost entirely) right. In Proceedings of the Fourth Bar-Ilan Symposium on the Foundations of Artificial Intelligence. url: http://rali. iro. umontreal. ca/Publications/urls/bisfai95.ps. find
  32. Philipp Koehn. 2009. A process study of computer-aided translation. Machine Translation, 23(4):241–263. https://doi.org/10 .1007/s10590-010-9076-3 find
  33. Takeshi Kojima, Shixiang (Shane) Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. In Advances in Neural Infor-Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020. On faithfulness and factuality in abstractive summarization. find our card
  34. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1906–1919, Online. Association for Computational Linguistics. https://doi.org /10.18653/v1/2020.acl-main.173 ↗
  35. Jonathan Pilault, Xavier Garcia, Arthur Bražinskas, and Orhan Firat. 2023. Interactive-chain-prompting: Ambiguity resolution for cross-lingual conditional generation with interaction. ArXiv preprint, abs/2301.10309. https://doi .org/10.48550/arXiv.2301.10309 — [doi:10.48550/arxiv.2301.10309](https://doi.org/10.48550/arxiv.2301.10309) find
  36. Shima Rahimi Moghaddam and Christopher J. Honey. 2023. Boosting theory-of-mind performance in large language models via prompting. ArXiv preprint, abs/2304.11490. https://doi.org/10.48550/arXiv.2304 .11490 — [doi:10.48550/arxiv.2304](https://doi.org/10.48550/arxiv.2304) ↗
  37. Anthony. Pym. 2014. Exploring Translation Theories, 2nd edition. Routledge. https:// doi.org/10.4324/9781315857633 — [doi:10.4324/9781315857633](https://doi.org/10.4324/9781315857633) find
  38. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pa… find our card
  39. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Train… find
  40. Ankur Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. 2020. ToTTo: A controlled table-to-text generation dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1173–1186, Online. Associat… find our card
  41. Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative agents: Interactive simulacra of human behavior. ArXiv preprint, abs/2304.03442. https://doi.org /10.48550/arXiv.2304.03442 ↗
  42. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards making the most of chatgpt for machine translation. ArXiv preprint, abs/2303.13780. https://doi.org/10.18653/v1/2023 .findings-emnlp.373 ↗ our card
  43. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C. Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. find
  44. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  45. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1 /202… find
  46. ShareGPT. 2023. Sharegpt: Share your wildest chatgpt conversations with one click. Available at: https://sharegpt.com/ ↗
  47. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large language models can be easily distracted by irrelevant context. ArXiv preprint, abs/2302.00093. https://doi.org/10.48550 /arXiv.2302.00093 preprint, abs/2211.09102. https://doi… find
  48. Longyue Wang, Zefeng Du, DongHuai Liu, Deng Cai, Dian Yu, Haiyun Jiang, Yan Wang, Shuming Shi, and Zhaopeng Tu. 2023a. Guofeng: A discourse-aware evaluation benchmark for language understanding, translation and generation. find
  49. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https://github.com/tatsu -lab/stanford alpaca Longyue Wang, Chenyang Lyu, Tianbo Ji, Zhirui Zhang, Dian Yu, Shu… ↗ our card
  50. Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023a. Llama: Open and efficient foundation language models. ArXiv… find
  51. Wenxuan Wang, Wenxiang Jiao, Yongchang Hao, Xing Wang, Shuming Shi, Zhaopeng Tu, and Michael Lyu. 2022a. Understanding and improving sequence-to-sequence pretraining for neural machine translation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1:… find
  52. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  53. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023c. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations. find our card
  54. Yiming Wang, Zhuosheng Zhang, and Rui Wang. 2023d. Element-aware summarization with large language models: Expert-aligned evaluation and chain-of-thought method. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8640–8665, Toron… find
  55. Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022b. Self-instruct: Aligning language model with self generated instructions. https://doi.org/10.18653 /v1/2023.acl-long.754 find
  56. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompting palm for translation: Assessing strategies and performance. find
  57. ArXiv Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emergent abilities of large language models. Transactions… find our card
  58. Chunting Zhou, Graham Neubig, Jiatao Gu, Mona Diab, Francisco Guzmán, Luke Zettlemoyer, and Marjan Ghazvininejad. 2021. Detecting hallucinated content in conditional neural sequence generation. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 1393–1404, Online. A… find
  59. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V. Le, Denny Zhou, et al. 2023. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. find our card
  60. Hao Wu, Wenxuan Wang, Yuxuan Wan, Wenxiang Jiao, and Michael R. Lyu. 2023. Chatgpt or grammarly? Evaluating chatgpt on grammatical error correction benchmark. ArXiv preprint, abs/2303.13648. https://doi.org /10.48550/arXiv.2303.13648 ↗
  61. Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elhoseiny. 2023a. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. ArXiv preprint, abs/2303.06594. https://doi.org/10.48550/arXiv.2303 .06594 — [doi:10.48550/arxiv.2303](https://doi.org/10.48550/arxiv.2303) ↗
  62. Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V. Le, Denny Zhou, and Xinyun Chen. 2023. Large language models as optimizers. ArXiv preprint, abs/2309.03409. https://doi.org/10.48550/arXiv.2309 .03409 ↗ our card
  63. Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Lingpeng Kong, Jiajun Chen, Lei Li, and Shujian Huang. 2023b. Multilingual machine translation with large language models: Empirical results and analysis. ArXiv preprint, abs/ 2304.04675. https://doi.org/10.48550 /arXiv.2304.04675 find our card
  64. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. 2023. Generate rather than retrieve: Large language models are strong context generators. In International Conference for Learning Representation (ICLR). find our card
  65. A Knowledge-specific Prompting for Rerank Biao Zhang, Barry Haddow, and Alexandra Birch. 2023a. Prompting large language model for machine translation: A case study. ArXiv preprint, abs/2301.07069. https://doi.org /10.48550/arXiv.2301.07069 ↗ our card
  66. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2023b. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations. *Table 10: Knowledge-specific prompting for Rerank. Four hypotheses are sampled from each knowledge-specifi… find our card