Elamur Research⌕
☰
home / papers / 2024 / Chain-of-Dictionary Prompting Elicits Translation in Large Language Models

Chain-of-Dictionary Prompting Elicits Translation in Large Language Models

Summary

Large language models translate fairly well without dedicated training, but they still handle low-resource languages poorly, often because they mistranslate rare words. The authors propose CoD (Chain-of-Dictionary Prompting) to address this. For a subset of words in the source sentence, CoD adds lexical hints to an ordinary zero-shot translation prompt. Each hint links the word to its equivalents in the target language and in three high-resource auxiliary languages: French, German and Portuguese. The dictionaries are built automatically. ChatGPT pulls out keywords from the input, and NLLB translates them. The approach is tested on FLORES-200 across translation directions between English and roughly 200 languages, using GPT-3.5-TURBO, text-davinci-003 and BLOOM-7b, with NLLB 3.3B as a reference system. CoD improves ChatGPT on most languages and sometimes recovers translation in directions where the model had nearly failed. It also outperforms bilingual hints, unchained (decomposed) hints and few-shot demonstrations for low-resource languages. This matters because dictionaries are far easier to obtain than parallel examples for under-resourced languages, Ukrainian-adjacent ones included.

Key points

  • The largest reported gain is about 13x in chrF++, from 3.08 to 42.63 for English→Serbian (Cyrillic) on the full FLORES-200 devtest set.
  • Each prompt chains 5 languages: source, target, and fra_Latn, deu_Latn and por_Latn as auxiliaries.
  • With CoD, ChatGPT clearly surpasses the SOTA translator NLLB 3.3B.
  • Several design choices hurt quality:
    • splitting the chain into separate bilingual pairs;
    • shuffling dictionary entries out of source-word order;
    • choosing auxiliary languages similar to the target, for example ones written in the same script.
  • The cost is up to 1.8x inference time and up to 3x prompt length. Dropping stopwords from the dictionary reduces this cost. Code and resources are released at github.com/HongyuanLuke/Chain-of-Dictionary.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Low-resource languages

Institutions

🏢MicrosoftRedmond · United States193 works · 110 465 citations🎓Chinese University of Hong KongHong Kong · Hong Kong42 works · 11 333 citations

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.emnlp-main.55

@inproceedings{lu2024chain,
  title = {{Chain-of-Dictionary Prompting Elicits Translation in Large Language Models}},
  author = {Lu, Hongyuan and Yang, Haoran and Huang, Haoyang and Zhang, Dongdong and Lam, Wai and Wei, Furu},
  booktitle = {Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.emnlp-main.55},
  url = {https://doi.org/10.18653/v1/2024.emnlp-main.55}
}

References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In-context Examples Selection for Machine Translation. arXiv e-prints, page arXiv:2212.02437. find our card
  2. Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1557–1567, Austin, Texas. Association for Computational Linguistics. find our card
  3. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find our card
  4. Guanhua Chen, Yun Chen, Yong Wang, and Victor O. K. Li. 2021. Lexical-constraint-aware neural machine translation via data augmentation. In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI’20. find
  5. Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3063– 3068, Florence, Italy. Association for Computa… find our card
  6. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia.… find
  7. Xavier Garcia and Orhan Firat. 2022. Using natural language prompts for machine translation. arXiv e-prints, page arXiv:2202.11822. — [arXiv:2202.11822](https://arxiv.org/abs/2202.11822) find
  8. Marjan Ghazvininejad, Hila Gonen, and Luke Zettlemoyer. 2023. Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation. arXiv e-prints, page arXiv:2302.07856. find our card
  9. Mika Hämäläinen and Khalid Alnajjar. 2020. A template based approach for training nmt for low-resource uralic languages - a pilot with finnish. In Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence, ACAI ’19, page 520–525, New York, NY, USA. Ass… find
  10. Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1535–1546, Vancouver, Canada. Association for Computational Linguis… find our card
  11. Wenxiang Jiao, Jen-tse Huang, Wenxuan Wang, Zhiwei He, Tian Liang, Xing Wang, Shuming Shi, and Zhaopeng Tu. 2023. ParroT: Translating during chat using large language models tuned with human translation and feedback. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 150… find
  12. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennigho… find
  13. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasunuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O’Horo, Jeff Wang, Luke Zettlemoyer, Zornitsa Kozareva, Mona Diab, Veselin Stoyano… find our card
  14. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. find
  15. Hongyuan Lu, Haoyang Huang, Shuming Ma, Dongdong Zhang, Wai Lam, Zhaochuan Gao, Anthony Aue, Arul Menezes, and Furu Wei. 2023. TRIP: Accelerating document-level multilingual pre-training via triangular document-level pre-training on parallel data triplets. In Findings of the Association for Computat… find
  16. Hongyuan Lu, Haoyang Huang, Dongdong Zhang, Furu Wei, and Wai Lam. 2024. Revamping multilingual agreement bidirectionally via switched back-translation for multilingual neural machine translation. In Findings of the Association for Computational Linguistics: EACL 2024, pages 264–275, St. Julian’s, M… find
  17. NLLB-Team. 2022. No language left behind: Scaling human-centered machine translation. find
  18. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational… find our card
  19. Keqin Peng, Liang Ding, Qihuang Zhong, Li Shen, Xuebo Liu, Min Zhang, Yuanxin Ouyang, and Dacheng Tao. 2023. Towards Making the Most of ChatGPT for Machine Translation. arXiv e-prints, page arXiv:2303.13780. find our card
  20. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. find our card
  21. Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers),… find our card
  22. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  23. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, CHI EA ’21, New York, NY, USA. Association for Computing Machinery. find our card
  24. Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou. 2023. Large Language Models Can Be Easily Distracted by Irrelevant Context. arXiv e-prints, page arXiv:2302.00093. find our card
  25. Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019. Code-switching for enhancing NMT with pre-specified translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long… find
  26. Tianyi Tang, Hongyuan Lu, Yuchen Jiang, Haoyang Huang, Dongdong Zhang, Xin Zhao, Tom Kocmi, and Furu Wei. 2024. Not all metrics are guilty: Improving NLG evaluation by diversifying references. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Li… find
  27. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompting PaLM for Translation: Assessing Strategies and Performance. arXiv e-prints, page arXiv:2211.09102. find
  28. Jiaan Wang, Yunlong Liang, Fandong Meng, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou. 2023. Is ChatGPT a Good NLG Evaluator? A Preliminary Study. arXiv e-prints, page arXiv:2303.04048. find our card
  29. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoc V Le, and Denny Zhou. 2022. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. find our card
  30. Jiajun Zhang and Chengqing Zong. 2016. Bridging Neural Machine Translation and Bilingual Dictionaries. arXiv e-prints, page arXiv:1610.07272. — [arXiv:1610.07272](https://arxiv.org/abs/1610.07272) find
  31. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: Open Pre-trai… find our card
  32. Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations. find
  33. Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. 2021. Non-parametric unsupervised domain adaptation for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4234–4241, Punta Cana, Dominican Republic. As… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lu-2024-chain-dictionary-prompting-elicits-translation/