Elamur Research⌕
☰
home / papers / 2023 / Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation

Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation

Summary

Prompted LLMs translate surprisingly well without MT-specific training, but they often mistranslate rare words, which are common in low-resource language pairs and out-of-domain text. The authors introduce DiPMT, a training-free prompting technique that adds bilingual dictionary knowledge to the prompt. After the translation instruction and source sentence, the prompt lists candidate target-language equivalents for selected source words. It then asks for the full translation. The model receives these as soft hints rather than enforced constraints. The approach is tested with OPT-175B and Bloom-176B in a 4-shot setup on ten Flores-101 languages, translating both into and out of English, using MUSE ground-truth dictionaries. It is also tested on German–English domain data (Medical, Law, IT, Koran), where the dictionaries are induced automatically from training data. DiPMT beats a hint-free prompt in both settings, with out-of-domain gains reaching 13 BLEU. A controllability analysis shows the model adopts suggested words far more often and largely ignores deliberately wrong hints. This makes cheap lexical resources a practical way to improve LLM translation for under-served languages and domains.

Key points

  • Prompt structure: Each prompt has three parts: the source sentence with the instruction, then "In this context, the word X means…" hints, then a request for the full translation. Four demonstrations come from the dev set, and the same ones are used across all test sentences and models.
  • Hint selection: The 500 most frequent source words receive no hints. Hints are given only for exact dictionary matches, with at most 3 randomly chosen translations per word.
  • Data:
    • In-domain experiments cover ten Flores-101 languages where OPT scores between 10 and 30 BLEU: Italian, Catalan, Dutch, Danish, Norwegian, Croatian, Slovak, Indonesian, Malay and Filipino.
    • Out-of-domain experiments use the De–En corpus of Aharoni and Goldberg (2020), with dictionaries extracted following Shi et al. (2021).
    • Scoring uses sacreBLEU.
  • Out-of-domain results: Improvements reach up to 13 BLEU points.
  • Controllability (OPT, Table 7):
    • Dictionary hints raise hint usage in the output by roughly 7–26 percentage points over the baseline.
    • Gold-only hints raise it by about 13–33 points.
    • Shuffled false hints change output almost not at all (≤1 point), so the method is robust to bad hints.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2302.07856 · arXiv 2302.07856

@misc{ghazvininejad2023dictionary,
  title = {{Dictionary-based Phrase-level Prompting of Large Language Models for Machine Translation}},
  author = {Ghazvininejad, Marjan and Gonen, Hila and Zettlemoyer, Luke},
  year = {2023},
  doi = {10.48550/arxiv.2302.07856},
  eprint = {2302.07856},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2302.07856}
}

References

30 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sweta Agrawal, Chunting Zhou, Mike Lewis, Luke Zettlemoyer, and Marjan Ghazvininejad. 2022. In-context examples selection for machine translation. arXiv preprint arXiv:2212.02437. find our card
  2. Roee Aharoni and Yoav Goldberg. 2020. Unsupervised domain clusters in pretrained language models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7747– 7763, Online. Association for Computational Linguistics. find our card
  3. Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. arXiv preprint arXiv:1606.02006. find our card
  4. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find our card
  5. Guanhua Chen, Yun Chen, Yong Wang, and Victor OK Li. 2021. Lexical-constraint-aware neural machine translation via data augmentation. In Proceedings of the Twenty-Ninth International Conference on International Joint Conferences on Artificial Intelligence, pages 3587–3593. find
  6. Alexis Conneau, Guillaume Lample, Marc’Aurelio Ranzato, Ludovic Denoyer, and Hervé Jégou. 2017. Word translation without parallel data. arXiv preprint arXiv:1710.04087. find
  7. Chris Hokamp and Qun Liu. 2017. Lexically constrained decoding for sequence generation using grid beam search. arXiv preprint arXiv:1704.07138. find our card
  8. Urvashi Khandelwal, Angela Fan, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2020. Nearest neighbor machine translation. arXiv preprint arXiv:2010.00710. — [arXiv:2010.00710](https://arxiv.org/abs/2010.00710) find
  9. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. arXiv preprint arXiv:1706.03872. find
  10. Hugo Laurençon, Lucile Saulnier, Thomas Wang, Christopher Akiki, Albert Villanova del Moral, Teven Le Scao, Leandro Von Werra, Chenghao Mou, Eduardo González Ponferrada, Huu Nguyen, et al. 2022. The bigscience roots corpus: A 1.6 tb composite multilingual dataset. In Thirty-sixth Conference on Neura… find
  11. Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, et al. 2021. Datasets: A community library for natural language processing. arXiv preprint arXiv:2109.02846. find our card
  12. Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang, Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, et al. 2021. Few-shot learning with multilingual language models. arXiv preprint arXiv:2112.10668. find our card
  13. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186– 191, Brussels, Belgium. Association for Computational Linguistics. find our card
  14. Georgiana Dinu, Prashant Mathur, Marcello Federico, and Yaser Al-Onaizan. 2019. Training neural machine translation to apply terminology constraints. arXiv preprint arXiv:1906.01105. find our card
  15. Matt Post and David Vilar. 2018. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. arXiv preprint arXiv:1804.06609. find our card
  16. Xavier Garcia and Orhan Firat. 2022. Using natural language prompts for machine translation. arXiv preprint arXiv:2202.11822. — [arXiv:2202.11822](https://arxiv.org/abs/2202.11822) find
  17. Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher D. Manning. 2020. Stanza: A Python natural language processing toolkit for many human languages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations. find our card
  18. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzman, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computat… find
  19. Mika Hämäläinen and Khalid Alnajjar. 2019. A template based approach for training nmt for low-resource uralic languages-a pilot with finnish. In Proceedings of the 2019 2nd International Conference on Algorithms, Computing and Artificial Intelligence, pages 520–525. find
  20. Laria Reynolds and Kyle McDonell. 2021. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7. find our card
  21. Masoud Jalili Sabet, Philipp Dufter, François Yvon, and Hinrich Schütze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728. find
  22. Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. find
  23. Haoyue Shi, Luke Zettlemoyer, and Sida I Wang. 2021. Bilingual lexicon induction via unsupervised bitext construction and word alignment. arXiv preprint arXiv:2101.00148. find
  24. Kai Song, Yue Zhang, Heng Yu, Weihua Luo, Kun Wang, and Min Zhang. 2019. Code-switching for enhancing nmt with pre-specified translation. arXiv preprint arXiv:1904.09107. find
  25. Zewei Sun, Qingnan Jiang, Shujian Huang, Jun Cao, Shanbo Cheng, and Mingxuan Wang. 2022. Zero-shot domain adaptation for neural machine translation with retrieved phrase-level prompts. arXiv preprint arXiv:2209.11409. — [arXiv:2209.11409](https://arxiv.org/abs/2209.11409) find
  26. David Vilar, Markus Freitag, Colin Cherry, Jiaming Luo, Viresh Ratnakar, and George Foster. 2022. Prompting palm for translation: Assessing strategies and performance. arXiv preprint arXiv:2211.09102. find
  27. Jiajun Zhang and Chengqing Zong. 2016. Bridging neural machine translation and bilingual dictionaries. arXiv preprint arXiv:1610.07272. — [arXiv:1610.07272](https://arxiv.org/abs/1610.07272) find
  28. Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. find our card
  29. Xin Zheng, Zhirui Zhang, Shujian Huang, Boxing Chen, Jun Xie, Weihua Luo, and Jiajun Chen. 2021. Non-parametric unsupervised domain adaptation for neural machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 4234–4241. find
  30. Xing Jie Zhong and David Chiang. 2020. Look it up: Bilingual and monolingual dictionaries improve neural machine translation. arXiv preprint arXiv:2010.05997. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/ghazvininejad-2023-dictionary-based-phrase-level-prompting/