Elamur Research⌕
☰
home / papers / 2018 / Guiding Neural Machine Translation with Retrieved Translation Pieces

Guiding Neural Machine Translation with Retrieved Translation Pieces

Paper · 2018 · Jingyi Zhang, Masao Utiyama, Eiichro Sumita et al.

Summary

Neural machine translation often fails to recall and correctly render rare words and phrases, especially in narrow domains where similar sentences recur. Zhang et al. propose a lightweight test-time method that leaves the trained NMT model and its parameters unchanged. For each input sentence, the search engine Lucene retrieves similar source sentences from the word-aligned training corpus. Edit distance identifies which source words match the input. Target-side n-grams (up to 4-grams) aligned only to those matching words become "translation pieces." Each piece is weighted by the highest similarity among the retrieved sentences that produced it. During beam search, the decoder adds a tunable bonus (λ) to output words that complete such pieces.

On three narrow-domain tasks (en-de, en-fr, en-es), this improves BLEU by up to 6 points with only a small increase in decoding time. It matches or beats the more complex SGNMT approach. Its value lies in giving NMT memory-like recall of domain phrasing without retraining or architectural changes.

Key points

  • Retrieval and weighting: Retrieval uses Lucene. Sentence similarity is computed as 1 − edit distance / max sentence length, following Gu et al. (2017). About 10 retrieved sentences per input offers a good speed–quality balance.
  • Effect on rare n-grams: Gains are largest for n-grams that appear in the training data but occur fewer than 5 times. N-grams absent from training see little benefit, and the improvement shrinks as frequency increases.
  • Timing (en-de, seconds per sentence): Baseline decoding takes 0.215 s and guided decoding 0.306 s. Retrieval is negligible at about 0.016 s. Translation-piece collection takes about 0.52 s in an unoptimized Python implementation.
  • Comparison with SGNMT on its datasets (test BLEU):
    • en-de: 50.36 vs. 48.80
    • en-fr: 65.69 vs. 64.60
    • en-es: 57.11 vs. 57.27 (slightly lower)
  • Training cost: Training is unaffected. By contrast, SGNMT changes the model structure and can roughly double translation time.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations🎓Nara Institute of Science and TechnologyIkoma · Japan34 works · 4,419 citations🔬National Institute of Information and Communications TechnologyTokyo · Japan13 works · 1,970 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksArtificial Intelligence · 8,889 worksArts and Humanities · 9,296 worksLanguage and Linguistics · 9,206 works

🧭 Topics 1Research subjects and languages the work is about.Machine translation · 958 works

Cite

Cite as Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig and Satoshi Nakamura. 2018. Guiding Neural Machine Translation with Retrieved Translation Pieces. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n18-1120

@inproceedings{zhang2018guiding,
  title = {{Guiding Neural Machine Translation with Retrieved Translation Pieces}},
  author = {Jingyi Zhang, Masao Utiyama, Eiichro Sumita, Graham Neubig and Satoshi Nakamura},
  booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
  year = {2018},
  doi = {10.18653/v1/n18-1120},
  url = {https://doi.org/10.18653/v1/n18-1120}
}

References

27 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Philip Arthur, Graham Neubig, and Satoshi Nakamura. 2016. Incorporating discrete translation lexicons into neural machine translation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pages 1557– 1567. https://aclweb.org/anthology/ D16-1162. ↗
  2. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473 https://arxiv.org/pdf/ 1409.0473.pdf. ↗
  3. Rajen Chatterjee, Matteo Negri, Marco Turchi, Marcello Federico, Lucia Specia, and Frédéric Blain. 2017. Guiding neural machine translation decoding with external knowledge. In Proceedings of the Second Conference on Machine Translation. pages 157–168. http://www.aclweb.org/ anthology/W17-4716. ↗
  4. Leonard Dahlmann, Evgeny Matusov, Pavel Petrushkov, and Shahram Khadivi. 2017. Neural machine translation leveraging phrase-based models in a hybrid search. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. pages 1411–1420. https://www.aclweb.org/anthology/ D… ↗
  5. M. Amin Farajian, Marco Turchi, Matteo Negri, and Marcello Federico. 2017. Multi-domain neural machine translation through unsupervised adaptation. In Proceedings of the Second Conference on Machine Translation. pages 127–137. http://www.aclweb.org/anthology/W/ W17/W17-4713.pdf. ↗
  6. Ulrich Germann. 2015. Sampling phrase tables for the moses statistical machine translation system. The Prague Bulletin of Mathematical Linguistics 104(1):39–50. https://ufal.mff.cuni.cz/pbml/104/art-germann.pdf. ↗
  7. Gregory Grefenstette. 1999. The world wide web as a resource for example-based machine translation tasks. In Proceedings of the ASLIB Conference on Translating and the Computer. volume 21. http://www.mt-archive.info/ Aslib-1999-Grefenstette.pdf. ↗
  8. Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. 2017. Search engine guided non-parametric neural machine translation. arXiv preprint arXiv:1705.07267 https://arxiv.org/pdf/1705.07267.pdf. — [arXiv:1705.07267](https://arxiv.org/abs/1705.07267) ↗
  9. Wei He, Zhongjun He, Hua Wu, and Haifeng Wang. 2016. Improved neural machine translation with smt features. In AAAI. pages 151–157. https://www.aaai.org/ocs/index.php/AAAI/ AAAI16/paper/view/12189/11577. ↗
  10. Łukasz Kaiser, Ofir Nachum, Aurko Roy, and Samy Bengio. 2017. Learning to remember rare events. arXiv preprint arXiv:1703.03129 https://arxiv.org/pdf/1703.03129.pdf. ↗
  11. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, Chris Dyer, Ondrej Bojar, Alexandra Constantin, and Evan Herbst. 2007. Moses: Open source toolkit for statistical machine translation. In Proc… find
  12. Philipp Koehn and Rebecca Knowles. 2017. Six challenges for neural machine translation. In Proceedings of the First Workshop on Neural Machine Translation. pages 28–39. http://www.aclweb.org/anthology/W17-3204. ↗
  13. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1. pages 48–54. http://www.aclweb. org/anthology/N/N0… ↗
  14. Xiaoqing Li, Jiajun Zhang, and Chengqing Zong. 2016. One sentence one model for neural machine translation. arXiv preprint arXiv:1609.06490 https://arxiv.org/pdf/1609.06490.pdf. ↗
  15. Adam Lopez. 2007. Hierarchical phrase-based translation with suffix arrays. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL). pages 976–985. http://www.aclweb.org/ anthology/D/D07/D07-1104. ↗
  16. Makoto Nagao. 1984. A framework of a mechanical translation between japanese and english by analogy principle. Artificial and human intelligence pages 351–354. http://www.mt-archive.info/ Nagao-1984.pdf. ↗
  17. Toan Q Nguyen and David Chiang. 2017. Improving lexical choice in neural machine translation. arXiv preprint arXiv:1710.01329 https://arxiv.org/pdf/1710.01329.pdf. ↗
  18. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational linguistics 29(1):19– 51. http://www.aclweb.org/anthology/ J/J03/J03-1002.pdf. ↗
  19. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016a. Edinburgh neural machine translation systems for WMT 16. In Proceedings of the First Conference on Machine Translation. pages 371–376. http://www.aclweb.org/ anthology/W/W16/W16-2323.pdf. ↗
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016b. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1715–1725. http://www.aclweb.org/ anthology/P/P16/P16-1162.p… ↗
  21. Felix Stahlberg, Adrià de Gispert, Eva Hasler, and Bill Byrne. 2017. Neural machine translation by minimising the Bayes-risk with respect to syntactic translation lattices. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 2, Shor… find
  22. Yaohua Tang, Fandong Meng, Zhengdong Lu, Hang Li, and Philip LH Yu. 2016. Neural machine translation with external phrase memory. arXiv preprint arXiv:1606.01792 https://arxiv.org/pdf/ 1606.01792.pdf. — [arXiv:1606.01792](https://arxiv.org/abs/1606.01792) ↗
  23. Masao Utiyama, Graham Neubig, Takashi Onishi, and Eiichiro Sumita. 2011. Searching translation memories for paraphrases. In Machine Translation Summit. volume 13, pages 325–331. http://mt-archive.info/ MTS-2011-Utiyama.pdf. ↗
  24. Rui Wang, Andrew Finch, Masao Utiyama, and Eiichiro Sumita. 2017a. Sentence embedding for neural machine translation domain adaptation. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pages 560–566. http://aclweb.org/anthology/P17… ↗
  25. Yuguang Wang, Shanbo Cheng, Liyang Jiang, Jiajun Yang, Wei Chen, Muze Li, Lin Shi, Yanfeng Wang, and Hongtao Yang. 2017b. Sogou neural machine translation systems for wmt17. In Proceedings of the Second Conference on Machine Translation. pages 410–415. http://www.aclweb.org/anthology/W17-4742. ↗
  26. Jiacheng Zhang, Yang Liu, Huanbo Luan, Jingfang Xu, and Maosong Sun. 2017. Prior knowledge integration for neural machine translation using posterior regularization. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pages 1514–1523. h… find
  27. Jian Zhang, Liangyou Li, Andy Way, and Qun Liu. 2016. Topic-informed neural machine translation. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers. pages 1807–1817. http://aclweb.org/anthology/C16-1170. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2018-guiding-neural-machine-translation-retrieved/