Elamur Research
☰
home / papers / 2015 / Addressing the Rare Word Problem in Neural Machine Translation

Addressing the Rare Word Problem in Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P15-1002.pdf

Summary

Neural machine translation systems usually keep a limited vocabulary and map every unseen word to a single unk token. As a result, they translate names, numbers and other rare terms poorly. The authors address this by changing the training data rather than the network. They run an unsupervised word aligner on the parallel corpus and annotate it so that each unknown target token carries a pointer to its aligned source word. They test three annotation schemes: Copyable, PosAll and PosUnk. After decoding, a post-processing step replaces each unknown token with a translation of the pointed-to source word. The translation comes from a dictionary built from the alignment links, or the source word is copied unchanged if no entry exists. Because the NMT model is treated as a black box, the method works with any architecture. On the WMT'14 English–French task, the deep LSTM encoder–decoder of Sutskever et al. gained up to 2.8 BLEU and reached 37.5 BLEU. This made it the first NMT system to outperform the best entry of a WMT'14 contest task.

Key points

  • Three annotation models:
    • Copyable uses numbered unk tokens shared between source and target.
    • PosAll inserts a relative-position token after every target word.
    • PosUnk attaches the relative offset (−7…7, or null) only to unknown target words. It gave better results than PosAll, which doubles target length.
  • Dictionary construction: the dictionary is derived from alignment links. Only pairs with more than 100 alignment counts are kept, and the most probable translation is used. When no entry exists, the source word is copied as is.
  • Model and training setup: deep LSTMs with 4 or 6 layers, 1000 cells and 1000-dimensional embeddings, with reversed source sentences. Training used SGD with a learning rate of 0.7 for 8 epochs and mini-batches of 128. Sentence pairs longer than 100 tokens were discarded.
  • Results on WMT'14 English–French: the gain was up to +2.8 BLEU, largest with smaller vocabularies, for a best score of 37.5 BLEU. The authors also found that a 0.5 drop in perplexity corresponded to roughly 1 BLEU point of improvement.
  • Remaining error sources: wrong dictionary entries (e.g., négociateur for "trader") and mispredicted alignments.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

198 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 181 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1002 · arXiv 1410.8206

@inproceedings{luong2015addressing,
  title = {{Addressing the Rare Word Problem in Neural Machine Translation}},
  author = {Luong, Thang and Sutskever, Ilya and Le, Quoc and Vinyals, Oriol and Zaremba, Wojciech},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1002},
  eprint = {1410.8206},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3115/v1/p15-1002}
}

References

19 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amittai Axelrod, Xiaodong He, and Jianfeng Gao. 2011. Domain adaptation via pseudo in-domain data selection. In EMNLP. find our card
  2. D. Bahdanau, K. Cho, and Y. Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR. find our card
  3. D. Cer, M. Galley, D. Jurafsky, and C. D. Manning. 2010. Phrasal: A statistical machine translation toolkit for exploring new model features. In ACL, Demonstration Session. find
  4. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228. find our card
  5. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP. find
  6. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. 2014. Edinburgh’s phrase-based machine translation systems for WMT-14. In WMT. find
  7. Chris Dyer, Jonathan Weese, Hendra Setiawan, Adam Lopez, Ferhan Ture, Vladimir Eidelman, Juri Ganitkevitch, Phil Blunsom, and Philip Resnik. 2010. cdec: A decoder, alignment, and learning framework for finite-state and context-free translation models. In ACL, Demonstration Session. find
  8. A. Graves, G. Wayne, and I. Danihelka. 2014. Neural turing machines. arXiv preprint arXiv:1410.5401. find
  9. A. Graves. 2013. Generating sequences with recurrent neural networks. In Arxiv preprint arXiv:1308.0850. find
  10. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015. On using very large target vocabulary for neural machine translation. In ACL. find our card
  11. N. Kalchbrenner and P. Blunsom. 2013. Recurrent continuous translation models. In EMNLP. find
  12. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In NAACL. find
  13. Philipp Koehn, Hieu Hoang, Alexandra Birch, Chris Callison-Burch, Marcello Federico, Nicola Bertoldi, Brooke Cowan, Wade Shen, Christine Moran, Richard Zens, et al. 2007. Moses: Open source toolkit for statistical machine translation. In ACL, Demonstration Session. find
  14. P. Liang, B. Taskar, and D. Klein. 2006. Alignment by agreement. In NAACL. find
  15. Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In ACL. find our card
  16. R. Pascanu, T. Mikolov, and Y. Bengio. 2012. On the difficulty of training recurrent neural networks. arXiv preprint arXiv:1211.5063. find
  17. H. Schwenk. 2014. University le mans. http://www-lium.univ-lemans.fr/[Online; ˜schwenk/cslm\_joint\_paper/. accessed 03-September-2014]. find
  18. I. Sutskever, O. Vinyals, and Q. V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS. find
  19. Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2015. Recurrent neural network regularization. In ICLR. find our card