Elamur Research
☰
home / papers / 2015 / On Using Very Large Target Vocabulary for Neural Machine Translation

On Using Very Large Target Vocabulary for Neural Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 3.0 (ACL Anthology, before 2016)" does not allow redistributing the text itself. Original — aclanthology.org/P15-1001.pdf

Summary

Neural machine translation (NMT) systems usually restrict the output side to a shortlist of the most frequent words, typically 30k–80k. Every other word becomes [UNK]. This hurts quality, especially for morphologically rich targets such as German, because computing the softmax normalization over all target words is costly in time and memory.

The authors adapt the importance-sampling idea of Bengio and Sénécal (2008) to the attention-based RNNsearch model of Bahdanau et al. Each update normalizes over a small sampled subset of the vocabulary instead of the whole list. In practice, they split the training corpus into partitions, each with its own subset of target words capped by a threshold τ. This keeps training cost and GPU memory independent of the full vocabulary size. At test time, decoding is restricted to a candidate list: frequent words plus likely dictionary translations of each source word. This brings decoding speed back near the baseline.

The resulting RNNsearch-LV models match or beat shortlist baselines on WMT'14 English→French and English→German. They approach state-of-the-art BLEU, which shows that NMT can scale to very large vocabularies.

Key points

  • Method: biased importance sampling restricts each update to the correct word plus a sampled subset V′. Per-partition vocabularies, bounded by τ (e.g. 15k or 30k), control the number of updated parameters. No gradient correction is applied, so test vocabularies similar to the training ones work best.
  • Decoding: the candidate list combines 30k common words with up to K′ dictionary candidates per source word. Moderate K′ outperforms K′ = 1, which suggests the model can pick sensibly among rare alternatives. A shared candidate list across sentences matches baseline RNNsearch decoding speed.
  • English→French: the best single model beats the best single NMT model of Luong et al. (2015) by about 1 BLEU. The ensemble is roughly 0.3 BLEU behind the top system.
  • English→German: reaches 21.59 BLEU, versus the earlier best of 20.67 (Buck et al., 2014).
  • Release: the authors published their source code, built on Theano.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

190 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 174 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.3115/v1/p15-1001 · arXiv 1412.2007

@inproceedings{jean2015using,
  title = {{On Using Very Large Target Vocabulary for Neural Machine Translation}},
  author = {Jean, Sébastien and Cho, Kyunghyun and Memisevic, Roland and Bengio, Yoshua},
  booktitle = {Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)},
  year = {2015},
  doi = {10.3115/v1/p15-1001},
  eprint = {1412.2007},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.3115/v1/p15-1001}
}

References

23 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In ICLR’2015, arXiv:1409.0473. find our card
  2. Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, James Bergstra, Ian J. Goodfellow, Arnaud Bergeron, Nicolas Bouchard, and Yoshua Bengio. 2012. Theano: new features and speed improvements. find
  3. Deep Learning and Unsupervised Feature Learning NIPS 2012 Workshop. find
  4. Yoshua Bengio and Jean-Sébastien Sénécal. 2008. Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Trans. Neural Networks, 19(4):713–722. find
  5. James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. 2010. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), June. Oral… find
  6. Christian Buck, Kenneth Heafield, and Bas van Ooyen. 2014. N-gram counts and language models from the common crawl. In Proceedings of the Language Resources and Evaluation Conference, Reykjavı́k, Iceland, May. find
  7. Kyunghyun Cho, Bart van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. 2014a. On the properties of neural machine translation: Encoder– Decoder approaches. In Eighth Workshop on Syntax, Semantics and Structure in Statistical Translation, October. find our card
  8. Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014b. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the Empiricial Methods in Natural Language Processing (EMNLP 2014), October… find
  9. Nadir Durrani, Barry Haddow, Philipp Koehn, and Kenneth Heafield. 2014. Edinburgh’s phrase-based machine translation systems for WMT-14. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 97–104. Association for Computational Linguistics Baltimore, MD, USA. find
  10. Chris Dyer, Victor Chahuneau, and Noah A. Smith. 2013. A simple, fast, and effective reparameterization of IBM Model 2. In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648, Atlanta, Georgia,… find
  11. Mikel L. Forcada and Ramón P. Ñeco. 1997. Recursive hetero-associative memories for translation. In José Mira, Roberto Moreno-Dı́az, and Joan Cabestany, editors, Biological and Artificial Computation: From Neuroscience to Technology, volume 1240 of Lecture Notes in Computer Science, pages 453–462… find
  12. Markus Freitag, Stephan Peitz, Joern Wuebker, Hermann Ney, Matthias Huck, Rico Sennrich, Nadir Durrani, Maria Nadejde, Philip Williams, Philipp Koehn, et al. 2014. Eu-bridge MT: Combined machine translation. In Proc. of the Workshop on Statistical Machine Translation, pages 105–113. find
  13. M. Gutmann and A. Hyvarinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of The Thirteenth International Conference on Artificial Intelligence and Statistics (AISTATS’10). find
  14. Nal Kalchbrenner and Phil Blunsom. 2013. Recurrent continuous translation models. In Proceedings of the ACL Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1700– 1709. Association for Computational Linguistics. find
  15. Philipp Koehn, Franz Josef Och, and Daniel Marcu. 2003. Statistical phrase-based translation. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology - Volume 1, NAACL ’03, pages 48–54. find
  16. Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York, NY, USA, 1st edition. find
  17. Liangyou Li, Xiaofeng Wu, Santiago Cortes Vaillo, Jun Xie, Andy Way, and Qun Liu. 2014. The DCU-ICTCAS MT system at WMT 2014 on German-English translation task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 136–141, Baltimore, Maryland, USA, June. Association for Com… find
  18. Minh-Thang Luong, Ilya Sutskever, Quoc V. Le, Oriol Vinyals, and Wojciech Zaremba. 2015. Addressing the rare word problem in neural machine translation. In Proceedings of ACL. find our card
  19. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In International Conference on Learning Representations: Workshops Track. find
  20. Andriy Mnih and Koray Kavukcuoglu. 2013. Learning word embeddings efficiently with noise-contrastive estimation. In C.J.C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances in Neural Information Processing Systems 26, pages 2265–2273. Curran Associates, Inc. find
  21. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, pages 311–318, Stroudsburg, PA, USA. Association for Computational Lingu… find our card
  22. Stephan Peitz, Joern Wuebker, Markus Freitag, and Hermann Ney. 2014. The RWTH Aachen German-English machine translation system for WMT 2014. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 157–162, Baltimore, Maryland, USA, June. Association for Computational Linguisti… find
  23. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In NIPS’2014. find