Elamur Research⌕
☰
home / papers / 2013 / Decoding with Large-Scale Neural Language Models Improves Translation

Decoding with Large-Scale Neural Language Models Improves Translation

Summary

Neural language models cope with data sparsity better than count-based n-gram models, because similar words share learned vector representations. Their use in machine translation had been limited by the cost of normalizing a softmax over a large vocabulary. The authors build a feed-forward 5-gram NPLM in the style of Bengio et al. (2003), use rectified linear units for cheaper hidden layers, and train it with noise-contrastive estimation (NCE), so that training never sums over the whole vocabulary. The network learns to output scores that are already close to normalized, so normalization can be skipped at query time. This makes it practical to place the model inside a hierarchical phrase-based decoder, not only to rescore its 1000-best output. The model was tested on NIST Chinese-English and on French, German and Spanish to English (Europarl v7), with vocabularies of 50k–100k words. It adds up to 1.1 Bleu, and integrated decoding usually outperforms reranking. The work shows that neural LMs can scale to realistic vocabularies and corpora without restricting either.

Key points

  • Architecture: 150-dimensional word embeddings, a 750-unit ReLU layer, then a 150-unit ReLU layer. It is trained for 10 epochs of SGD with minibatches of 1000, and draws 100 unigram noise samples per example using the alias method.
  • Chinese-English (NIST 2012 constrained data, tuned on NIST 2003, tested on 2004–2006): reranking gave +0.2–0.4 Bleu, and decoder integration gave +0.6–1.1 Bleu.
  • Europarl (about 50M words, tuned on WMT 2005, tested on nc-test2006): gains were smaller, up to +0.5 Bleu from reranking and up to +0.6 from integration. For Spanish-English, integration was better on dev but slightly worse on test.
  • Training data for the NPLMs consisted of the English half of the bitext plus English Gigaword's Xinhua section (LDC2011T07), which contains 329M words. They were added alongside the Kneser-Ney 5-gram LMs rather than replacing them.
  • On a 5.2M-word sample, NCE training time per epoch stayed roughly flat as vocabulary size grew. MLE training in the CSLM toolkit grew about linearly with |V|. One n-gram lookup costs about 40 µs, and the authors released the code.

Interesting ideas

  • I1. Self-normalization by fixing Z to one during training: the model learns to output probabilities that are approximately normalized, so expensive normalization can be dropped at inference. This is useful for any large-output classifier that has to be queried millions of times.
  • I2. A <null> token for shorter contexts: its embedding is the weighted average of all input embeddings, and it pads contexts shorter than the model order. This gives a cheap backoff-like estimate for lower-order n-grams in a fixed-window neural model.
  • I3. Choosing a fixed-window model so it fits existing infrastructure: unlike an RNN, a feed-forward n-gram NPLM slots into a standard decoder in the same way a count-based LM does. Choosing an architecture by how well it integrates with the pipeline can matter as much as raw accuracy.
  • I4. Comparing reranking against integration explicitly: measuring both shows how much is lost when a strong model only rescores a pruned candidate list. This evaluation design transfers well to any pipeline that uses a late-stage rescorer.
  • I5. Engineering tricks for scale: the authors use sparse matrix inputs and outputs, compute output activations only for the positive word and its noise samples, and split the vocabulary into blocks that are updated in parallel. Together with caching n-gram scores during decoding, these keep large-vocabulary training and inference affordable on ordinary CPUs.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Topics: Machine translation

Cited by

101 papers of our library cite it (of the 140884 whose bibliographies we read); 3 of them have a page here, 98 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{vaswani2013decoding,
  title = {{Decoding with Large-Scale Neural Language Models Improves Translation}},
  author = {Vaswani, Ashish and Zhao, Yinggong and Fossum, Victoria and Chiang, David},
  booktitle = {Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing},
  year = {2013},
  doi = {10.18653/v1/d13-1140},
  url = {https://doi.org/10.18653/v1/d13-1140}
}

References

24 references, 1 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Jauvin. 2003. A neural probabilistic language model. Journal of Machine Learning Research. find
  2. Yoshua Bengio. 2012. Practical recommendations for gradient-based training of deep architectures. CoRR, abs/1206.5533. find
  3. Stanley F. Chen and Joshua Goodman. 1998. An empirical study of smoothing techniques for language mod-http://nlg.isi.edu/software/nplm eling. Technical Report TR-10-98, Harvard University Center for Research in Computing Technology. find
  4. David Chiang. 2007. Hierarchical phrase-based translation. Computational Linguistics, 33(2):201–228. find our card
  5. Michael Gutmann and Aapo Hyvärinen. 2010. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of AISTATS. find
  6. Kenneth Heafield, Philipp Koehn, and Alon Lavie. 2012. Language model rest costs and space-efficient storage. In Proceedings of EMNLP-CoNLL, pages 1169–1178. find
  7. Richard Kronmal and Arthur Peterson. 1979. On the alias method for generating random variables from a discrete distribution. The American Statistician, 33(4):214–218. find
  8. Hai-Son Le, Ilya Oparin, Alexandre Allauzen, Jean-Luc Gauvain, and François Yvon. 2011. Structured output layer neural network language model. In Proceedings of ICASSP, pages 5524–5527. find
  9. Tomáš Mikolov, Anoop Deoras, Stefan Kombrink, Lukáš Burget, and Jan “Honza” Černocký. 2011. Empirical evaluation and combination of advanced language modeling techniques. In Proceedings of IN-TERSPEECH, pages 605–608. find
  10. Tomáš Mikolov. 2012. Statistical Language Models Based on Neural Networks. Ph.D. thesis, Brno University of Technology. find
  11. Andriy Mnih and Geoffrey Hinton. 2009. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems. find
  12. Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of ICML. find
  13. Frederic Morin and Yoshua Bengio. 2005. Hierarchical probabilistic neural network language model. In Proceedings of AISTATS, pages 246–252. find
  14. Vinod Nair and Geoffrey E. Hinton. 2010. Rectified linear units improve restricted Boltzmann machines. In Proceedings of ICML, pages 807–814. find
  15. Jan Niehues and Alex Waibel. 2012. Continuous space language models using Restricted Boltzmann Machines. In Proceedings of IWSLT. find
  16. Franz Josef Och and Hermann Ney. 2003. A systematic comparison of various statistical alignment models. Computational Linguistics, 29(1):19–51. find
  17. Franz Josef Och. 2003. Minimum error rate training in statistical machine translation. In Proceedings of ACL, pages 160–167. find
  18. Holger Schwenk and Jean-Luc Gauvain. 2005. Training neural network language models on very large corpora. In Proceedings of EMNLP. find
  19. Holger Schwenk, Anthony Rousseau, and Mohammed Attik. 2012. Large, pruned or continuous space language models on a GPU for statistical machine translation. In Proceedings of the NAACL-HLT 2012 Workshop: Will We Ever Really Replace the N-gram Model? On the Future of Language Modeling for HLT, pages 1… find
  20. Holger Schwenk. 2004. Efficient training of large neural networks for language modeling. In Proceedings of IJCNN, pages 3059–3062. find
  21. Holger Schwenk. 2007. Continuous space language models. Computer Speech and Language, 21:492– 518. find
  22. Holger Schwenk. 2010. Continuous-space language models for statistical machine translation. Prague Bulletin of Mathematical Linguistics, 93:137–146. find
  23. Holger Schwenk. 2013. CSLM - a modular open-source continuous space language modeling toolkit. In Proceedings of Interspeech. find
  24. M.D. Zeiler, M. Ranzato, R. Monga, M. Mao, K. Yang, Q.V. Le, P. Nguyen, A. Senior, V. Vanhoucke, J. Dean, and G.E. Hinton. 2013. On rectified linear units for speech processing. In Proceedings of ICASSP. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vaswani-2013-decoding-large-scale-neural-language/