Elamur Research⌕
☰
home / papers / 2012 / A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

A Fast and Simple Algorithm for Training Neural Probabilistic Language Models

Paper · 2012 · Andriy Mnih and Yee Whye Teh

Summary

Neural probabilistic language models (NPLMs) tend to beat n-gram models, but they are slow to train. The softmax normalization forces every gradient step to score the entire vocabulary. Mnih and Teh adapt noise-contrastive estimation (NCE), introduced by Gutmann and Hyvärinen for unnormalized densities, to discrete word distributions. NCE turns learning into a binary classification task: the model learns to tell observed next words apart from samples drawn from a simple unigram noise distribution, and normalizing constants are handled as parameters rather than computed explicitly. As the number of noise samples grows, the gradient approaches the maximum likelihood gradient.

The method is tested with a log-bilinear (LBL) model. On Penn Treebank it cuts training time by more than tenfold with no loss in model quality, and it is more stable than importance sampling while needing fewer samples. Trained at scale on Project Gutenberg texts, the LBL models reach a new best accuracy on the Microsoft Research Sentence Completion Challenge. This makes neural language models practical for large corpora.

Key points

  • NCE uses a fixed, context-independent unigram noise distribution. Unlike importance sampling (Bengio & Senécal), it needs no adaptive proposal and no growing sample count to stay stable.
  • On Penn Treebank, NCE-trained LBL models match maximum-likelihood quality in under one-tenth of the training time.
  • The large-scale corpus comes from 522 Project Gutenberg works, tokenized with NLTK (Punkt sentence tokenizer and Penn Treebank word tokenizer). It yields a 47M-word training set and a vocabulary of just under 80K words. Each model trained in 1–2 days on a single CPU core.
  • On all 1,040 MSR Sentence Completion sentences, the results compare as follows:
    • An LBL model with a 10-word context and 300-dimensional features scored 54.7% correct.
    • The previous LSA record was 49%.
    • The best Kneser-Ney n-gram baseline (SRILM) scored 39.1%.
  • Bidirectional-context LBL models had much lower perplexity but were noticeably worse at sentence completion than unidirectional ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University College LondonLondon · United Kingdom107 works · 22,923 citations

Also on the byline: Gatsby Computational Neuroscience Unit.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Andriy Mnih and Yee Whye Teh. 2012. A Fast and Simple Algorithm for Training Neural Probabilistic Language Models. arXiv:1206.6426 · In Proceedings of the 29th International Conference on Machine Learning, pages 1751-1758, 2012.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1206.6426 · arXiv 1206.6426

@misc{mnih2012fast,
  title = {{A Fast and Simple Algorithm for Training Neural Probabilistic Language Models}},
  author = {Andriy Mnih and Yee Whye Teh},
  year = {2012},
  doi = {10.48550/arxiv.1206.6426},
  eprint = {1206.6426},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1206.6426}
}

References

22 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Yoshua and Senécal, Jean-Sébastien. Quick training of probabilistic neural nets by importance sampling. In AISTATS’03, 2003. find
  2. Bengio, Yoshua and Senécal, Jean-Sébastien. Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Transactions on Neural Networks, 19(4):713–722, 2008. find
  3. Bengio, Yoshua, Ducharme, Réjean, and Vincent, Pascal. A neural probabilistic language model. In NIPS, pp. 932–938, 2000. find
  4. Bengio, Yoshua, Ducharme, Rejean, Vincent, Pascal, and Jauvin, Christian. A neural probabilistic language model. Journal of Machine Learning Research, 3:1137–1155, 2003.
  5. Berger, A.L., Pietra, V.J.D., and Pietra, S.A.D. A maximum entropy approach to natural language processing. Computational linguistics, 22(1):39–71, 1996.
  6. Bird, S., Klein, E., and Loper, E. Natural language processing with Python. O’Reilly Media, 2009. find
  7. Collobert, R. and Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, 2008. find
  8. Gutmann, M. and Hyvärinen, A. Noise-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proc. Int. Conf. on Artificial Intelligence and Statistics (AISTATS2010), 2010. find
  9. Gutmann, M.U. and Hyvärinen, A. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13:307–361, 2012. find
  10. Maas, A.L. and Ng, A.Y. A probabilistic model for semantic word vectors. In NIPS 2010 Workshop on Deep Learning and Unsupervised Feature Learning, 2010. find
  11. Mikolov, T., Karafiát, M., Burget, L., Černockỳ, J., and Khudanpur, S. Recurrent neural network based language model. In Eleventh Annual Conference of the International Speech Communication Association, 2010. find
  12. Mikolov, T., Deoras, A., Kombrink, S., Burget, L., and Černockỳ, J. Empirical evaluation and combination of advanced language modeling techniques. In Proceedings of Interspeech, pp. 605–608, 2011. find
  13. Mnih, A. and Hinton, G. Three new graphical models for statistical language modelling. Proceedings of the 24th International Conference on Machine Learning, pp. 641–648, 2007. find
  14. Mnih, A., Yuecheng, Z., and Hinton, G. Improving a statistical language model through non-linear prediction. Neurocomputing, 72(7-9):1414–1418, 2009. find
  15. Mnih, Andriy and Hinton, Geoffrey. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, volume 21, 2009. find
  16. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In AISTATS’05, pp. 246–252, 2005. find
  17. Pihlaja, M., Gutmann, M., and Hyvärinen, A. A family of computationally efficient and simple estimators for unnormalized statistical models. In Proc. Conf. on Uncertainty in Artificial Intelligence (UAI2010)., 2010. find
  18. Schwenk, Holger and Gauvain, Jean-Luc. Training neural network language models on very large corpora. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pp. 201–208, 2005. find
  19. Socher, R., Lin, C.C., Ng, A.Y., and Manning, C.D. Parsing natural scenes and natural language with recursive neural networks. In International Conference on Machine Learning (ICML), 2011. find
  20. Stolcke, A. SRILM – an extensible language modeling toolkit. In Proceedings of the International Conference on Spoken Language Processing, volume 2, pp. 901–904, 2002. find
  21. Turian, J., Ratinov, L., and Bengio, Y. Word representations: A simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384–394, 2010.
  22. Zweig, G. and Burges, C.J.C. The Microsoft Research Sentence Completion Challenge. Technical Report MSR-TR-2011-129, Microsoft Research, 2011. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mnih-2012-fast-simple-algorithm-training-neural/