Elamur Research⌕
☰
home / papers / 2003 / A Neural Probabilistic Language Model

A Neural Probabilistic Language Model

Summary

The paper addresses statistical language modeling, that is, estimating the probability of the next word given the preceding ones. The main obstacle here is the "curse of dimensionality": almost every test word sequence is absent from the training data. N-gram models generalize only by gluing together short fragments and do not take similarity between words into account. The authors propose a neural network model that simultaneously learns two things: a continuous feature vector for each word (distributed representation) and a probability function that depends on the vectors of the context words. Words with similar meaning or grammatical role receive similar vectors. As a result, a single training sentence increases the probability of a vast number of "neighboring" sentences. The model was evaluated on two corpora: Brown (over a million examples) and AP News (over 15 million words). It was compared against the best smoothed trigram models. The neural network achieved 10–20% lower perplexity and makes better use of longer context. The paper laid the foundation for word embeddings and neural language models, on which modern NLP rests.

Key points

  • The scale problem: to describe the joint distribution of 10 words with a vocabulary of 100,000 words, the n-gram approach potentially requires about 10^50 free parameters. A model with continuous vectors avoids this exponential growth: computation and memory grow linearly with the number of conditioning variables.
  • Architecture: a word vector table C shared across all positions, with dimensionality m = 30, 60 or 100 (the vocabulary is about 17,000 words). The context vectors are fed into a multilayer neural network with a softmax output. The vectors and the network weights are trained jointly by maximizing a regularized log-likelihood.
  • Result: on the Brown and AP News corpora, perplexity is 10–20% lower than that of a smoothed (interpolated / back-off) trigram. Mixing the neural network with a trigram improves the result even further.
  • Engineering contribution: the authors showed that a model with millions of parameters can be trained on tens of millions of examples in reasonable time, in particular thanks to parallel computation. Later, Bengio and Senécal (2003) proposed a speedup of roughly 100 times via importance sampling.
  • Future directions: a hierarchical (tree-structured) softmax with a gain of up to |V|/log|V|, recurrent or time-delay networks for longer context, multiple vectors for polysemous words, and the use of WordNet and parts of speech. Practical benefit for speech recognition has already been demonstrated by Schwenk and Gauvain (2002).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: AI & machine learning

Topics: Language models & pretraining

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{bengio2003neural,
  title = {{A Neural Probabilistic Language Model}},
  author = {Bengio, Yoshua and Ducharme, Réjean and Vincent, Pascal and Jauvin, Christian},
  booktitle = {JMLR 3:1137–1155},
  year = {2003},
  url = {https://www.jmlr.org/papers/v3/bengio03a.html}
}

References

33 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. D. Baker and A. McCallum. Distributional clustering of words for text classification. In SIGIR’98, 1998. find
  2. J.R. Bellegarda. A latent semantic analysis framework for large–span language modeling. In Proceedings of Eurospeech 97, pages 1451–1454, Rhodes, Greece, 1997. find
  3. S. Bengio and Y. Bengio. Taking on the curse of dimensionality in joint distributions using neural networks. IEEE Transactions on Neural Networks, special issue on Data Mining and Knowledge Discovery, 11(3):550–557, 2000a. find
  4. Y. Bengio. New distributed probabilistic language models. Technical Report 1215, Dept. IRO, Université de Montréal, 2002. find
  5. Y. Bengio and S. Bengio. Modeling high-dimensional discrete data with multi-layer neural networks. In S. A. Solla, T. K. Leen, and K-R. Müller, editors, Advances in Neural Information Processing Systems, volume 12, pages 400–406. MIT Press, 2000b. find
  6. Y. Bengio and J-S. Senécal. Quick training of probabilistic neural nets by importance sampling. In AISTATS, 2003. find
  7. A. Berger, S. Della Pietra, and V. Della Pietra. A maximum entropy approach to natural language processing. Computational Linguistics, 22:39–71, 1996. find our card
  8. A. Brown and G.E. Hinton. Products of hidden markov models. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London, 2000. find
  9. P.F. Brown, V.J. Della Pietra, P.V. DeSouza, J.C. Lai, and R.L. Mercer. Class-based n-gram models of natural language. Computational Linguistics, 18:467–479, 1992. find
  10. S.F. Chen and J.T. Goodman. An empirical study of smoothing techniques for language modeling. Computer, Speech and Language, 13(4):359–393, 1999. find
  11. S. Deerwester, S.T. Dumais, G.W. Furnas, T.K. Landauer, and R. Harshman. Indexing by latent semantic analysis. Journal of the American Society for Information Science, 41(6):391–407, 1990. find
  12. J. Dongarra, D. Walker, and The Message Passing Interface Forum. MPI: A message passing interface standard. Technical Report http://www-unix.mcs.anl.gov/mpi, University of Tenessee, 1995. ↗
  13. J.L. Elman. Finding structure in time. Cognitive Science, 14:179–211, 1990. find
  14. C. Fellbaum. WordNet: An Electronic Lexical Database. MIT Press, 1998. find
  15. J .Goodman. A bit of progress in language modeling. Technical Report MSR-TR-2001-72, Microsoft Research, 2001. find
  16. G.E. Hinton. Learning distributed representations of concepts. In Proceedings of the Eighth Annual Conference of the Cognitive Science Society, pages 1–12, Amherst 1986, 1986. Lawrence Erlbaum, Hillsdale. find
  17. G.E. Hinton. Training products of experts by minimizing contrastive divergence. Technical Report GCNU TR 2000-004, Gatsby Unit, University College London, 2000. find
  18. F. Jelinek and R. L. Mercer. Interpolated estimation of Markov source parameters from sparse data. In E. S. Gelsema and L. N. Kanal, editors, Pattern Recognition in Practice. North-Holland, Amsterdam, 1980. find
  19. K.J. Jensen and S. Riis. Self-organizing letter code-book for text-to-phoneme neural network model. In Proceedings ICSLP, 2000. find
  20. S.M. Katz. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing, ASSP-35 (3):400–401, March 1987. find
  21. R. Kneser and H. Ney. Improved backing-off for m-gram language modeling. In International Conference on Acoustics, Speech and Signal Processing, pages 181–184, 1995. find
  22. Y. LeCun, L. Bottou, G.B. Orr, and K.-R. Müller. Efficient backprop. In G.B. Orr and K.-R. Müller, editors, Neural Networks: Tricks of the Trade, pages 9–50. Springer, 1998. find
  23. R. Miikkulainen and M.G. Dyer. Natural language processing with modular neural networks and distributed lexicon. Cognitive Science, 15:343–399, 1991. find
  24. H. Ney and R. Kneser. Improved clustering techniques for class-based statistical language modelling. In European Conference on Speech Communication and Technology (Eurospeech), pages 973–976, Berlin, 1993. find
  25. T.R. Niesler, E.W.D. Whittaker, and P.C. Woodland. Comparison of part-of-speech and automatically derived category-based language models for speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pages 177–180, 1998. find
  26. A. Paccanaro and G.E. Hinton. Extracting distributed representations of concepts and relations from positive and negative propositions. In Proceedings of the International Joint Conference on Neural Network, IJCNN’2000, Como, Italy, 2000. IEEE, New York. find
  27. F. Pereira, N. Tishby, and L. Lee. Distributional clustering of english words. In 30th Annual Meeting of the Association for Computational Linguistics, pages 183–190, Columbus, Ohio, 1993. find our card
  28. S. Riis and A. Krogh. Improving protein secondary structure prediction using structured neural networks and multiple sequence profiles. Journal of Computational Biology, pages 163–183, 1996. find
  29. J. Schmidhuber. Sequential neural text compression. IEEE Transactions on Neural Networks, 7(1): 142–146, 1996. find
  30. H. Schutze. Word space. In S. J. Hanson, J. D. Cowan, and C. L. Giles, editors, Advances in Neural Information Processing Systems 5, pages pp. 895–902, San Mateo CA, 1993. Morgan Kaufmann. find
  31. H. Schwenk and J-L. Gauvain. Connectionist language modeling for large vocabulary continuous speech recognition. In International Conference on Acoustics, Speech and Signal Processing, pages 765–768, Orlando, Florida, 2002. find
  32. A. Stolcke. SRILM - an extensible language modeling toolkit. In Proceedings of the International Conference on Statistical Language Processing, Denver, Colorado, 2002. find
  33. W. Xu and A. Rudnicky. Can artificial neural network learn language models. In International Conference on Statistical Language Processing, pages M1–13, Beijing, China, 2000. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bengio-2003-neural-probabilistic-language-model/