Elamur Research
☰
home / papers / 2014 / Distributed Representations of Sentences and Documents

Distributed Representations of Sentences and Documents

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1405.4053

Summary

Classifiers and clustering algorithms typically need each text as a fixed-size vector. The standard bag-of-words or bag-of-n-grams encoding discards word order and treats all words as equally dissimilar. Le and Mikolov propose Paragraph Vector, an unsupervised method that gives every sentence, paragraph or document its own dense vector. That vector is trained together with shared word vectors to help predict the words of its text. In the PV-DM variant, the document vector is concatenated with a window of context words to predict the next word. In PV-DBOW, the document vector alone predicts words sampled from the text. For an unseen text, a vector is fitted by gradient descent while the word vectors and softmax weights stay frozen. The result then serves as input to ordinary classifiers such as logistic regression. On the Stanford Sentiment Treebank, IMDB reviews and a retrieval task built from search-result snippets, it beat bag-of-words baselines and set state-of-the-art sentiment results at the time. The approach, later popularised as doc2vec, learns its representations without a parser or labelled data.

Key points

  • Two architectures: PV-DM (Distributed Memory) and PV-DBOW (Distributed Bag of Words). PV-DM alone usually does better, but the authors recommend combining both. Within PV-DM, concatenating vectors works better than summing them (IMDB error 7.63% vs 8.06%).
  • IMDB: the combined model reached 7.42% error. Inferring vectors for the 25,000 test reviews (about 230 words each on average) took roughly 30 minutes on a 16-core machine.
  • Stanford Sentiment Treebank: about 12.2% error on the binary task and 51.3% on the fine-grained task. The authors report a relative error reduction of more than 16% over earlier methods, including parse-based recursive models.
  • Retrieval triplet task (snippets from search results for popular queries): Paragraph Vector had 3.82% error, compared with 5.67% for weighted bag-of-bigrams, 7.28% for bag-of-bigrams, 8.10% for bag-of-words and 10.25% for averaged word vectors. That is roughly a 32% relative improvement.
  • Practical notes: the context window size should be tuned by cross-validation, with 5–12 words a reasonable range. Training uses hierarchical softmax over a Huffman tree. The model has N×p + M×q parameters, but updates are sparse, so training stays efficient.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

811 papers of our library cite it (of the 140673 whose bibliographies we read); 30 of them have a page here, 781 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1405.4053 · arXiv 1405.4053

@misc{le2014distributed,
  title = {{Distributed Representations of Sentences and Documents}},
  author = {Le, Quoc V. and Mikolov, Tomas},
  year = {2014},
  doi = {10.48550/arxiv.1405.4053},
  eprint = {1405.4053},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1405.4053}
}

References

38 references, 5 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Bengio, Yoshua, Schwenk, Holger, Senécal, Jean-Sébastien, Morin, Fréderic, and Gauvain, Jean-Luc. Neural probabilistic language models. In Innovations in Machine Learning, pp. 137–186. Springer, 2006. find
  2. Collobert, Ronan and Weston, Jason. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, pp. 160– 167. ACM, 2008. find
  3. Collobert, Ronan, Weston, Jason, Bottou, Léon, Karlen, Michael, Kavukcuoglu, Koray, and Kuksa, Pavel. Natural language processing (almost) from scratch. The Journal of Machine Learning Research, 12:2493–2537, 2011. find our card
  4. Dahl, George E., Adams, Ryan P., and Larochelle, Hugo. Training Restricted Boltzmann Machines on word observations. In International Conference on Machine Learning, 2012. find
  5. Elman, Jeff. Finding structure in time. In Cognitive Science, pp. 179–211, 1990. find
  6. Frome, Andrea, Corrado, Greg S., Shlens, Jonathon, Bengio, Samy, Dean, Jeffrey, Ranzato, Marc’Aurelio, and Mikolov, Tomas. DeViSE: A deep visual-semantic embedding model. In Advances in Neural Information Processing Systems, 2013. find
  7. Grefenstette, E., Dinu, G., Zhang, Y., Sadrzadeh, M., and Baroni, M. Multi-step regression learning for compositional distributional semantics. In Conference on Empirical Methods in Natural Language Processing, 2013. find
  8. Harris, Zellig. Distributional structure. Word, 1954. find
  9. Huang, Eric, Socher, Richard, Manning, Christopher, and Ng, Andrew Y. Improving word representations via global context and multiple word prototypes. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics: Long Papers-Volume 1, pp. 873–882. Association for Computa… find
  10. Jaakkola, Tommi and Haussler, David. Exploiting generative models in discriminative classifiers. In Advances in Neural Information Processing Systems 11, pp. 487– 493, 1999. find
  11. Klein, Dan and Manning, Chris D. Accurate unlexicalized parsing. In Proceedings of Association for Computational Linguistics, 2003. find our card
  12. Larochelle, Hugo and Lauly, Stanislas. A neural autoregressive topic model. In Advances in Neural Information Processing Systems, 2012. find
  13. Maas, Andrew L., Daly, Raymond E., Pham, Peter T., Huang, Dan, Ng, Andrew Y., and Potts, Christopher. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics, 2011. find our card
  14. Mikolov, Tomas. Statistical Language Models based on Neural Networks. PhD thesis, Brno University of Technology, 2012. find
  15. Mikolov, Tomas, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a. find
  16. Mikolov, Tomas, Le, Quoc V., and Sutskever, Ilya. Exploiting similarities among languages for machine translation. CoRR, abs/1309.4168, 2013b. find
  17. Mikolov, Tomas, Sutskever, Ilya, Chen, Kai, Corrado, Greg, and Dean, Jeffrey. Distributed representations of phrases and their compositionality. In Advances on Neural Information Processing Systems, 2013c. find
  18. Mikolov, Tomas, Yih, Scott Wen-tau, and Zweig, Geoffrey. Linguistic regularities in continuous space word representations. In NAACL HLT, 2013d. find
  19. Mitchell, Jeff and Lapata, Mirella. Composition in distributional models of semantics. Cognitive Science, 2010. find
  20. Mnih, Andriy and Hinton, Geoffrey E. A scalable hierarchical distributed language model. In Advances in Neural Information Processing Systems, pp. 1081–1088, 2008. find
  21. Morin, Frederic and Bengio, Yoshua. Hierarchical probabilistic neural network language model. In Proceedings of the International Workshop on Artificial Intelligence and Statistics, pp. 246–252, 2005. find
  22. Pang, Bo and Lee, Lillian. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of Association for Computational Linguistics, pp. 115–124, 2005. find
  23. Perronnin, Florent and Dance, Christopher. Fisher kernels on visual vocabularies for image categorization. In IEEE Conference on Computer Vision and Pattern Recognition, 2007. find
  24. Perronnin, Florent, Liu, Yan, Sanchez, Jorge, and Poirier, Herve. Large-scale image retrieval with compressed fisher vectors. In IEEE Conference on Computer Vision and Pattern Recognition, 2010. find
  25. Rumelhart, David E, Hinton, Geoffrey E, and Williams, Ronald J. Learning representations by back-propagating errors. Nature, 323(6088):533–536, 1986. find
  26. Socher, Richard, Huang, Eric H., Pennington, Jeffrey, Manning, Chris D., and Ng, Andrew Y. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Advances in Neural Information Processing Systems, 2011a. find
  27. Socher, Richard, Lin, Cliff C, Ng, Andrew, and Manning, Chris. Parsing natural scenes and natural language with recursive neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pp. 129–136, 2011b. find
  28. Socher, Richard, Pennington, Jeffrey, Huang, Eric H, Ng, Andrew Y, and Manning, Christopher D. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, 2011c. find our card
  29. Socher, Richard, Chen, Danqi, Manning, Christopher D., and Ng, Andrew Y. Reasoning with neural tensor networks for knowledge base completion. In Advances in Neural Information Processing Systems, 2013a. find
  30. Socher, Richard, Perelygin, Alex, Wu, Jean Y., Chuang, Jason, Manning, Christopher D., Ng, Andrew Y., and Potts, Christopher. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing, 2013b. find
  31. Srivastava, Nitish, Salakhutdinov, Ruslan, and Hinton, Geoffrey. Modeling documents with deep boltzmann machines. In Uncertainty in Artificial Intelligence, 2013. find
  32. Turian, Joseph, Ratinov, Lev, and Bengio, Yoshua. Word representations: a simple and general method for semi-supervised learning. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pp. 384–394. Association for Computational Linguistics, 2010. find our card
  33. Turney, Peter D. and Pantel, Patrick. From frequency to meaning: Vector space models of semantics. Journal of Artificial Intelligence Research, 2010. find
  34. Wang, Sida and Manning, Chris D. Baselines and bigrams: Simple, good sentiment and text classification. In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics, 2012. find
  35. Yessenalina, Ainur and Cardie, Claire. Compositional matrix-space models for sentiment analysis. In Conference on Empirical Methods in Natural Language Processing, 2011. find
  36. Zanzotto, Fabio, Korkontzelos, Ioannis, Fallucchi, Francesca, and Manandhar, Suresh. Estimating linear models for compositional distributional semantics. In COLING, 2010. find
  37. Zhila, A., Yih, W.T., Meek, C., Zweig, G., and Mikolov, T. Combining heterogeneous models for measuring relational similarity. In NAACL HLT, 2013. find
  38. Zou, Will, Socher, Richard, Cer, Daniel, and Manning, Christopher. Bilingual word embeddings for phrase-based machine translation. In Conference on Empirical Methods in Natural Language Processing, 2013. find