Elamur Research
☰
home / papers / 2016 / Learning Distributed Representations of Sentences from Unlabelled Data

Learning Distributed Representations of Sentences from Unlabelled Data

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N16-1162.pdf

Summary

Word embeddings learned from raw text are well established, but there is little agreement on how to learn vector representations of phrases and sentences without labelled data. Hill, Cho and Korhonen systematically compare unsupervised and lightly structured approaches:

  • SkipThought
  • ParagraphVector (DM, DBOW)
  • additive CBOW/Skip-Gram composition
  • C-PHRASE
  • NMT encoders
  • a TFIDF BOW baseline
  • DictRep, trained on dictionary definitions
  • CaptionRep, trained on COCO captions

They also propose two new objectives. Sequential Denoising Autoencoders (SDAEs) reconstruct a sentence from a version with deleted and swapped words. FastSent is a cheap log-bilinear bag-of-words model that predicts the words of neighbouring sentences.

Models are tested in two ways: as features for trained classifiers (supervised) and through direct cosine-similarity queries (unsupervised). The best choice depends on the use case. Deeper models such as SkipThought lead most supervised tasks, while shallow log-bilinear models do best when vector spaces are compared directly. DictRep is the strongest all-rounder. These findings offer practical guidance for choosing sentence encoders by application.

Key points

  • SkipThought wins most supervised tasks. SDAE is best on MSRP paraphrase identification, and FastSent tops SICK sentence relatedness.
  • On unsupervised benchmarks, BOW models average 0.55 versus 0.42 for order-sensitive RNN models. In supervised tasks the two groups are nearly equal (76.6 vs 76.3).
  • NMT encoders were trained on WMT 2015 En-Fr and En-De data. CaptionRep maps COCO captions to CNN image vectors, and nearest-neighbour examples come from the Toronto Books Corpus.
  • Cronbach's α across all evaluations is only 0.81. It rises to 0.90 (supervised) and 0.93 (unsupervised) within cohorts, suggesting the two cohorts reward different properties.
  • Practical limits: ParagraphVector could only be trained with 200-dimensional vectors, and needs an extra inference step to encode new sentences.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

196 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 177 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/n16-1162 · arXiv 1602.03483

@inproceedings{hill2016learning,
  title = {{Learning Distributed Representations of Sentences from Unlabelled Data}},
  author = {Hill, Felix and Cho, Kyunghyun and Korhonen, Anna},
  booktitle = {Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies},
  year = {2016},
  doi = {10.18653/v1/n16-1162},
  eprint = {1602.03483},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/n16-1162}
}

References

41 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2… find our card
  2. Amjad Almahairi, Kyle Kastner, Kyunghyun Cho, and Aaron Courville. 2015. Learning distributed representations from reviews for collaborative filtering. In Proceedings of the 9th ACM Conference on Recommender Systems, pages 147–154. find
  3. ACM. Marco Baroni, Raffaela Bernardi, and Roberto Zamparelli. 2014a. Frege in space: A program of compositional distributional semantics. Linguistic Issues in Language Technology, 9. find
  4. We make all code for training and evaluating these new models publicly available, together with pre-trained models and an online demo of the FastSent sentence space. Marco Baroni, Georgiana Dinu, and Germán Kruszewski. 2014b. Don’t count, predict! a systematic comparison of context-counting vs. con… find our card
  5. Elia Bruni, Nam-Khanh Tran, and Marco Baroni. 2014. Multimodal distributional semantics. J. Artif. Intell. Res. (JAIR), 49:1–47. find
  6. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325. Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougar… find our card
  7. Stephen Clark and Stephen Pulman. 2007. Combining symbolic and distributional models of meaning. In AAAI Spring Symposium: Quantum Interaction, pages 52–55. Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) fro… find
  8. Andrew M Dai and Quoc V Le. 2015. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems, pages 3061–3069. find
  9. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, page 350. Association for Computational Linguistics. find our card
  10. Zellig S Harris. 1954. Distributional structure. find
  11. Word. Felix Hill, Kyunghyun Cho, Anna Korhonen, and Yoshua Bengio. 2015a. Learning to understand phrases by embedding the dictionary. Transactions of the Association for Computational Linguistics. find
  12. Felix Hill, Roi Reichart, and Anna Korhonen. 2015b. Simlex-999: Evaluating semantic models with (genuine) similarity estimation. find our card
  13. Computational Linguistics. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pages 168–177. ACM. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015.… find
  14. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics. Yangfeng Ji and Jacob Eisenstein. 2013. Discriminative improvements to distributional sentence similarity. In EMNLP, pages 891–896. find
  15. Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In Proceedings of EMNLP. find our card
  16. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Advances in Neural Information Processing Systems, pages 3276–3284. find
  17. Quoc V Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proceedings of ICML. find our card
  18. Omer Levy and Yoav Goldberg. 2014. Neural word embedding as implicit matrix factorization. In Advances in Neural Information Processing Systems, pages 2177–2185. find
  19. Omer Levy, Yoav Goldberg, and Ido Dagan. 2015. Improving distributional similarity with lessons learned from word embeddings. Transactions of the Association for Computational Linguistics, 3:211–225. find
  20. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan Yulle. 2015. Deep captioning with multimodal recurrent neural networks (m-rnn). In Proceedings of ICLR. find
  21. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A sick cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC, pages 216–223. find
  22. Citeseer. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013a. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781. find
  23. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013b. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111–3119. find
  24. Dmitrijs Milajevs, Dimitri Kartsaklis, Mehrnoosh Sadrzadeh, and Matthew Purver. 2014. Evaluating neural word representations in tensor-based compositional settings. In Proceedings of EMNLP. find
  25. Jeff Mitchell and Mirella Lapata. 2008. Vector-based models of semantic composition. In ACL, pages 236– 244. find
  26. Jeff Mitchell and Mirella Lapata. 2010. Composition in distributional models of semantics. Cognitive science, 34(8):1388–1429. find
  27. Donald A Norman. 1972. Memory, knowledge, and the answering of questions. find
  28. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd annual meeting on Association for Computational Linguistics, page 271. Association for Computational Linguistics. find
  29. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics, pages 115–124. Association for Computational Linguistics. find
  30. Nghia The Pham, Germán Kruszewski, Angeliki Lazaridou, and Marco Baroni. 2015. Jointly optimizing word representations for lexical and sentential tasks with the c-phrase model. In Proceedings of ALC. find
  31. Tamara Polajnar, Laura Rimell, and Stephen Clark. 2015. An exploration of discourse-based sentence spaces for compositional distributional semantics. In Workshop on Linking Models of Lexical, Sentential and Discourse-level Semantics (LSDSem), page 1. find
  32. Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. 2014. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, pages 1–42. find our card
  33. Iulian V Serban, Alessandro Sordoni, Yoshua Bengio, Aaron Courville, and Joelle Pineau. 2015. Building end-to-end dialogue systems using generative hierarchical neural network models. arXiv preprint arXiv:1507.04808. find our card
  34. Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011. Semi-supervised recursive autoencoders for predicting sentiment distributions. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 151–161. Association for Computat… find our card
  35. Ilya Sutskever, James Martens, and Geoffrey E Hinton. 2011. Generating text with recurrent neural networks. In Proceedings of the 28th International Conference on Machine Learning (ICML-11), pages 1017–1024. find
  36. Ilya Sutskever, Oriol Vinyals, and Quoc VV Le. 2014. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, pages 3104–3112. find
  37. Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. 2014. Going deeper with convolutions. arXiv preprint arXiv:1409.4842. — [arXiv:1409.4842](https://arxiv.org/abs/1409.4842) find
  38. Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. 2008. Extracting and composing robust features with denoising autoencoders. In Proceedings of the 25th international conference on Machine learning, pages 1096–1103. find
  39. ACM. Pascal Vincent, Hugo Larochelle, Isabelle Lajoie, Yoshua Bengio, and Pierre-Antoine Manzagol. 2010. Stacked denoising autoencoders: Learning useful representations in a deep network with a local denoising criterion. The Journal of Machine Learning Research, 11:3371–3408. find
  40. Ellen M Voorhees. 2002. Overview of the trec 2001 question answering track. NIST special publication, pages 42–51. find
  41. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2-3):165–210. find