Elamur Research
☰
home / papers / 2018 / An efficient framework for learning sentence representations

An efficient framework for learning sentence representations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1803.02893

Summary

Earlier unsupervised sentence encoders such as skip-thought vectors learn by generating the words of neighbouring sentences with a decoder. This is slow because of the softmax over the vocabulary. It also wastes capacity on surface wording that has little to do with meaning. Logeswaran and Lee propose quick thoughts (QT), which drops generation entirely. An encoder f embeds a sentence, a second encoder g embeds candidate sentences, and a classifier must pick the true context sentence from contrastive candidates drawn from the training data. Because the loss is defined over sentence embeddings rather than tokens, the framework works with different encoders, including RNN and bag-of-words variants. It generalises Skip-gram from words to sentences. Models are trained on BookCorpus and evaluated on transfer benchmarks such as MSRP, TREC, MR, CR, SUBJ, MPQA, SICK and STS14. QT matches or beats prior unsupervised and supervised encoders while training several times faster. This makes high-quality sentence embeddings cheap to train on large unlabelled corpora.

Key points

  • Training efficiency: a single-layer GRU encoder (H = 1000, W = 300, V = 20,000) was trained for one epoch with each objective. QT took 6.5 hours, while the skip-thought objective took 31 hours, an overall 4.8x speedup. The QT setup has about 19.8M parameters versus about 57.7M for skip-thought.
  • Fast bag-of-words variant: a BoW variant of QT trains on BookCorpus in about 2 hours on one Titan X GPU.
  • STS14 results (overall Pearson): QT (RNN) scores 0.49, against 0.39 for Skip-Thoughts and 0.15 for SDAE. QT (BoW) scores 0.65, on par with CBOW (0.65) and above FastSent (0.64) and Siamese CBOW (0.62).
  • Embedding size study (multi-channel MC-QT):
    • A 1600-dimensional model trains in 4 hours and averages 84.57 accuracy across classification tasks, beating the 4800-dimensional Skip-thought model (83.75).
    • The 3200-dimensional model (85.74) surpasses Combine-CNN (85.33).
  • Pre-trained encoders: the authors state that the pre-trained encoders will be released publicly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

164 papers of our library cite it (of the 140673 whose bibliographies we read); 14 of them have a page here, 150 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.02893 · arXiv 1803.02893

@misc{logeswaran2018efficient,
  title = {{An efficient framework for learning sentence representations}},
  author = {Logeswaran, Lajanugen and Lee, Honglak},
  year = {2018},
  doi = {10.48550/arxiv.1803.02893},
  eprint = {1803.02893},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.02893}
}

References

49 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yossi Adi, Einat Kermany, Yonatan Belinkov, Ofer Lavi, and Yoav Goldberg. Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In ICLR, 2017. find our card
  2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel M Cer, Mona T Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. Semeval-2014 task 10: Multilingual semantic textual similarity. In SemEval@ COLING, pp. 81–91, 2014. find our card
  3. Sanjeev Arora, Yingyu Liang, and Tengyu Ma. A simple but tough-to-beat baseline for sentence embeddings.(2016). 2016. find
  4. Samuel R Bowman, Luke Vilnis, Oriol Vinyals, Andrew M Dai, Rafal Jozefowicz, and Samy Bengio. Generating sentences from a continuous space. arXiv preprint arXiv:1511.06349, 2015. find our card
  5. Junyoung Chung, Caglar Gülçehre, Kyunghyun Cho, and Yoshua Bengio. Gated feedback recurrent neural networks. In ICML, pp. 2067–2075, 2015. find
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364, 2017. find our card
  7. Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE International Conference on Computer Vision, pp. 1422–1430, 2015. find
  8. Bill Dolan, Chris Quirk, and Chris Brockett. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th international conference on Computational Linguistics, pp. 350. Association for Computational Linguistics, 2004. find our card
  9. Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, and Lawrence Carin. Unsupervised learning of sentence representations using convolutional neural networks. arXiv preprint arXiv:1611.07897, 2016. — [arXiv:1611.07897](https://arxiv.org/abs/1611.07897) find
  10. Kelvin Guu, Tatsunori B Hashimoto, Yonatan Oren, and Percy Liang. Generating sentences by editing prototypes. arXiv preprint arXiv:1709.08878, 2017. find our card
  11. Lushan Han, Abhay L. Kashyap, Tim Finin, James Mayfield, and Johnathan Weese. UMBC EBIQUITY-CORE: Semantic Textual Similarity Systems. In Proceedings of the Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics, June 2013. find
  12. Karl Moritz Hermann and Phil Blunsom. Multilingual distributed representations without word alignment. arXiv preprint arXiv:1312.6173, 2013. — [arXiv:1312.6173](https://arxiv.org/abs/1312.6173) find
  13. Felix Hill, Kyunghyun Cho, and Anna Korhonen. Learning distributed representations of sentences from unlabelled data. arXiv preprint arXiv:1602.03483, 2016. find our card
  14. Minqing Hu and Bing Liu. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 168–177. ACM, 2004. find
  15. Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. On using very large target vocabulary for neural machine translation. arXiv preprint arXiv:1412.2007, 2014. find our card
  16. Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557, 2017. — [arXiv:1705.00557](https://arxiv.org/abs/1705.00557) find
  17. Yangfeng Ji and Jacob Eisenstein. Discriminative improvements to distributional sentence similarity. In EMNLP, pp. 891–896, 2013. find
  18. Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3128–3137, 2015. find our card
  19. Tom Kenter, Alexey Borisov, and Maarten de Rijke. Siamese cbow: Optimizing word embeddings for sentence representations. arXiv preprint arXiv:1606.04640, 2016. find
  20. Yoon Kim. Convolutional neural networks for sentence classification. arXiv:1408.5882, 2014. find our card
  21. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. Skip-thought vectors. In Advances in Neural Information Processing Systems, pp. 3276–3284, 2015. find
  22. Benjamin Klein, Guy Lev, Gil Sadeh, and Lior Wolf. Associating neural word embeddings with deep image representations using fisher vectors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4437–4446, 2015. find
  23. Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoencoding beyond pixels using a learned similarity metric. arXiv preprint arXiv:1512.09300, 2015. — [arXiv:1512.09300](https://arxiv.org/abs/1512.09300) find
  24. Quoc V Le and Tomas Mikolov. Distributed representations of sentences and documents. In ICML, volume 14, pp. 1188–1196, 2014. find our card
  25. Jiwei Li and Eduard H Hovy. A model of coherence based on distributed sentence representation. In EMNLP, pp. 2039–2048, 2014. find
  26. Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740–755. Springer, 2014. find our card
  27. Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, and Alan Yuille. Deep captioning with multimodal recurrent neural networks (m-rnn). arXiv preprint arXiv:1412.6632, 2014. find
  28. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. A sick cure for the evaluation of compositional distributional semantic models. In LREC, pp. 216–223, 2014. find
  29. Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781, 2013a. find
  30. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013b. find
  31. Tomas Mikolov, Wen-tau Yih, and Geoffrey Zweig. Linguistic regularities in continuous space word representations. In hlt-Naacl, volume 13, pp. 746–751, 2013c. find
  32. Andriy Mnih and Geoffrey E Hinton. A scalable hierarchical distributed language model. In Advances in neural information processing systems, pp. 1081–1088, 2009. find
  33. Mehdi Noroozi and Paolo Favaro. Unsupervised learning of visual representations by solving jigsaw puzzles. In European Conference on Computer Vision, pp. 69–84. Springer, 2016. arXiv preprint Bo Pang and Lillian Lee. A sentimental education: Sentiment analysis using subjectivity summarization based… find
  34. Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of the 43rd annual meeting on association for computational linguistics, pp. 115–124. Association for Computational Linguistics, 2005. find
  35. Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2536–2544, 2016. find
  36. Deepak Pathak, Pulkit Agrawal, Alexei A Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. arXiv preprint arXiv:1705.05363, 2017. find
  37. Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In EMNLP, volume 14, pp. 1532–1543, 2014. find
  38. Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint arXiv:1508.07909, 2015. find
  39. Richard Socher, Eric H Huang, Jeffrey Pennington, Andrew Y Ng, and Christopher D Manning. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In NIPS, volume 24, pp. 801–809, 2011. find
  40. Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, Christopher Potts, et al. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the conference on empirical methods in natural language processing (EMNLP), volume 1… find
  41. Kai Sheng Tai, Richard Socher, and Christopher D Manning. Improved semantic representations from tree-structured long short-term memory networks. arXiv preprint arXiv:1503.00075, 2015. find
  42. Ivan Vendrov, Ryan Kiros, Sanja Fidler, and Raquel Urtasun. Order-embeddings of images and language. arXiv preprint arXiv:1511.06361, 2015. find
  43. Ellen M Voorhees and L Buckland. Overview of the trec 2003 question answering track. In TREC, volume 2003, pp. 54–68, 2003. find
  44. Janyce Wiebe, Theresa Wilson, and Claire Cardie. Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165–210, 2005. find
  45. John Wieting and Kevin Gimpel. Revisiting recurrent networks for paraphrastic sentence embeddings. arXiv preprint arXiv:1705.00364, 2017. find
  46. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. Towards universal paraphrastic sentence embeddings. arXiv preprint arXiv:1511.08198, 2015. find our card
  47. John Wieting, Jonathan Mallinson, and Kevin Gimpel. Learning paraphrastic sentence embeddings from back-translated bitext. arXiv preprint arXiv:1706.01847, 2017. find
  48. Yelp. Yelp dataset challenge. https://www.yelp.com/dataset/challenge, 2017. ↗
  49. Han Zhao, Zhengdong Lu, and Pascal Poupart. Self-adaptive hierarchical sentence model. In IJCAI, pp. 4069–4076, 2015. find