Elamur Research
☰
home / papers / 2018 / Universal Sentence Encoder

Universal Sentence Encoder

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1803.11175

Summary

Most NLP tasks have only small labelled datasets, and the usual workaround is to reuse pretrained word vectors such as word2vec or GloVe. Cer et al. instead release pretrained encoders that turn a whole English sentence into a single 512-dimensional vector for reuse across tasks. There are two variants. One is built on the Transformer encoder and favours accuracy. The other is a deep averaging network (DAN) that favours speed. Both were trained jointly on several objectives: a Skip-Thought-style task, conversational input-response pairs, and supervised classification on SNLI. Training text came from Wikipedia, news, Q&A sites and forums.

Across seven benchmarks (MR, CR, SUBJ, MPQA, TREC, SST, STS Benchmark), sentence-level transfer generally beat word-level transfer. The best results came from combining sentence and word embeddings. The gains were especially large when task data was scarce.

The authors also measured CPU, GPU and memory costs and probed social bias with WEAT. The models are distributed through TF Hub, which makes strong sentence representations easy to plug into downstream systems.

Key points

  • Two encoders with different costs. USE T (Transformer) scales as O(n²) in both time and memory with sentence length. USE D (DAN) has linear compute time and constant memory. For short inputs, however, the Transformer needs roughly half the DAN's memory, because it stores no bigram embeddings.
  • STS Benchmark. Similarity is computed from angular distance between embeddings, without task-specific training. Pearson r reaches 0.814 (dev) / 0.782 (test) for USE T and 0.763 / 0.719 for USE D.
  • Best classification scores. USE T combined with a word2vec-based CNN gave the top results, for example 98.07 on TREC and 86.69 on SST.
  • Low-data setting. With only 1,000 SST training examples, USE T approaches many models trained on the full 67.3k set.
  • WEAT bias probes. The DAN model reproduces the flowers/insects pleasantness association. It shows weaker age, race and gender associations than GloVe.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

397 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 381 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.11175 · arXiv 1803.11175

@misc{cer2018universal,
  title = {{Universal Sentence Encoder}},
  author = {Cer, Daniel and Yang, Yinfei and Kong, Sheng-yi and Hua, Nan and Limtiaco, Nicole and John, Rhomni St. and Constant, Noah and Guajardo-Cespedes, Mario and Yuan, Steve and Tar, Chris and Sung, Yun-Hsuan and Strope, Brian and Kurzweil, Ray},
  year = {2018},
  doi = {10.48550/arxiv.1803.11175},
  eprint = {1803.11175},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.11175}
}

References

25 references, 2 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yu… find
  2. Marianne Bertrand and Sendhil Mullainathan. 2004. Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination. The American Economic Review, 94(4). find
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. find
  4. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  5. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of SemEval-2017. find
  6. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. find our card
  7. Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. Google vizier: A service for black-box optimization. In Proceedings of KDD ’17. find
  8. Anthony G. Greenwald, Debbie E. McGhee, and Jordan L. K. Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6). find
  9. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR, abs/1705.00652. — [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) find
  10. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. find
  11. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of KDD ’04. find
  12. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of ACL/IJCNLP. find
  13. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP. find our card
  14. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In In Proceedings of NIPS. find
  15. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of COLING ’02. find
  16. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS’13. find
  17. Lindsey L. Monteith and Jeremy W. Pettit. 2011. Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal of Social and Clinical Psychology, 30(5). find
  18. Brian A. Nosek, Mahzarin R. Banaji, and Anthony G. Greenwald. 2002a. Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics, 6(1). find
  19. Brian A. Nosek, Mahzarin R. Banaji, and Anthony G Greenwald. 2002b. Math = male, me = female, therefore math me. Journal of Personality and Social Psychology,, 83(1). find
  20. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume. find
  21. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL’05. find
  22. Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceeding of EMNLP. find
  23. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP. find
  24. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. find
  25. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210. find