Elamur Research
☰
home / papers / 2018 / Universal Sentence Encoder for English

Universal Sentence Encoder for English

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-2029.pdf

Summary

The paper addresses the need for general-purpose sentence representations that transfer well to many NLP tasks, especially when labeled data is scarce. The authors release two English encoders on TensorFlow Hub. The first is based on the transformer encoder, which is more accurate but more expensive. The second is a deep averaging network (DAN), which is cheaper but less precise. Both are trained with multi-task learning: a Skip-Thought-style objective, conversational response prediction, and supervised classification on SNLI. Unsupervised data comes from Wikipedia, web news, question-answer pages and discussion forums. Evaluated on classification benchmarks and the STS Benchmark, sentence-level transfer generally beats baselines with no transfer and often beats those relying only on word2vec embeddings. Combining sentence- and word-level features frequently gives the best scores. The advantage is largest in low-data settings. The authors also measure CPU, GPU and memory costs as sentence length grows, and probe social bias with WEAT. The work matters because it offers ready-made, openly licensed embeddings with an explicit accuracy-versus-cost choice.

Key points

  • Two public models (Apache 2.0, TensorFlow Hub) take raw untokenized text as input.
    • The transformer variant scales as O(n²) in compute and memory with sentence length.
    • The DAN variant scales as O(n).
    • Out-of-vocabulary words are hashed into 400,000 OOV embeddings.
  • On the STS Benchmark, the transformer encoder reaches Pearson r 0.766 on test (DAN: 0.717), beating InferSent (0.758). Task-tuned systems such as gConv (0.808) remain stronger.
  • With only 1k SST training examples, the transformer embeddings reach 84.8. This exceeds InferSent trained on the full 67.3k set.
  • The transformer encoder beats InferSent on MR, SUBJ and TREC, and outperforms the DAN encoder on every classification task.
  • On WEAT, both encoders reproduce harmless associations such as flowers vs. insects. They show weaker ageist, racial and gender associations than GloVe, although the authors did not specifically optimize for bias reduction.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

267 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 257 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-2029

@inproceedings{cer2018universal,
  title = {{Universal Sentence Encoder for English}},
  author = {Cer, Daniel and Yang, Yinfei and Kong, Sheng-yi and Hua, Nan and Limtiaco, Nicole and John, Rhomni St. and Constant, Noah and Guajardo-Cespedes, Mario and Yuan, Steve and Tar, Chris and Strope, Brian and Kurzweil, Ray},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations},
  year = {2018},
  doi = {10.18653/v1/d18-2029},
  url = {https://doi.org/10.18653/v1/d18-2029}
}

References

27 references, 3 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Martı́n Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. Tensorflow: A system for large-scale machine learning. In Proceedings of USENIX OSDI’16. find
  2. Ryan Kiros, Yukun Zhu, Ruslan R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In In Proceedings of NIPS. find
  3. Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. CoRR, abs/1607.06450. ↗ our card
  4. Marianne Bertrand and Sendhil Mullainathan. 2004. Are emily and greg more employable than lakisha and jamal? a field experiment on labor market discrimination. The American Economic Review, 94(4). find
  5. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proceedings of COLING ’02. find
  6. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of NIPS’13. find
  7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. find
  8. Lindsey L. Monteith and Jeremy W. Pettit. 2011. Implicit and explicit stigmatizing attitudes and stereotypes about depression. Journal of Social and Clinical Psychology, 30(5). find
  9. Aylin Caliskan, Joanna J. Bryson, and Arvind Narayanan. 2017. Semantics derived automatically from language corpora contain human-like biases. Science, 356(6334):183–186. find
  10. Brian A. Nosek, Mahzarin R. Banaji, and Anthony G. Greenwald. 2002a. Harvesting implicit group attitudes and beliefs from a demonstration web site. Group Dynamics, 6(1). find
  11. Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of SemEval-2017. find
  12. Brian A. Nosek, Mahzarin R. Banaji, and Anthony G Greenwald. 2002b. Math = male, me = female, therefore math me. Journal of Personality and Social Psychology,, 83(1). find
  13. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364. ↗ our card
  14. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of the 42nd Meeting of the Association for Computational Linguistics (ACL’04), Main Volume. find
  15. Daniel Golovin, Benjamin Solnik, Subhodeep Moitra, Greg Kochanski, John Karro, and D. Sculley. 2017. Google vizier: A service for black-box optimization. In Proceedings of KDD ’17. find
  16. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL’05. find
  17. Anthony G. Greenwald, Debbie E. McGhee, and Jordan L. K. Schwartz. 1998. Measuring individual differences in implicit cognition: the implicit association test. Journal of personality and social psychology, 74(6). find
  18. Yang Shao. 2017. Hcti at semeval-2017 task 1: Use convolutional neural network to evaluate semantic textual similarity. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 130–133. find
  19. Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017. Efficient natural language response suggestion for smart reply. CoRR, abs/1705.00652. ↗
  20. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP. find
  21. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural Comput., 9(8):1735– 1780. find
  22. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS. find
  23. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proceedings of KDD ’04. find
  24. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation, 39(2):165–210. find
  25. Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daumé III. 2015. Deep unordered composition rivals syntactic methods for text classification. In Proceedings of ACL/IJCNLP. find
  26. Yinfei Yang, Steve Yuan, Daniel Cer, Sheng yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Learning semantic textual similarity from conversations. Proceedings of RepL4NLP workshop at ACL. find
  27. Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of EMNLP. find our card