Elamur Research
☰
home / papers / 2021 / Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2108.08877

Summary

The paper asks how to obtain fixed-size sentence embeddings from T5. T5 is an encoder-decoder model pre-trained with span corruption, and it is strong on GLUE-style tasks. However, using it directly for retrieval or clustering is costly, because every query-candidate pair would need full cross-attention inference.

The authors test three pooling strategies:

  • the first encoder token,
  • the mean over encoder outputs,
  • the first decoder output.

Each strategy is placed inside a shared-weight dual encoder and fine-tuned with an in-batch contrastive loss. Training runs in two stages: about 2 billion mined Community QA question-answer pairs first, then a 275K-example contrastive NLI set. Model sizes range from Base up to 11B parameters.

The authors also build SentGLUE, which applies SentEval-style embedding evaluation to nine GLUE tasks. Larger models improve steadily on both transfer and STS. Encoder-only variants are better for transfer, while the encoder-decoder variant performs best on STS. This shows that growth in pre-trained text-to-text models can carry over to efficient embedding models. The models are released on TF Hub.

Key points

  • The three variants are ST5-Enc first, ST5-Enc mean and ST5-EncDec first. Each adds a projection layer and L2 normalization, so a dot product gives cosine similarity. Training uses Adafactor with a learning rate of 0.001, τ of 0.01, and batch sizes of 512 (NLI) and 2048 (Community QA).
  • At 11B parameters, ST5 averages about 84.9 on SentEval STS, ahead of SimCSE-RoBERTa large (83.76) and SBERT large (76.55). Its average on transfer tasks is above 90.
  • SentGLUE covers CoLA, SST-2, MRPC, STS-B, QQP, MNLI-m, MNLI-mm, QNLI and RTE; WNLI was dropped as too hard. ST5-Enc 11B scores 80.07 overall, compared with 83.40 for cross-attention T5 Base.
  • Larger models lower the uniformity loss and raise the alignment loss only slightly. Pre-training on Community QA before NLI adds gains on top of model scaling.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

228 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 217 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2108.08877 · arXiv 2108.08877

@misc{ni2021sentence,
  title = {{Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models}},
  author = {Ni, Jianmo and Abrego, Gustavo Hernandez and Constant, Noah and Ma, Ji and Hall, Keith B. and Cer, Daniel and Yang, Yinfei},
  year = {2021},
  doi = {10.48550/arxiv.2108.08877},
  eprint = {2108.08877},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2108.08877}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. SemEval-2015 task 2: Semantic textual similarity, English, Spanish and pilot on interpretabi… find
  2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. SemEval-2014 task 10: Multilingual semantic textual similarity. In SemEval 2014. find our card
  3. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. SemEval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In SemEval-2016. find our card
  4. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. SemEval-2012 task 6: A pilot on semantic textual similarity. In *SEM 2012/SemEval 2012. find our card
  5. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In *SEM. find our card
  6. Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. ReQA: An evaluation for end-to-end answer retrieval models. In Workshop on Machine Reading for Question Answering. find
  7. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP. find
  8. Daniel Matthew Cer, Mona T. Diab, Eneko Agirre, I. Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In SemEval-2016. find
  9. Daniel Matthew Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, C. Tar, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2018. Universal sentence encoder. In EMNLP. find our card
  10. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In ICLR. find our card
  11. Alexis Conneau and Douwe Kiela. 2018. Senteval: An evaluation toolkit for universal sentence representations. In LREC. find our card
  12. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In EMNLP. find our card
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  14. Kawin Ethayarajh. 2019. How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings. In EMNLP. find
  15. W. Fedus, Barret Zoph, and Noam M. Shazeer. 2021. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. ArXiv, abs/2101.03961. find
  16. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic BERT sentence embedding. CoRR, abs/2007.01852. find our card
  17. Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. Simcse: Simple contrastive learning of sentence embeddings. ArXiv, abs/2104.08821. find our card
  18. D. Gillick, A. Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space. ArXiv, abs/1811.08008. — [arXiv:1811.08008](https://arxiv.org/abs/1811.08008) find
  19. Matthew Henderson, Rami Al-Rfou, B. Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and R. Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652. — [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) find
  20. Ryan Kiros, Yukun Zhu, R. Salakhutdinov, R. Zemel, R. Urtasun, A. Torralba, and S. Fidler. 2015. Skip-thought vectors. In NIPS. find
  21. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv, abs/1907.11692. find
  22. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In ICLR. find our card
  23. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21/140. find
  24. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In EMNLP/IJCNLP. find
  25. Nils Reimers and Iryna Gurevych. 2020. Making monolingual sentence embeddings multilingual using knowledge distillation. In EMNLP. find our card
  26. Noam M. Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In ICML. find
  27. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Neurips. find
  28. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. In NeurIPS. find our card
  29. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In BlackboxNLP@EMNLP. find our card
  30. Tongzhou Wang and Phillip Isola. 2020. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML. find
  31. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. CoRR, abs/1511.08198. find our card
  32. Yinfei Yang, Daniel Matthew Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, G. Ábrego, Steve Yuan, C. Tar, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2020a. Multilingual universal sentence encoder for semantic retrieval. In ACL. find our card
  33. Yinfei Yang, Steve Yuan, Daniel Matthew Cer, Sheng yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2018. Learning semantic textual similarity from conversations. In Rep4NLP@ACL. find
  34. Ziyi Yang, Yinfei Yang, Daniel Cer, Jax Law, and Eric Darve. 2020b. Universal sentence representation learning with conditional masked language model. find