Elamur Research
☰
home / papers / 2018 / SentEval: An Evaluation Toolkit for Universal Sentence Representations

SentEval: An Evaluation Toolkit for Universal Sentence Representations

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1803.05449

Summary

Comparing general-purpose sentence encoders has been difficult because each group builds its own evaluation code, with different preprocessing, downstream classifiers and hyperparameters. Many benchmark datasets are small, so these setup differences can shift scores noticeably, and results from different papers are often not truly comparable. Re-implementing every benchmark also wastes effort. Conneau and Kiela (Facebook AI Research) address this with SentEval, an open-source Python toolkit. It bundles the transfer tasks the community has converged on since SkipThought: sentence classification, natural language inference, semantic relatedness, unsupervised STS, paraphrase detection and caption-image retrieval. It ships download and preprocessing scripts and one shared pipeline with fixed defaults, where only classifier regularization is tuned on validation data. Users connect any encoder, in any framework, by writing a function that turns a batch of sentences into vectors. The contribution is a common, reproducible benchmark rather than a new model. It makes evaluation cheaper and reported numbers fairer to compare, and it can also be used to test word embeddings on downstream tasks.

Key points

  • Classification tasks: MR, CR, SUBJ, MPQA, TREC, SST-2 and SST-5, ranging from about 4k to 70k samples. A Logistic Regression or a one-hidden-layer MLP is trained on frozen embeddings, using nested 10-fold cross-validation for MR, CR, SUBJ and MPQA.
  • Sentence-pair tasks: SNLI, SICK-E, SICK-R, the STS Benchmark and MRPC. Pairs are fed to the classifier as [u, v, |u−v|, u∗v]. Relatedness is scored with Pearson and Spearman correlation.
  • Unsupervised similarity: STS 2012–2016 is evaluated using only cosine similarity, with plain and sample-weighted averages over subtasks. The 2013 SMT subtask is left out for licensing reasons.
  • Caption-image retrieval: COCO with Karpathy splits (113k training images with 5 captions each, 5k validation, 5k test) and precomputed 2048-dimensional ResNet-101 features. Results are reported as Recall@1/5/10 and median rank, averaged over five 1k-image folds.
  • Interface: Users implement batcher and optionally prepare. Settings include task_path, seed, batch_size and kfold (default 10). PyTorch classifiers are trained with Adam at batch size 64, and a bag-of-words example model is included. Code: github.com/facebookresearch/SentEval.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

289 papers of our library cite it (of the 140673 whose bibliographies we read); 24 of them have a page here, 265 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1803.05449 · arXiv 1803.05449

@misc{conneau2018senteval,
  title = {{SentEval: An Evaluation Toolkit for Universal Sentence Representations}},
  author = {Conneau, Alexis and Kiela, Douwe},
  year = {2018},
  doi = {10.48550/arxiv.1803.05449},
  eprint = {1803.05449},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1803.05449}
}

References

42 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Adi, Y., Kermany, E., Belinkov, Y., Lavi, O., and Goldberg, Y. (2017). Fine-grained analysis of sentence embeddings using auxiliary prediction tasks. In Proceedings of ICLR Conference Track, Toulon, France. Published online: https://openreview.net/group?id=ICLR.cc/2017/conference. ↗ our card
  2. Agirre, E., Diab, M., Cer, D., and Gonzalez-Agirre, A. (2012). Semeval-2012 task 6: A pilot on semantic textual similarity. In Proceedings of Semeval-2012, pages 385–393. find our card
  3. Agirre, E., Cer, D., Diab, M., Gonzalez-agirre, A., and Guo, W. (2013). sem 2013 shared task: Semantic textual similarity, including a pilot on typed-similarity. In In *SEM 2013: The Second Joint Conference on Lexical and Computational Semantics. Association for Computational Linguistics. find our card
  4. Agirre, E., Baneab, C., Cardiec, C., Cerd, D., Diabe, M., Gonzalez-Agirre, A., Guof, W., Mihalceab, R., Rigaua, G., and Wiebeg, J. (2014). Semeval-2014 task 10: Multilingual semantic textual similarity. find our card
  5. Agirre, E., Banea, C., Cardie, C., Cer, D. M., Diab, M. T., Gonzalez-Agirre, A., Guo, W., Lopez-Gazpio, I., Maritxalar, M., Mihalcea, R., et al. (2015). Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretability. In SemEval@ NAACL-HLT, pages 252–263. find
  6. Agirre, E., Baneab, C., Cerd, D., Diabe, M., Gonzalez-Agirre, A., Mihalceab, R., Rigaua, G., Wiebef, J., and Donostia, B. C. (2016). Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. Proceedings of SemEval, pages 497–511. find our card
  7. Ba, J. L., Kiros, J. R., and Hinton, G. E. (2016). Layer normalization. Advances in neural information processing systems (NIPS). find our card
  8. Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. (2015). A large annotated corpus for learning natural language inference. In Proceedings of EMNLP. find
  9. Cer, D., Diab, M., Agirre, E., Lopez-Gazpio, I., and Specia, L. (2017). Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evalua-SICK-R SST-B 79.9 82.0 85.8 88.3 - 64.7 70.2 72.1 75.6 - 86.8 2 - tion. arXiv preprint arXiv:1708.00055. find
  10. Chiu, B., Korhonen, A., and Pyysalo, S. (2016). Intrinsic evaluation of word vectors fails to predict extrinsic performance. In First Workshop on Evaluating Vector Space Representations for NLP (RepEval). find
  11. Conneau, A., Kiela, D., Schwenk, H., Barrault, L., and Bordes, A. (2017). Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP, Copenhagen, Denmark. find our card
  12. Dolan, B., Quirk, C., and Brockett, C. (2004). Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of ACL, page 350. find our card
  13. Ettinger, A., Elgohary, A., and Resnik, P. (2016). Probing for semantic evidence of composition by means of simple classification tasks. In First Workshop on Evaluating Vector Space Representations for NLP (RepEval), page 134. find
  14. Faruqui, M., Tsvetkov, Y., Rastogi, P., and Dyer, C. (2016). Problems with evaluation of word embeddings using word similarity tasks. arXiv preprint arXiv:1605.02276. find
  15. He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of CVPR. find
  16. Hill, F., Cho, K., and Korhonen, A. (2016a). Learning distributed representations of sentences from unlabelled data. In Proceedings of NAACL. find our card
  17. Hill, F., Reichart, R., and Korhonen, A. (2016b). Simlex-999: Evaluating semantic models with (genuine) similarity estimation. Computational Linguistics. find our card
  18. Hu, M. and Liu, B. (2004). Mining and summarizing customer reviews. In Proceedings of SIGKDD, pages 168– 177. find
  19. Ji, Y. and Eisenstein, J. (2013). Discriminative improvements to distributional sentence similarity. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  20. Karpathy, A. and Fei-Fei, L. (2015). Deep visual-semantic alignments for generating image descriptions. In Proceedings of CVPR, pages 3128–3137. find our card
  21. Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR). find
  22. Kiros, R., Zhu, Y., Salakhutdinov, R. R., Zemel, R., Urtasun, R., Torralba, A., and Fidler, S. (2015). Skip-thought vectors. In Advances in neural information processing systems, pages 3294–3302. find
  23. Koehn, P., Hoang, H., Birch, A., Callison-Burch, C., Federico, M., Bertoldi, N., Cowan, B., Shen, W., Moran, C., Zens, R., Dyer, C., Bojar, O., Constantin, A., and Herbst, E. (2007). Moses: Open source toolkit for statistical machine translation. In Proceedings of the 45th Annual Meeting of the ACL… find
  24. Lai, A. and Hockenmaier, J. (2014). Illinois-lh: A denotational and distributional approach to semantics. Proc. SemEval, 2:5. find
  25. Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. L. (2014). Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740–755. Springer International Publishing. find our card
  26. Marelli, M., Menini, S., Baroni, M., Bentivogli, L., Bernardi, R., and Zamparelli, R. (2014). A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of LREC. find
  27. Mikolov, T., Grave, E., Bojanowski, P., Puhrsch, C., and Joulin, A. (2017). Advances in pre-training distributed word representations. find our card
  28. Nayak, N., Angeli, G., and Manning, C. D. (2016). Evaluating word embeddings using a representative suite of practical tasks. In First Workshop on Evaluating Vector Space Representations for NLP (RepEval), page 19. find
  29. Pang, B. and Lee, L. (2004). A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of ACL, page 271. find
  30. Pang, B. and Lee, L. (2005). Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proceedings of ACL, pages 115–124. find
  31. Pennington, J., Socher, R., and Manning, C. D. (2014). Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), volume 14, pages 1532–1543. find
  32. Shi, X., Padhi, I., and Knight, K. (2016). Does string-based neural MT learn source syntax? In Proceedings of EMNLP, pages 1526–1534, Austin, Texas. find our card
  33. Socher, R., Perelygin, A., Wu, J. Y., Chuang, J., Manning, C. D., Ng, A. Y., Potts, C., et al. (2013). Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP, pages 1631— 1642. find
  34. Tai, K. S., Socher, R., and Manning, C. D. (2015a). Improved semantic representations from tree-structured long short-term memory networks. Proceedings of ACL. find
  35. Tai, K. S., Socher, R., and Manning, C. D. (2015b). Improved semantic representations from tree-structured long short-term memory networks. Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics (ACL). find
  36. Voorhees, E. M. and Tice, D. M. (2000). Building a question answering test collection. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pages 200–207. ACM. find
  37. Vulić, I., Kiela, D., and Korhonen, A. (2017). Evaluation by association: A systematic study of quantitative word association evaluation. In Proceedings of EACL, volume 1, pages 163–175. find
  38. Wiebe, J., Wilson, T., and Cardie, C. (2005). Annotating expressions of opinions and emotions in language. Language resources and evaluation, 39(2):165–210. find
  39. Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2015). Towards universal paraphrastic sentence embeddings. Proceedings of the 4th International Conference on Learning Representations (ICLR). find our card
  40. Wieting, J., Bansal, M., Gimpel, K., and Livescu, K. (2016). Charagram: Embedding words and sentences via character n-grams. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  41. Zhao, H., Lu, Z., and Poupart, P. (2015). Self-adaptive hierarchical sentence model. In Proceedings of the 24th International Conference on Artificial Intelligence, IJ-CAI’15, pages 4069–4076. AAAI Press. find
  42. Zhou, P., Qi, Z., Zheng, S., Xu, J., Bao, H., and Xu, B. (2016). Text classification improved by integrating bidirectional lstm with two-dimensional max pooling. Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics. find