Elamur Research⌕
☰
home / papers / 2020 / An Unsupervised Sentence Embedding Method by Mutual Information Maximization

An Unsupervised Sentence Embedding Method by Mutual Information Maximization

Summary

Cross-encoding with BERT scales poorly for semantic search and clustering, because every candidate pair must pass through the network. SBERT avoids this by producing standalone sentence vectors, but it relies on labeled NLI pairs and transfers poorly to domains unlike NLI. The authors propose IS-BERT (Info-Sentence BERT), which learns sentence embeddings without labels. It adds a small module on top of BERT: 1-D CNNs with several window sizes turn token outputs into local n-gram features, and mean pooling compresses these into a single sentence vector. Training uses a contrastive objective inspired by DIM. It maximizes mutual information between each sentence vector and its own local features, with local features from other sentences serving as negatives. Because it needs no annotations, the model can be trained directly on a target-domain corpus. On STS and SentEval benchmarks it beats other unsupervised embedding methods and roughly matches supervised encoders like InferSent and USE. It also outperforms SBERT when no in-domain labels exist. This makes it useful for low-resource or specialized domains.

Key points

  • The motivating cost: finding the most similar pair among 10k sentences with BERT requires about 50 million inferences, which takes roughly 65 hours on a V100 GPU.
  • On SentEval, IS-BERT-task averages 85.91 across seven tasks (MR, CR, SUBJ, MPQA, SST, TREC, MRPC) and leads the unsupervised baselines on 6 of them.
    • This average is slightly above InferSent-GloVe (85.59) and USE (85.10).
    • It remains below SBERT-NLI (87.41).
  • Supervised STSb evaluation (8,628 pairs, split 5,749/1,500/1,379) reports Spearman ρ:
    • IS-BERT-STSb (ssl+ft): 84.84
    • SBERT-STSb: 84.67
    • BERT-STSb: 84.30
  • Without the self-supervised stage, IS-BERT-STSb (ft) reaches only 74.25. Running MI pretraining before fine-tuning therefore gives a gain of more than 10 points, so the objective also works as a domain-adaptation step.
  • The authors find that a more complex encoder alone does not guarantee better sentence embeddings; the self-supervised objective is what makes the difference.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

73 papers of our library cite it (of the 140884 whose bibliographies we read); 5 of them have a page here, 68 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.emnlp-main.124

@inproceedings{zhang2020unsupervised,
  title = {{An Unsupervised Sentence Embedding Method by Mutual Information Maximization}},
  author = {Zhang, Yan and He, Ruidan and Liu, Zuozhu and Lim, Kwan Hui and Bing, Lidong},
  booktitle = {Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)},
  year = {2020},
  doi = {10.18653/v1/2020.emnlp-main.124},
  url = {https://doi.org/10.18653/v1/2020.emnlp-main.124}
}

References

44 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Iñigo Lopez-Gazpio, Montse Maritxalar, Rada Mihalcea, German Rigau, Larraitz Uria, and Janyce Wiebe. 2015. Semeval-2015 task 2: Semantic textual similarity, english, spanish and pilot on interpretab… find our card
  2. Eneko Agirre, Carmen Banea, Claire Cardie, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Weiwei Guo, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2014. Semeval-2014 task 10: Multilingual semantic textual similarity. In Proc. of SemEval@ACL. find our card
  3. Eneko Agirre, Carmen Banea, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, Rada Mihalcea, German Rigau, and Janyce Wiebe. 2016. Semeval-2016 task 1: Semantic textual similarity, monolingual and cross-lingual evaluation. In Proc. of SemEval@ACL. find our card
  4. Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. Semeval-2012 task 6: A pilot on semantic textual similarity. In SemEval@ACL. find our card
  5. Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *sem 2013 shared task: Semantic textual similarity. In The Second Joint Conference on Lexical and Computational Semantics. find our card
  6. Mohamed Ishmael Belghazi, Aristide Baratin, Sai Rajeswar, Sherjil Ozair, Yoshua Bengio, Aaron Courville, and R Devon Hjelm. 2018. Mine: mutual information neural estimation. find
  7. Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proc. of EMNLP. find our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL-HLT. find
  9. Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proc. of COLING. find our card
  10. Zhe Gan, Yunchen Pu, Ricardo Henao, Chunyuan Li, Xiaodong He, and Lawrence Carin. 2017. Learning generic sentence representations using convolutional neural networks. In Proc. of EMNLP. find
  11. Michael U Gutmann and Aapo Hyvärinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. Journal of Machine Learning Research, 13(Feb):307–361. find
  12. Felix Hill, Kyunghyun Cho, and Anna Korhonen. 2016. Learning distributed representations of sentences from unlabelled data. In Proc. of NAACL-HLT. find our card
  13. Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Philip Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In Proc. of ICLR. find
  14. Minqing Hu and Bing Liu. 2004. Mining and summarizing customer reviews. In Proc. of KDD. find
  15. Anthony J Bell and Terrence J Sejnowski. 1995. An information-maximization approach to blind separation and blind deconvolution. Neural computation, 7(6):1129–1159. find
  16. Aapo Hyvärinen and Erkki Oja. 2000. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411–430. find
  17. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proc. of EMNLP. find
  18. Yacine Jernite, Samuel R. Bowman, and David A. Sontag. 2017. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557. — [arXiv:1705.00557](https://arxiv.org/abs/1705.00557) find
  19. Xu Ji, F. João Henriques, and Andrea Vedaldi. 2019. Invariant information clustering for unsupervised image classification and segmentation. arXiv preprint arXiv:1807.06653. — [arXiv:1807.06653](https://arxiv.org/abs/1807.06653) find
  20. Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882. find our card
  21. Ryan Kiros, Yukun Zhu, Russ R Salakhutdinov, Richard Zemel, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Skip-thought vectors. In Proc. of NIPS. find
  22. Lingpeng Kong, Cyprien de Masson d’Autume, Lei Yu, Wang Ling, Zihang Dai, and Dani Yogatama. 2020. A mutual information maximization perspective of language representation learning. In Proc. of ICLR. find
  23. Quoc V. Le and Tomas Mikolov. 2014. Distributed representations of sentences and documents. In Proc. of ICML. find our card
  24. Xin Li and Dan Roth. 2002. Learning question classifiers. In Proc. of COLING. find our card
  25. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692. find
  26. Lajanugen Logeswaran and Honglak Lee. 2018. An efficient framework for learning sentence representations. In Proc. of ICLR. find our card
  27. Marco Marelli, Stefano Menini, Marco Baroni, Luisa Bentivogli, Raffaella Bernardi, and Roberto Zamparelli. 2014. A SICK cure for the evaluation of compositional distributional semantic models. In Proc. of LREC. find
  28. Amita Misra, Brian Ecker, and Marilyn Walker. 2016. Measuring the similarity of sentential arguments in dialogue. In Proc. of the 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue. find
  29. Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. 2016. f-gan: Training generative neural samplers using variational divergence minimization. In Proc. of NIPS. find
  30. Aäron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748. find our card
  31. Matteo Pagliardini, Prakhar Gupta, and Martin Jaggi. 2017. Unsupervised learning of sentence embeddings using compositional n-gram features. arXiv preprint arXiv:1703.02507. find our card
  32. Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Proc. of ACL. find our card
  33. Nils Reimers, Philip Beyer, and Iryna Gurevych. 2016. Task-oriented intrinsic evaluation of semantic textual similarity. In Proc. of COLING. find
  34. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proc. of EMNLP-IJCNLP. find
  35. Nils Reimers, Benjamin Schiller, Tilman Beck, Johannes Daxenberger, Christian Stab, and Iryna Gurevych. 2019. Classification and clustering of arguments with contextualized word embeddings. In Proc. of ACL. find our card
  36. Richard Socher, Eric H. Huang, Jeffrey Pennin, Christopher D Manning, and Andrew Y. Ng. 2011. Dynamic pooling and unfolding recursive autoencoders for paraphrase detection. In Proc. of NIPS. find
  37. Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. of EMNLP. find
  38. Fan-Yun Sun, Jordan Hoffmann, and Jian Tang. 2020. Infograph: Unsupervised and semi-supervised graph-level representation learning via mutual information maximization. In Proc. of ICLR. find
  39. Petar Velickovic, William Fedus, William L. Hamilton, Pietro Liò, Yoshua Bengio, and R. Devon Hjelm. 2019. Deep graph infomax. In Proc. of ICLR. find
  40. Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005. Annotating expressions of opinions and emotions in language. Language Resources and Evaluation. find
  41. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proc. of NAACL-HLT. find
  42. Yinfei Yang, Steve Yuan, Daniel Matthew Cer, Sheng yi Kong, Noah Constant, P. Pilar, Heming Ge, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2018. Learning semantic textual similarity from conversations. In Proc. of RepL4NLP@ACL. find
  43. Yukun Zhu, Ryan Kiros, S. Richard Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proc. of ICCV. find our card
  44. Bo Pang and Lillian Lee. 2004. A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts. In Proc. of ACL. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2020-unsupervised-sentence-embedding-method-mutual/