Elamur Research
☰
home / papers / 2021 / Unsupervised Dense Information Retrieval with Contrastive Learning

Unsupervised Dense Information Retrieval with Contrastive Learning

Written for the Research library (claude-opus-5-5) from the paper text. Original — arxiv.org/pdf/2112.09118

Summary

Neural (dense) retrievers work well when there are many labelled query–document pairs. On new domains or languages without such data, they often do worse than BM25, the classic term-matching baseline. The authors ask whether a dense bi-encoder can be trained with no supervision at all and still compete. Their model, Contriever, uses contrastive learning. Two spans cropped independently at random from the same text form a positive pair, and other texts act as negatives. The negatives are stored in a MoCo queue, so their number can grow without larger batches. Pre-training uses Wikipedia and CCNet. With no labels at all, Contriever beats BM25 on Recall@100 for 11 of 15 BEIR datasets. It also works as a starting point for fine-tuning, either on a few thousand in-domain examples or on MS MARCO, and improves BEIR scores in both cases. Adding a cross-encoder re-ranker gives a new BEIR state of the art. A multilingual version also retrieves well without supervision, including across scripts. This matters for low-resource languages, where annotated retrieval data is rare.

Key points

  • Data augmentation: independent random cropping works clearly better than the inverse Cloze task (ICT). In the unsupervised setting, the average nDCG@10 is 32.2 for cropping and 25.9 for ICT. Adding word deletion raises it slightly to 33.8.
  • Number of negatives: the authors tried MoCo queue sizes from 2,048 to 131,072. More negatives generally improve BEIR results, most strongly before fine-tuning, but the gain differs from one dataset to another.
  • Training corpus: neither source wins overall. CCNet helps on out-of-domain sets such as FiQA and Quora, while Wikipedia is better on FEVER. Mixing the two ("uniform" or "50/50%" sampling) balances these effects.
  • Effect of pre-training: with the same MS MARCO fine-tuning, average nDCG@10 is 46.5 for Contriever and 42.0 for BERT base uncased. So most of the improvement comes from the contrastive pre-training itself.
  • Multilingual results: a model fine-tuned only on English transfers well to low-resource languages such as Swahili. It can also retrieve English documents for Arabic queries, which term matching cannot do. Code and models are at github.com/facebookresearch/contriever.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

672 papers of our library cite it (of the 140673 whose bibliographies we read); 37 of them have a page here, 635 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2112.09118 · arXiv 2112.09118

@misc{izacard2021unsupervised,
  title = {{Unsupervised Dense Information Retrieval with Contrastive Learning}},
  author = {Izacard, Gautier and Caron, Mathilde and Hosseini, Lucas and Riedel, Sebastian and Bojanowski, Piotr and Joulin, Armand and Grave, Edouard},
  year = {2021},
  doi = {10.48550/arxiv.2112.09118},
  eprint = {2112.09118},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.09118}
}

References

65 references, 26 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Xinyan Yu, Jungo Kasai, and Hannaneh Hajishirzi. One question answering model for many languages with cross-lingual dense passage retrieval. CoRR, abs/2107.11976, 2021. URL https://arxiv. org/abs/2107.11976. 10, 11 — [arXiv:2107.11976](https://arxiv.org/abs/2107.11976) find
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. 3, 6 find our card
  3. Adam Berger, Rich Caruana, David Cohn, Dayne Freitag, and Vibhu Mittal. Bridging the lexical chasm: statistical approaches to answer-finding. In Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval, pp. 192–199, 2000. 1 find
  4. Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv preprint arXiv:2104.14294, 2021. 2 find our card
  5. Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding-based large-scale retrieval. arXiv preprint arXiv:2002.03932, 2020. 2, 3 — [arXiv:2002.03932](https://arxiv.org/abs/2002.03932) find
  6. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open-domain questions. In Proc. ACL, 2017a. 1 find our card
  7. Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. On sampling strategies for neural network-based collaborative filtering. arXiv preprint arXiv:1706.07881, 2017b. 5 — [arXiv:1706.07881](https://arxiv.org/abs/1706.07881) find
  8. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597–1607. PMLR, 2020. 2, 4, 5 find our card
  9. Xilun Chen, Kushal Lakhotia, Barlas Oğuz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one?, 2021. URL https://arxiv.org/abs/2110.06918. 3 ↗
  10. Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. Tydi QA: A benchmark for information-seeking question answering in typologically diverse languages. CoRR, abs/2003.05002, 2020. URL https://arxiv.org/abs/2003.05002. 9 ↗ our card
  11. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116, 2019. URL http://arxiv.org/abs/1911.02116. 9, 1… ↗
  12. Scott Deerwester, Susan T Dumais, George W Furnas, Thomas K Landauer, and Richard Harshman. Indexing by latent semantic analysis. Journal of the American society for information science, 41(6):391–407, 1990. 2 find
  13. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proc. NAACL, 2019. 3, 4 find
  14. Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. Cert: Contrastive self-supervised learning for language understanding. arXiv preprint arXiv:2005.12766, 2020. 3 — [arXiv:2005.12766](https://arxiv.org/abs/2005.12766) find
  15. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. Splade v2: Sparse lexical and expansion model for information retrieval. arXiv preprint arXiv:2109.10086, 2021. 7 find
  16. Luyu Gao and Jamie Callan. Unsupervised corpus aware language model pre-training for dense passage retrieval. arXiv preprint arXiv:2108.05540, 2021. 3 find our card
  17. Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simcse: Simple contrastive learning of sentence embeddings. arXiv preprint arXiv:2104.08821, 2021. 5, 7 find our card
  18. Daniel Gillick, Alessandro Presta, and Gaurav Singh Tomar. End-to-end retrieval in continuous space. arXiv preprint arXiv:1811.08008, 2018. 3 — [arXiv:1811.08008](https://arxiv.org/abs/1811.08008) find
  19. John M Giorgi, Osvald Nitski, Gary D Bader, and Bo Wang. Declutr: Deep contrastive learning for unsupervised textual representations. arXiv preprint arXiv:2006.03659, 2020. 3 find our card
  20. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909, 2020. 3, 5, 7 find
  21. Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729–9738, 2020. 2, 6, 17 find our card
  22. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently teaching an effective dense retriever with balanced topic aware sampling. arXiv preprint arXiv:2104.06967, 2021. 7 find our card
  23. Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pp. 2333–2338, 2013. 2, 4 find
  24. Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. Poly-encoders: Transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring. arXiv preprint arXiv:1905.01969, 2019. 3 — [arXiv:1905.01969](https://arxiv.org/abs/1905.01969) find
  25. Gautier Izacard and Edouard Grave. Distilling knowledge from reader to retriever for question answering, 2020a. URL https://arxiv.org/abs/2012.04584. 3, 7 — [arXiv:2012.04584](https://arxiv.org/abs/2012.04584) ↗
  26. Gautier Izacard and Edouard Grave. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020b. 6 find our card
  27. Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Sebastian Riedel, and Edouard Grave. A memory efficient baseline for open domain question answering. arXiv preprint arXiv:2012.15156, 2020. 3 — [arXiv:2012.15156](https://arxiv.org/abs/2012.15156) find
  28. Yacine Jernite, Samuel R Bowman, and David Sontag. Discourse-based objectives for fast unsupervised sentence representation learning. arXiv preprint arXiv:1705.00557, 2017. 3 — [arXiv:1705.00557](https://arxiv.org/abs/1705.00557) find
  29. Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 2019. 3 find our card
  30. Karen Sparck Jones. A statistical interpretation of term specificity and its application in retrieval. Journal of documentation, 1972. 2 find
  31. Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. ACL, 2017. 6, 7 find our card
  32. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906, 2020. 3, 4, 5, 7 find our card
  33. Omar Khattab, Christopher Potts, and Matei Zaharia. Relevance-guided supervision for openqa with colbert, 2020. 3, 7 find
  34. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:4… find our card
  35. Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and In Kwon Choi. ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. arXiv preprint arXiv:2102.11600, 2021. 17 — [arXiv:2102.11600](https://arxiv.org/abs/2102.11600) find
  36. Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942, 2019. 3 find
  37. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. Latent retrieval for weakly supervised open domain question answering. In Proc. ACL, 2019. 2, 3, 4, 5, 6 find our card
  38. Mike Lewis, Marjan Ghazvininejad, Gargi Ghosh, Armen Aghajanyan, Sida Wang, and Luke Zettlemoyer. Pre-training via paraphrasing. arXiv preprint arXiv:2006.15020, 2020. 3 find
  39. Jimmy Lin, Rodrigo Nogueira, and Andrew Yates. Pretrained transformers for text ranking: Bert and beyond. arXiv preprint arXiv:2010.06467, 2020. 2 — [arXiv:2010.06467](https://arxiv.org/abs/2010.06467) find
  40. Shayne Longpre, Yi Lu, and Joachim Daiber. MKQA: A linguistically diverse benchmark for multilingual open domain question answering. CoRR, abs/2007.15207, 2020. URL https://arxiv.org/abs/2007.15207. 9 ↗
  41. Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. 17, 18 find
  42. Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181, 2020. 3 find our card
  43. Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. A replication study of dense passage retriever, 2021. 7 find
  44. Christopher D Manning, Hinrich Schütze, and Prabhakar Raghavan. Introduction to information retrieval. Cambridge university press, 2008. 2 find
  45. Yu Meng, Chenyan Xiong, Payal Bajaj, Saurabh Tiwary, Paul Bennett, Jiawei Han, and Xia Song. Coco-lm: Correcting and contrasting text sequences for language model pretraining. arXiv preprint arXiv:2102.08473, 2021. 3 find
  46. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pp. 3111–3119, 2013. 3 find
  47. Bhaskar Mitra, Nick Craswell, et al. An introduction to neural information retrieval. Foundations and Trends® in Information Retrieval, 13(1):1–126, 2018. 2 find
  48. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. In CoCo@ NIPS, 2016. 1 find our card
  49. Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with bert. arXiv preprint arXiv:1901.04085, 2019. 2, 4 find our card
  50. Hamid Palangi, Li Deng, Yelong Shen, Jianfeng Gao, Xiaodong He, Jianshu Chen, Xinying Song, and Rabab Ward. Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 24(4):694–707… find
  51. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Associati… find our card
  52. Ori Ram, Gal Shachaf, Omer Levy, Jonathan Berant, and Amir Globerson. Learning to retrieve passages without supervision, 2021. URL https://arxiv.org/abs/2112.07708. 3 ↗
  53. Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019. 3, 4 find
  54. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc, 2009. 1 find our card
  55. Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. Okapi at TREC-3. NIST Special Publication Sp, 1995. 2 find our card
  56. Devendra Singh Sachan, Siva Reddy, William Hamilton, Chris Dyer, and Dani Yogatama. End-to-end training of multi-document reader and retriever for open-domain question answering, 2021. 2, 7 find
  57. Yelong Shen, Xiaodong He, Jianfeng Gao, Li Deng, and Grégoire Mesnil. Learning semantic representations using convolutional neural networks for web search. In Proceedings of the 23rd international conference on world wide web, pp. 373–374, 2014. 2 find
  58. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. Beir: A heterogenous benchmark for zero-shot evaluation of information retrieval models. arXiv preprint arXiv:2104.08663, 2021. 2, 6, 18, 19 find our card
  59. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. Fever: a large-scale dataset for fact extraction and verification. arXiv preprint arXiv:1803.05355, 2018. 1 find our card
  60. Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave. CCNet: Extracting high quality monolingual datasets from web crawl data. In Proceedings of the 12th Language Resources and Evaluation Conference, 2020. 6, 9, 17 find our card
  61. Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non-parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733–3742, 2018. 2, 4, 6 find
  62. Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. Clear: Contrastive learning for sentence representation. arXiv preprint arXiv:2012.15466, 2020. 3 — [arXiv:2012.15466](https://arxiv.org/abs/2012.15466) find
  63. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808, 2020. 3, 4, 7 find our card
  64. Sohee Yang and Minjoon Seo. Is retriever merely an approximator of reader? arXiv preprint arXiv:2010.10999, 2020. 3 — [arXiv:2010.10999](https://arxiv.org/abs/2010.10999) find
  65. Xinyu Zhang, Xueguang Ma, Peng Shi, and Jimmy Lin. Mr. tydi: A multi-lingual benchmark for dense retrieval. CoRR, abs/2108.08787, 2021. URL https://arxiv.org/abs/2108.08787. 9, 10 ↗