Elamur Research
☰
home / papers / 2021 / Large Dual Encoders Are Generalizable Retrievers

Large Dual Encoders Are Generalizable Retrievers

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2112.07899

Summary

Dense retrievers built as dual encoders often transfer poorly to new domains. A common explanation is that a single dot product between fixed-size query and passage vectors is too weak a scoring function. This paper tests that view by enlarging the encoder while keeping the 768-dimensional embedding unchanged. The authors build Generalizable T5-based dense Retrievers (GTR) from the encoder part of T5 checkpoints, from Base up to XXL. Each model uses a single parameter-shared tower and mean pooling. Training has two stages. First, the model is pre-trained with a contrastive in-batch softmax loss on about two billion web-mined community question-answer pairs. Second, it is fine-tuned on human-annotated search data, namely MS Marco or Natural Questions. In zero-shot evaluation on BEIR, larger models consistently generalize better. GTR-XXL reaches state-of-the-art results and beats several models with richer query–document interaction. The findings suggest that encoder capacity, rather than the bottleneck, was holding back single-vector retrievers. They also indicate that strong out-of-domain retrieval is possible with little labelled data.

Key points

  • GTR comes in four sizes: Base (110M), Large (335M), XL (1.24B) and XXL (4.8B) parameters. All sizes produce 768-dimensional embeddings scored with cosine similarity. The models are released on TF Hub.
  • Evaluation covers 18 BEIR tasks across 9 domains, with MS Marco excluded from the zero-shot comparison.
  • Pre-training on community QA from sites such as Reddit and Stack Overflow and fine-tuning on human-curated data are both needed to exploit the larger models. GTR matches its best out-of-domain performance using only 10% of MS Marco's 532K pairs.
  • Larger models tend to retrieve longer documents. On Web-Touche2020, GTR-XXL returns documents about twice as long as those from GTR-Base. Trec-Covid is an exception, where the shorter retrievals may explain XXL's weaker score.
  • Inference latency (batch size 1, 128 tokens) rises from 17 to 349 ms across model sizes. GTR-Base is comparable to TAS-B, while GTR-XXL is close to re-ranking models.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

271 papers of our library cite it (of the 140673 whose bibliographies we read); 19 of them have a page here, 252 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2112.07899 · arXiv 2112.07899

@misc{ni2021large,
  title = {{Large Dual Encoders Are Generalizable Retrievers}},
  author = {Ni, Jianmo and Qu, Chen and Lu, Jing and Dai, Zhuyun and Abrego, Gustavo Hernandez and Ma, Ji and Zhao, Vincent Y. and Luan, Yi and Hall, Keith B. and Chang, Ming-Wei and Yang, Yinfei},
  year = {2021},
  doi = {10.48550/arxiv.2112.07899},
  eprint = {2112.07899},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2112.07899}
}

References

32 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Amin Ahmad, Noah Constant, Yinfei Yang, and Daniel Cer. 2019. ReQA: An evaluation for end-to-end answer retrieval models. In Workshop on Machine Reading for Question Answering. find
  2. Anonymous. 2022. Contrastive pre-training for zero-shot information retrieval. In Submitted to The Tenth International Conference on Learning Representations. Under review. find
  3. Zhuyun Dai and Jamie Callan. 2020. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’20, page 1533–1536, New York, NY, USA. Association for Computing Machinery. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In NAACL. find
  5. Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. find our card
  6. D. Gillick, A. Presta, and Gaurav Singh Tomar. 2018. End-to-end retrieval in continuous space. ArXiv, abs/1811.08008. — [arXiv:1811.08008](https://arxiv.org/abs/1811.08008) find
  7. Matthew Henderson, Rami Al-Rfou, B. Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and R. Kurzweil. 2017. Efficient natural language response suggestion for smart reply. ArXiv, abs/1705.00652. — [arXiv:1705.00652](https://arxiv.org/abs/1705.00652) find
  8. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. arXiv:2104.06967. arXiv preprint Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with… find our card
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769– 6781, Onl… find our card
  10. Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 39–48. find our card
  11. Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. Transactions of the Association for Computational Linguistic… find our card
  12. Jing Lu, Gustavo Hernández Ábrego, Ji Ma, Jianmo Ni, and Yinfei Yang. 2021. Multi-stage training with improved negative contrast for neural passage retrieval. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6091–6103. find
  13. Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2020. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181. find our card
  14. Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2020. Zero-shot neural retrieval via domain-targeted synthetic query generation. arXiv e-prints, pages arXiv–2004. find
  15. Ji Ma, Ivan Korotkov, Yinfei Yang, Keith Hall, and Ryan McDonald. 2021. Zero-shot neural passage retrieval via domain-targeted synthetic question generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 1075–1088,… find
  16. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. Ms marco: A human generated machine reading comprehension dataset. find our card
  17. Jianmo Ni, Gustavo Hernández Ábrego, Noah Constant, Ji Ma, Keith B Hall, Daniel Cer, and Yinfei Yang. 2021. Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models. arXiv preprint arXiv:2108.08877. find our card
  18. Rodrigo Nogueira. 2019. From doc2query to doctttttquery. find
  19. Chen Qu, Liu Yang, Cen Chen, Minghui Qiu, W. Bruce Croft, and Mohit Iyyer. 2020. Open-retrieval conversational question answering. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval. find
  20. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. Rocketqa: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Ass… find our card
  21. Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, W. Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR, 21/140. find
  22. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find our card
  23. Devendra Sachan, Mostofa Patwary, Mohammad Shoeybi, Neel Kant, Wei Ping, William L. Hamilton, and Bryan Catanzaro. 2021. End-to-end training of neural retrievers for open-domain question answering. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11t… find
  24. Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, pages 4596–4604. PMLR. find
  25. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2). find
  26. Dan Vanderkam, Rob Schonberger, Henry Rowley, and Sanjiv Kumar. 2013. Nearest neighbor search in google correlate. Technical report, Google. find
  27. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  28. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate nearest neighbor negative contrastive learning for dense text retrieval. arXiv preprint arXiv:2007.00808. find our card
  29. Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel. 2021. Byt5: Towards a token-free future with pre-trained byte-to-byte models. arXiv preprint arXiv:2105.13626. find
  30. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020. mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. find
  31. Yinfei Yang, Daniel Matthew Cer, Amin Ahmad, Mandy Guo, Jax Law, Noah Constant, G. Ábrego, Steve Yuan, C. Tar, Yun-Hsuan Sung, B. Strope, and R. Kurzweil. 2020. Multilingual universal sentence encoder for semantic retrieval. In ACL. find our card
  32. Model NDCG@10 MRR@10 Recall@1000 0.388 0.408 0.401 / 0.420 0.430 0.439 0.442 0.330 0.340 0.360 0.370 0.366 0.379 0.385 0.388 0.959 0.975 0.968 0.979 0.983 0.991 0.989 0.990 ANCE TAS-Balanced ColBERT RocketQA GTR-Base GTR-Large GTR-XL GTR-XXL Table 7: Comparisons of different models on MS Marco. Scal… find