Elamur Research⌕
☰
home / papers / 2023 / How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval

How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval

Paper · 2023 · Sheng-Chieh Lin, Akari Asai, Minghan Li et al.

Summary

Dense retrievers (DR) of BERT-base size have usually forced a choice: models tuned for in-domain supervised retrieval on MS MARCO tend to transfer poorly to new domains, and the reverse also holds. Some researchers attributed this to limited model capacity. The authors dispute that explanation. They treat recent DR training tricks, such as distillation, contrastive pre-training and pseudo-query generation, as variants of data augmentation (DA) applied to queries and relevance labels. Their pilot studies show two things. First, labels drawn from several diverse retrievers work better than labels from a single strong cross-encoder. Second, cheap cropped sentences can serve as training queries as well as costly generated ones. The resulting recipe, DRAGON, combines 28M augmented queries (cropped sentences plus doct5query outputs) with relevance labels added progressively from sparse, dense and multi-vector teachers. DRAGON is the first 110M-parameter DR to reach state-of-the-art results in both settings, matching far larger models and late-interaction systems. This suggests better training data, not more parameters, is the key lever for general-purpose retrievers.

Key points

  • Evaluation setup: supervised results are measured on MS MARCO Dev (6,980 queries; RR@10, R@1K) using the 8.8M-passage corpus. Zero-shot results are the average nDCG@10 over BEIR-13.
  • Query augmentation: about 28M sentences were cropped from the MS MARCO corpus and 28M synthetic queries were drawn from doct5query. The two types are mixed for training.
  • Progressive label augmentation: five teacher retrievers of mixed types (sparse, dense, multi-vector) are used. Feeding all teachers' labels at once, or merging their scores into soft labels, gave worse results.
  • Why cropping helps: cropped sentences cover more varied topics and cause more disagreement among teachers. This exposes the student to richer relevance signals, so DRAGON-S generalizes better than DRAGON-Q.
  • Cost and availability: training takes roughly 5 days on 32 A100 (40 GB) GPUs and requires already-trained teacher models. Code and checkpoints are released in facebookresearch/dpr-scale.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of WashingtonSeattle · United States263 works · 180,581 citations🎓University of WaterlooWaterloo · Canada44 works · 12,783 citations

📚 Shelves 6 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works Computer Vision and Pattern Recognition · 1,282 works Information Systems · 523 works

🧭 Topics 1 Information retrieval & search · 145 works

Cite

Cite as Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih and Xilun Chen. 2023. How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval. Findings of the Association for Computational Linguistics: EMNLP 2023.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2023.findings-emnlp.423

@inproceedings{lin2023how,
  title = {{How to Train Your Dragon: Diverse Augmentation Towards Generalizable Dense Retrieval}},
  author = {Sheng-Chieh Lin, Akari Asai, Minghan Li, Barlas Oguz, Jimmy Lin, Yashar Mehdad, Wen-tau Yih and Xilun Chen},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
  year = {2023},
  doi = {10.18653/v1/2023.findings-emnlp.423},
  url = {https://doi.org/10.18653/v1/2023.findings-emnlp.423}
}

References

50 references, 23 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Timo Schick, Patrick Lewis, Xilun Chen, Gautier Izacard, Sebastian Riedel, Hannaneh Hajishirzi, and Wen-tau Yih. 2023. Task-aware retrieval with instructions. In Proc. Findings of ACL, pages 3650–3675. our card
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. 2016. MS MARCO: A human generated machine reading comprehension dataset. arXiv:1611.09268. our card
  3. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009. Curriculum learning. In Proc. ICML, page 41–48. our card
  4. Wei-Cheng Chang, Felix X. Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. 2020. Pre-training tasks for embedding-based large-scale retrieval. In Proc. ICLR. find
  5. Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In Proc. ICML. our card
  6. Xilun Chen, Kushal Lakhotia, Barlas Oguz, Anchit Gupta, Patrick Lewis, Stan Peshterliev, Yashar Mehdad, Sonal Gupta, and Wen-tau Yih. 2022. Salient phrase aware dense retrieval: Can a dense retriever imitate a sparse one? In Proc. Findings of EMNLP, pages 250–262. find
  7. Zhuyun Dai, Vincent Y. Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang. 2022. Promptagator: Few-shot dense retrieval from 8 examples. arXiv:2209.11755. our card
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv:1810.04805. our card
  9. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2022. From distillation to hard negative sampling: Making sparse neural IR models more effective. In Proc. SIGIR, page 2353–2359. find
  10. Thibault Formal, Carlos Lassance, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE v2: Sparse lexical and expansion model for information retrieval. arXiv:2109.10086. find
  11. Luyu Gao and Jamie Callan. 2021. Condenser: a pre-training architecture for dense retrieval. In Proc. EMNLP, pages 981–993. our card
  12. Luyu Gao and Jamie Callan. 2022. Unsupervised corpus aware language model pre-training for dense passage retrieval. In Proc. ACL, pages 2843–2853. our card
  13. Ruiqi Guo, Philip Sun, Erik Lindgren, Quan Geng, David Simcha, Felix Chern, and Sanjiv Kumar. 2020. Accelerating large-scale inference with anisotropic vector quantization. In Proc. ICML. find
  14. Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. In Proc. NIPS. our card
  15. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently teaching an effective dense retriever with balanced topic aware sampling. In Proc. SIGIR, page 113–122. our card
  16. Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving efficient neural ranking models with cross-architecture knowledge distillation. arXiv:2010.02666. — [arXiv:2010.02666](https://arxiv.org/abs/2010.02666) find
  17. Gautier Izacard, Mathilde Caron, Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. 2021. Unsupervised dense information retrieval with contrastive learning. arXiv:2112.09118. our card
  18. Nick Craswell, Bhaskar Mitra, and Daniel Campos. 2019. Overview of the TREC 2019 deep learning track. In Proc. TREC. our card
  19. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2022. Few-shot Learning with Retrieval Augmented Language Models. arXiv:2208.03299. find
  20. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. 2020. Overview of the TREC 2020 deep learning track. In Proc. TREC. find
  21. Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2021. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, pages 535–547. find
  22. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proc. EMNLP, pages 6769–6781. our card
  23. Carlos Lassance and Stephane Clinchant. 2023. The tale of two MSMARCO - and their unfair comparisons. In Proc. SIGIR, page 2431–2435. find
  24. Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proc. ACL, pages 6086–6096. our card
  25. Barlas Oguz, Kushal Lakhotia, Anchit Gupta, Patrick Lewis, Vladimir Karpukhin, Aleksandra Piktus, Xilun Chen, Sebastian Riedel, Scott Yih, Sonal Gupta, and Yashar Mehdad. 2022. Domain-matched pre-training tasks for dense retrieval. In Proc. NAACL, pages 1524–1534. find
  26. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proc. NAACL, pages 5835–5847. our card
  27. Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, and Rodrigo Nogueira. 2021a. Pyserini: A python toolkit for reproducible information retrieval research with sparse and dense representations. In Proc. SIGIR, page 2356–2362. find
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res., 21(140):1–67. our card
  29. Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021b. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Proc. RepL4NLP, pages 163–173. find
  30. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proc. EMNLP, pages 3982–3992. our card
  31. Zhenghao Lin, Yeyun Gong, Xiao Liu, Hang Zhang, Chen Lin, Anlei Dong, Jian Jiao, Jingwen Lu, Daxin Jiang, Rangan Majumder, and Nan Duan. 2022. Prod: Progressive distillation for dense retrieval. arXiv:2209.13335. — [arXiv:2209.13335](https://arxiv.org/abs/2209.13335) find
  32. Shuqi Lu, Di He, Chenyan Xiong, Guolin Ke, Waleed Malik, Zhicheng Dou, Paul Bennett, Tie-Yan Liu, and Arnold Overwijk. 2021. Less is more: Pretrain a strong Siamese encoder for dense text retrieval using a weak decoder. In Proc. EMNLP, pages 2780–2791. find
  33. Xueguang Ma, Kai Sun, Ronak Pradeep, and Jimmy Lin. 2021. A replication study of dense passage retriever. arXiv:2104.05740. — [arXiv:2104.05740](https://arxiv.org/abs/2104.05740) find
  34. Joel Mackenzie, Andrew Trotman, and Jimmy Lin. 2021. Wacky weights in learned sparse representations and the revenge of score-at-a-time query evaluation. arXiv:2110.11540. — [arXiv:2110.11540](https://arxiv.org/abs/2110.11540) find
  35. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proc. ACL, pages 9802–9822. find
  36. Rui Meng, Ye Liu, Semih Yavuz, Divyansh Agarwal, Lifu Tu, Ning Yu, Jianguo Zhang, Meghana Bhat, and Yingbo Zhou. 2023. AugTriever: Unsupervised dense retrieval by scalable data augmentation. arXiv:2212.08841. — [arXiv:2212.08841](https://arxiv.org/abs/2212.08841) find
  37. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Proc. EMNLP, pages 9844–9855. our card
  38. Rodrigo Nogueira and Jimmy Lin. 2019. doc2query to docTTTTTquery. find
  39. From Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, QiaoQiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. 2021. RocketQAv2: A joint training method for dense passage retrieval and passage re-ranking. In Proc. EMNLP, pages 2825–2835. our card
  40. Keshav Santhanam, Omar Khattab, Christopher Potts, and Matei Zaharia. 2022a. Plaid: An efficient engine for late interaction retrieval. arXiv:2205.09707. find
  41. Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022b. Col-BERTv2: Effective and efficient retrieval via lightweight late interaction. In Proc. NAACL, pages 3715–3734. find
  42. Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen-tau Yih. 2023. Replug: Retrieval-augmented black-box language models. arXiv:2301.12652. our card
  43. Nandan Thakur, Nils Reimers, Johannes Daxenberger, and Iryna Gurevych. 2021a. Augmented SBERT: Data augmentation method for improving bi-encoders for pairwise sentence scoring tasks. In Proc. NAACL, pages 296–310. our card
  44. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021b. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Proc. NIPS. find
  45. Kexin Wang, Nandan Thakur, Nils Reimers, and Iryna Gurevych. 2022. GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. In Proc. NAACL, pages 2345–2360. our card
  46. Xing Wu, Guangyuan Ma, Meng Lin, Zijia Lin, Zhongyuan Wang, and Songlin Hu. 2022. Contextual masked auto-encoder for dense passage retrieval. arXiv:2208.07670. — [arXiv:2208.07670](https://arxiv.org/abs/2208.07670) find
  47. Shitao Xiao, Zheng Liu, Yingxia Shao, and Zhao Cao. 2022. RetroMAE: Pre-training retrieval-oriented language models via masked auto-encoder. In Proc. EMNLP, pages 538–548. our card
  48. Ji Xin, Chenyan Xiong, Ashwin Srinivasan, Ankita Sharma, Damien Jose, and Paul Bennett. 2022. Zero-shot dense retrieval with momentum adversarial domain invariant representations. In Proc. ACL, pages 4008–4020. find
  49. Yue Yu, Chenyan Xiong, Si Sun, Chao Zhang, and Arnold Overwijk. 2022. COCO-DR: Combating distribution shift in zero-shot dense retrieval with contrastive and distributionally robust learning. In Proc. EMNLP, pages 1462–1479. find
  50. Hansi Zeng, Hamed Zamani, and Vishwa Vinay. 2022. Curriculum learning for dense retrieval distillation. In Proc. SIGIR, page 1979–1983. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/lin-2023-how-train-your-dragon-diverse/