Elamur Research⌕
☰
home / papers / 2021 / SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

Paper · 2021 · Thibault Formal, Carlos Lassance, Benjamin Piwowarski et al.

Summary

First-stage retrieval needs to be both accurate and fast. Dense retrievers do well but lack explicit term matching. Learned sparse models keep the benefits of bag-of-words search, such as inverted indexes, exact matching and interpretability. This paper improves SPLADE, a sparse model that turns BERT MLM logits into term weights over the WordPiece vocabulary and keeps those weights sparse with a FLOPS regularizer.

The authors make three changes:

  • Max pooling. Token contributions are combined with max pooling instead of summation (SPLADE-max).
  • Document-only variant. SPLADE-doc expands and weights only documents, so all encoding happens offline.
  • Distillation. A cross-encoder teacher and hard negatives mined by SPLADE itself are used to distill the model (DistilSPLADE-max).

Training and evaluation use MS MARCO passages and TREC DL 2019, plus zero-shot tests on BEIR. Each change helps. The distilled model is close to the best dense retrievers in-domain and beats them on most BEIR datasets. This matters because it shows sparse, index-friendly retrieval can match or exceed dense methods, especially when moving to new domains.

Key points

  • Effect of max pooling: MRR@10 on MS MARCO dev rises from 0.322 to 0.340, and NDCG@10 on TREC DL 2019 from 0.665 to 0.684. SPLADE-max gives a better effectiveness/FLOPS trade-off at every regularization strength.
  • DistilSPLADE-max results: 0.368 MRR@10 and 0.979 R@1000 on MS MARCO dev, and 0.729 NDCG@10 on TREC DL 2019. The TREC result is roughly 9% above the original SPLADE. The model still reaches about 0.35 MRR@10 at around 0.3 FLOPS.
  • SPLADE-doc: It matches original SPLADE (0.322 MRR@10) without any query encoder and beats doc2query-T5. With only about 19 non-zero weights per document it still gets 29.6 MRR@10.
  • BEIR (NDCG@10, 14 available datasets): DistilSPLADE-max averages 0.500, compared with 0.455 for ColBERT, 0.440 for tuned BM25 and 0.435 for TAS-B. It is the top model on 11 of the 14 datasets.
  • Model details: The vocabulary has |V| = 30,522 WordPiece tokens. Term weights are log-saturated ReLU activations. Training uses a contrastive loss with in-batch negatives and separate FLOPS weights for queries and documents. Code is released at github.com/naver/splade.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬Centre National de la Recherche ScientifiqueParis · France168 works · 62,369 citations🎓Sorbonne UniversitéParis · France32 works · 7,270 citations🔬LIP6Paris · France5 works · 1,593 citations

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 worksInformation Systems · 1,437 works

🧭 Topics 1Research subjects and languages the work is about.Information retrieval & search · 473 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Inverted index · 26 works

Cite

Cite as Thibault Formal, Carlos Lassance, Benjamin Piwowarski and Stéphane Clinchant. 2021. SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval. arXiv:2109.10086.

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 2109.10086

@misc{formal2021splade,
  title = {{SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval}},
  author = {Thibault Formal, Carlos Lassance, Benjamin Piwowarski and Stéphane Clinchant},
  year = {2021},
  eprint = {2109.10086},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2109.10086}
}

References

33 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval. arXiv:2010.00768 [cs.IR] — [arXiv:2010.00768](https://arxiv.org/abs/2010.00768) find
  2. Leonid Boytsov. 2018. Efficient and Accurate Non-Metric k-NN Search with Applications to Text Matching. Ph.D. Dissertation. Carnegie Mellon University. find
  3. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020).
  4. Zhuyun Dai and Jamie Callan. 2019. Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval. arXiv:1910.10687 [cs.IR] — [arXiv:1910.10687](https://arxiv.org/abs/1910.10687) find
  5. Zhuyun Dai and Jamie Callan. 2020. Context-Aware Document Term Weighting for Ad-Hoc Search. Association for Computing Machinery, New York, NY, USA, 1897–1907. https://doi.org/10.1145/3366423.3380258 — [doi:10.1145/3366423.3380258](https://doi.org/10.1145/3366423.3380258) ↗
  6. Zhuyun Dai and Jamie Callan. 2020. Context-Aware Term Weighting For First Stage Passage Retrieval. Association for Computing Machinery, New York, NY, USA, 1533–1536. https://doi.org/10.1145/3397271.3401204 — [doi:10.1145/3397271.3401204](https://doi.org/10.1145/3397271.3401204) ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805 ↗
  8. Thibault Formal, Benjamin Piwowarski, and Stéphane Clinchant. 2021. SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR ’21). Association…
  9. Luyu Gao, Zhuyun Dai, and Jamie Callan. 2021. COIL: Revisit Exact Lexical Match in Information Retrieval with Contextualized Inverted List. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association f…
  10. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909 [cs.CL]
  11. Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. 2021. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling. In Proc. of SIGIR.
  12. Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 [cs.IR] — [arXiv:2010.02666](https://arxiv.org/abs/2010.02666) find
  13. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906 [cs.CL]
  14. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for…
  15. Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC. 1–6. find
  16. Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2021. In-Batch Negatives for Knowledge Distillation with Tightly-Coupled Teachers for Dense Retrieval. In Proceedings of the 6th Workshop on Representation Learning for NLP (RepL4NLP-2021). Association for Computational Linguistics, Online, 163–173. h… find
  17. Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via Prediction of Importance with Contextualization. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Jul 2020)… — [doi:10.1145/3397271.3401262](https://doi.org/10.1145/3397271.3401262) find
  18. Antonio Mallia, Omar Khattab, Torsten Suel, and Nicola Tonellotto. 2021. Learning Passage Impacts for Inverted Indexes. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, Canada) (SIGIR ’21). Association for Computing Ma… — [doi:10.1145/3404835.3463030](https://doi.org/10.1145/3404835.3463030) find
  19. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv:1901.04085 [cs.IR]
  20. Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery. find
  21. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. arXiv:1904.08375 [cs.IR]
  22. Biswajit Paria, Chih-Kuan Yeh, Ian E. H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020. Minimizing FLOPs to Learn Efficient Sparse Representations. arXiv:2004.05665 [cs.LG] — [arXiv:2004.05665](https://arxiv.org/abs/2004.05665) find
  23. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library.. In NeurIPS.
  24. Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2021. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2021 Conference of the North American Chapter of the Ass…
  25. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. http://arxiv.org/abs/1908.10084 ↗
  26. Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, and Iryna Gurevych. 2021. BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models. CoRR abs/2104.08663 (2021). arXiv:2104.08663 https://arxiv.org/abs/2104.08663 ↗
  27. Zhengkai Tu, Wei Yang, Zihang Fu, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2020. Approximate Nearest Neighbor Search and Lightweight Dense Vector Reranking in Multi-Stage Retrieval Architectures. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Informat… find
  28. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… — Transformers: State-of-the-Art Natural Language Processing
  29. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. arXiv:2007.00808 [cs.IR]
  30. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwikj. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In International Conference on Learning Representations. https://openreview.net/forum?id=zeFrfgy… ↗
  31. Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the “Neural Hype”. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Jul 2019). https://doi.org/10.1145/3331184.3331340 — [doi:10.1145/3331184.3331340](https://doi.org/10.1145/3331184.3331340) ↗
  32. Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps. 2018. From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy)… — [doi:10.1145/3269206.3271800](https://doi.org/10.1145/3269206.3271800) find
  33. Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2020. SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval. arXiv:2009.13013 [cs.CL] find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/formal-2021-splade-v2-sparse-lexical/