Elamur Research
☰
home / papers / 2021 / SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "CC BY-NC-SA 4.0 (arXiv)" does not allow redistributing the text itself. Original — arxiv.org/pdf/2107.05720

Summary

First-stage retrieval in neural IR usually relies on either BM25-style bag-of-words matching, which suffers from vocabulary mismatch, or dense embeddings, which require approximate nearest-neighbor search and do not model exact term matching. The authors propose SPLADE, a sparse retriever that builds on SparTerm. SPLADE uses BERT's MLM head to predict a weight for every term in the WordPiece vocabulary, so a document or query can gain expansion terms as well as reweighted original terms. Three changes make the approach work. A logarithmic saturation of term weights stops a few terms from dominating and makes representations sparser. Training uses in-batch negatives. A FLOPS regularizer controls sparsity so that posting lists stay balanced. Unlike SparTerm, the model is trained end-to-end in one stage. On MS MARCO passage retrieval and TREC DL 2019, SPLADE clearly beats other sparse methods and comes close to strong dense models like ANCE. Its compute cost stays near that of BOW approaches, which makes it a simple option for serving from an inverted index.

Key points

  • Scoring and vocabulary: Term importance is computed over the BERT WordPiece vocabulary (|V| = 30522) as log(1 + ReLU(·)), summed over input tokens. Relevance is scored by a dot product between the sparse query and document vectors.
  • MS MARCO dev results: SPLADE-ℓFLOPS reaches MRR@10 0.322 and R@1000 0.955 at 0.73 FLOPS. For comparison:
    • BM25: 0.184 MRR@10
    • doc2query-T5: 0.277 MRR@10 at 0.81 FLOPS
    • ANCE: 0.330 MRR@10
    • TCT-ColBERT: 0.335 MRR@10
  • TREC DL 2019 results: SPLADE-ℓFLOPS obtains NDCG@10 0.665 and R@1000 0.813. Retuned SparTerm baselines trained by the authors already outperform the numbers from the original SparTerm paper.
  • Efficiency trade-off: The regularization strength controls how sparse the model is, and the FLOPS regularizer consistently does better than ℓ1 at the same efficiency level. A heavily regularized model (0.05 FLOPS) still scores MRR@10 0.296. It keeps about 18 non-zero terms per document and 6 per query, and its index takes under 1.4 GB.
  • Expansion behavior: On average, the model drops about 20 uninformative terms per document and adds about 32 expansion terms, such as stem variants (legs → leg) and topical words. The code is available at github.com/naver/splade.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

115 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 111 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1145/3404835.3463098 · arXiv 2107.05720

@inproceedings{formal2021splade,
  title = {{SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking}},
  author = {Formal, Thibault and Piwowarski, Benjamin and Clinchant, Stéphane},
  booktitle = {SIGIR 2021},
  year = {2021},
  doi = {10.1145/3404835.3463098},
  eprint = {2107.05720},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.1145/3404835.3463098}
}

References

29 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yang Bai, Xiaoguang Li, Gang Wang, Chaoliang Zhang, Lifeng Shang, Jun Xu, Zhaowei Wang, Fangshan Wang, and Qun Liu. 2020. SparTerm: Learning Term-based Sparse Representation for Fast Text Retrieval. arXiv:2010.00768 [cs.IR] — [arXiv:2010.00768](https://arxiv.org/abs/2010.00768) find
  2. Leonid Boytsov. 2018. Efficient and Accurate Non-Metric k-NN Search with Applications to Text Matching. Ph.D. Dissertation. Carnegie Mellon University. find
  3. Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Ellen M Voorhees. 2020. Overview of the trec 2019 deep learning track. arXiv preprint arXiv:2003.07820 (2020). find our card
  4. Zhuyun Dai and Jamie Callan. 2019. Context-Aware Sentence/Passage Term Importance Estimation For First Stage Retrieval. arXiv:1910.10687 [cs.IR] — [arXiv:1910.10687](https://arxiv.org/abs/1910.10687) find
  5. Zhuyun Dai and Jamie Callan. 2020. Context-Aware Document Term Weighting for Ad-Hoc Search. Association for Computing Machinery, New York, NY, USA, 1897–1907. https://doi.org/10.1145/3366423.3380258 — [doi:10.1145/3366423.3380258](https://doi.org/10.1145/3366423.3380258) ↗
  6. Zhuyun Dai and Jamie Callan. 2020. Context-Aware Term Weighting For First Stage Passage Retrieval. Association for Computing Machinery, New York, NY, USA, 1533–1536. https://doi.org/10.1145/3397271.3401204 — [doi:10.1145/3397271.3401204](https://doi.org/10.1145/3397271.3401204) ↗
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR abs/1810.04805 (2018). arXiv:1810.04805 http://arxiv.org/abs/1810.04805 ↗
  8. Yingqi Qu Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. 2020. RocketQA: An Optimized Training Approach to Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2010.08191 [cs.CL] find our card
  9. Hui Fang, Tao Tao, and ChengXiang Zhai. 2004. A Formal Study of Information Retrieval Heuristics. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (Sheffield, United Kingdom) (SIGIR ’04). Association for Computing Machinery, Ne… — [doi:10.1145/1008992.1009004](https://doi.org/10.1145/1008992.1009004) find
  10. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. REALM: Retrieval-Augmented Language Model Pre-Training. arXiv:2002.08909 [cs.CL] find
  11. Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. 2020. Improving Efficient Neural Ranking Models with Cross-Architecture Knowledge Distillation. arXiv:2010.02666 [cs.IR] — [arXiv:2010.02666](https://arxiv.org/abs/2010.02666) find
  12. Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. 2020. Dense Passage Retrieval for Open-Domain Question Answering. arXiv:2004.04906 [cs.CL] find our card
  13. Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR ’20). Association for… find our card
  14. Siu Kwan Lam, Antoine Pitrou, and Stanley Seibert. 2015. Numba: A llvm-based python jit compiler. In Proceedings of the Second Workshop on the LLVM Compiler Infrastructure in HPC. 1–6. find
  15. Sheng-Chieh Lin, Jheng-Hong Yang, and Jimmy Lin. 2020. Distilling Dense Representations for Ranking using Tightly-Coupled Teachers. arXiv:2010.11386 [cs.IR] — [arXiv:2010.11386](https://arxiv.org/abs/2010.11386) find
  16. Sean MacAvaney, Franco Maria Nardini, Raffaele Perego, Nicola Tonellotto, Nazli Goharian, and Ophir Frieder. 2020. Expansion via Prediction of Importance with Contextualization. Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Jul 2020)… — [doi:10.1145/3397271.3401262](https://doi.org/10.1145/3397271.3401262) find
  17. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv:1901.04085 [cs.IR] find our card
  18. Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery. find
  19. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. arXiv:1904.08375 [cs.IR] find our card
  20. Biswajit Paria, Chih-Kuan Yeh, Ian E. H. Yen, Ning Xu, Pradeep Ravikumar, and Barnabás Póczos. 2020. Minimizing FLOPs to Learn Efficient Sparse Representations. arXiv:2004.05665 [cs.LG] — [arXiv:2004.05665](https://arxiv.org/abs/2004.05665) find
  21. Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library.. In NeurIPS. find our card
  22. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics. http://arxiv.org/abs/1908.10084 ↗
  23. Zhengkai Tu, Wei Yang, Zihang Fu, Yuqing Xie, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2020. Approximate Nearest Neighbor Search and Lightweight Dense Vector Reranking in Multi-Stage Retrieval Architectures. In Proceedings of the 2020 ACM SIGIR on International Conference on Theory of Informat… find
  24. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoe… find our card
  25. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. arXiv:2007.00808 [cs.IR] find our card
  26. Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwikj. 2021. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. In International Conference on Learning Representations. https://openreview.net/forum?id=zeFrfgy… ↗ our card
  27. Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. 2019. Critically Examining the “Neural Hype”. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (Jul 2019). https://doi.org/10.1145/3331184.3331340 — [doi:10.1145/3331184.3331340](https://doi.org/10.1145/3331184.3331340) ↗
  28. Hamed Zamani, Mostafa Dehghani, W. Bruce Croft, Erik Learned-Miller, and Jaap Kamps. 2018. From Neural Re-Ranking to Neural Ranking: Learning a Sparse Representation for Inverted Indexing. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (Torino, Italy)… — [doi:10.1145/3269206.3271800](https://doi.org/10.1145/3269206.3271800) find
  29. Tiancheng Zhao, Xiaopeng Lu, and Kyusong Lee. 2020. SPARTA: Efficient Open-Domain Question Answering via Sparse Transformer Matching Retrieval. arXiv:2009.13013 [cs.CL] find