Elamur Research
☰
home / papers / 2019 / Passage Re-ranking with BERT

Passage Re-ranking with BERT

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1901.04085

Summary

Nogueira and Cho look at the middle step of a typical retrieval-based question-answering pipeline. In that step, a cheap first-stage retriever (BM25) returns about a thousand candidate passages, and a more expensive model must reorder them so the truly relevant ones come first. Neural rankers had not yet clearly beaten classical IR methods, partly because large training sets were scarce. The authors argue that MS MARCO and BERT together change this. They fine-tune a pretrained BERT model as a binary relevance classifier: the query and the candidate passage go in as one sentence pair, and the [CLS] representation is used to predict a relevance probability. Passages are then sorted by that probability. Evaluation covers MS MARCO and TREC-CAR. For TREC-CAR, the authors pre-train BERT again on only the training half of Wikipedia so that test documents do not leak into training. The model took first place on the MS MARCO leaderboard and set a new best result on TREC-CAR, with large margins on both. It needed only a small share of the available training data. The work matters because it turned the BERT cross-encoder into a standard re-ranking baseline.

Key points

  • Setup: BERT Large is fine-tuned with cross-entropy loss. Positives are the annotated relevant passages and negatives come from the BM25 top-1,000. Queries are capped at 64 tokens and the full input at 512 tokens. Each passage is scored on its own.
  • MS MARCO (MRR@10): BERT Large reaches 36.5 on dev and 35.8 on eval. The previous leaderboard leader, IRNet, scored 27.8 and 28.1, so the gain is about 27% relative. BERT Base gets 34.7 on dev.
  • TREC-CAR (MAP, 2017 test queries, automatic annotations): BERT Large scores 33.5 and BERT Base 31.0. Earlier systems scored about 15 or lower, including Co-PACRR (14.8) and tuned Anserini BM25 (15.3).
  • Data efficiency: On MS MARCO the model trained for about 30 hours on a TPU v3-8 and saw 12.8M query–passage pairs, a small fraction of the training set. Training longer brought no gain. After only 100k pairs it already beat IRNet by 1.4 MRR@10 points.
  • Resource: The code is released at github.com/nyu-dl/dl4marco-bert. BM25 retrieval uses the Anserini toolkit.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

357 papers of our library cite it (of the 140673 whose bibliographies we read); 20 of them have a page here, 337 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.1901.04085 · arXiv 1901.04085

@misc{nogueira2019passage,
  title = {{Passage Re-ranking with BERT}},
  author = {Nogueira, Rodrigo and Cho, Kyunghyun},
  year = {2019},
  doi = {10.48550/arxiv.1901.04085},
  eprint = {1901.04085},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.1901.04085}
}

References

22 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading wikipedia to answer open-domain questions. arXiv preprint arXiv:1704.00051, 2017. find our card
  2. Christopher Clark and Matt Gardner. Simple and effective multi-paragraph reading comprehension. arXiv preprint arXiv:1710.10723, 2017. find our card
  3. Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. Convolutional neural networks for soft-matching n-grams in ad-hoc search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 126–134. ACM, 2018. find
  4. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  5. Bhuwan Dhingra, Kathryn Mazaitis, and William W Cohen. Quasar: Datasets for question answering by search and reading. arXiv preprint arXiv:1707.03904, 2017. — [arXiv:1707.03904](https://arxiv.org/abs/1707.03904) find
  6. Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. Trec complex answer retrieval overview. TREC, 2017. find
  7. Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179, 2017. find our card
  8. Jiafeng Guo, Yixing Fan, Qingyao Ai, and W Bruce Croft. A deep relevance matching model for ad-hoc retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pp. 55–64. ACM, 2016. find
  9. Kai Hui, Andrew Yates, Klaus Berberich, and Gerard de Melo. Co-pacrr: A context-aware neural ir model for ad-hoc retrieval. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, pp. 279–287. ACM, 2018. find
  10. Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv preprint arXiv:1705.03551, 2017. find our card
  11. Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014. find
  12. Jimmy Lin. The neural hype and comparisons against weak baseline. 2019. find
  13. Sean MacAvaney, Andrew Yates, and Kai Hui. Contextualized pacrr for complex answer retrieval. 2017. find
  14. Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In Proceedings of the 26th International Conference on World Wide Web, pp. 1291–1299. International World Wide Web Conferences Steering Committee, 2017. find
  15. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. find our card
  16. Matthew E Peters, Waleed Ammar, Chandra Bhagavatula, and Russell Power. Semi-supervised sequence tagging with bidirectional language models. arXiv preprint arXiv:1705.00108, 2017. find our card
  17. Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding by generative pre-training. URL https://s3-us-west-2. amazonaws. com/openai-assets/research-covers/language-unsupervised/language understanding paper. pdf, 2018. find
  18. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv:1606.05250, 2016. find our card
  19. Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. Bidirectional attention flow for machine comprehension. arXiv preprint arXiv:1611.01603, 2016. find our card
  20. Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. End-to-end neural ad-hoc ranking with kernel pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 55–64. ACM, 2017. find
  21. Peilin Yang, Hui Fang, and Jimmy Lin. Anserini: Reproducible ranking baselines using lucene. Journal of Data and Information Quality (JDIQ), 10(4):16, 2018. find
  22. Adams Wei Yu, David Dohan, Minh-Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V Le. Qanet: Combining local convolution with global self-attention for reading comprehension. arXiv preprint arXiv:1804.09541, 2018. find our card