Elamur Research
☰
home / papers / 2020 / Overview of the TREC 2019 deep learning track

Overview of the TREC 2019 deep learning track

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2003.07820

Summary

Neural ranking for ad hoc search has mostly been studied on small, proprietary or synthetic data. This left open whether deep models truly beat strong classical baselines. The TREC 2019 Deep Learning Track addresses this by releasing large human-labeled training sets derived from MS MARCO and evaluating systems blindly, with judgments created only after submissions closed. It has two tasks:

  • Document retrieval: about 3.2M documents.
  • Passage retrieval: about 8.8M passages.

Each task can be entered as full end-to-end retrieval or as reranking of a fixed candidate list. NIST assessors graded results on a four-level relevance scale, using pooling plus classifier-guided (HiCAL) selection, with the aim of producing reusable test collections. Fifteen groups submitted 75 runs. Neural runs clearly outperformed traditional IR runs, and the strongest systems relied on transfer learning from pretrained language models such as BERT. Reranking and full retrieval performed similarly. The work matters because it provides an open, large-scale benchmark for ad hoc ranking that can be reused in later studies.

Key points

  • Training data: the document task has 3,213,835 documents and 367,013 training queries. The passage task uses the MS MARCO passage corpus, with 8,841,823 passages and 502,940 training queries.
  • Label construction: training labels are sparse, usually a single positive per query with no negatives. Document labels were obtained by transferring passage-level labels to the source document.
  • Test queries: 200 queries were released to participants, and NIST judged 43 of them per task.
  • Reranking candidates: documents came from an Indri top-100 list (Krovetz stemming), and passages from a BM25 top-1000 list.
  • Participation: 10 groups submitted 38 document runs and 11 groups submitted 37 passage runs. Runs were categorized as nnlm, nn and trad.
  • Reusability: simulations varying the stopping criteria kept system rankings stable, with Kendall's τ of roughly 0.90–1.00 against the official qrels.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

156 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 149 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2003.07820 · arXiv 2003.07820

@misc{craswell2020overview,
  title = {{Overview of the TREC 2019 deep learning track}},
  author = {Craswell, Nick and Mitra, Bhaskar and Yilmaz, Emine and Campos, Daniel and Voorhees, Ellen M.},
  year = {2020},
  doi = {10.48550/arxiv.2003.07820},
  eprint = {2003.07820},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2003.07820}
}

References

32 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nasreen Abdul-Jaleel, James Allan, W Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Mark D Smucker, and Courtney Wade. Umass at trec 2004: Novelty and hard. 2004. find
  2. Mustafa Abualsaud, Nimesh Ghelani, Haotian Zhang, Mark D Smucker, Gordon V Cormack, and Maura R Grossman. A system for efficient high-recall retrieval. In The 41st international ACM SIGIR conference on research & development in information retrieval, pages 1317–1320, 2018. find
  3. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, et al. Ms marco: A human generated machine reading comprehension dataset. arXiv preprint arXiv:1611.09268, 2016. find our card
  4. Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253–279, 2013. find
  5. Leonid Boytsov, David Novak, Yury Malkov, and Eric Nyberg. Off the beaten path: Let’s replace term-based retrieval with k-nn search. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 1099–1108. ACM, 2016. find
  6. Chris Buckley, Darrin Dimmick, Ian Soboroff, and Ellen Voorhees. Bias and the limits of pooling for large collections. Information retrieval, 10(6):491–508, 2007. find
  7. Gordon V Cormack and Maura R Grossman. Beyond pooling. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pages 1169–1172, 2018. find
  8. Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W Bruce Croft. Neural ranking models with weak supervision. In Proc. SIGIR, pages 65–74. ACM, 2017. find
  9. Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proc. CVPR, pages 248–255. Ieee, 2009. find
  10. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018. find
  11. Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. Trec complex answer retrieval overview. In Proceedings of TREC, 2017. find
  12. Armand Joulin, Edouard Grave, Piotr Bojanowski, and Tomas Mikolov. Bag of tricks for efficient text classification. arXiv preprint arXiv:1607.01759, 2016. find
  13. Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature, 521(7553):436–444, 2015. find
  14. Tie-Yan Liu. Learning to rank for information retrieval. Foundation and Trends in Information Retrieval, 3(3):225– 331, March 2009. find
  15. Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9 (Nov):2579–2605, 2008. find
  16. Irina Matveeva, Chris Burges, Timo Burkard, Andy Laucius, and Leon Wong. High accuracy retrieval with multiple nested ranker. In Proceedings of the 29th annual international ACM SIGIR conference on Research and development in information retrieval, pages 437–444. ACM, 2006. find
  17. Bhaskar Mitra and Nick Craswell. An introduction to neural information retrieval. Foundations and Trends R in Information Retrieval (to appear), 2018. find
  18. Bhaskar Mitra and Nick Craswell. An updated duet model for passage re-ranking. arXiv preprint arXiv:1903.07666, 2019. — [arXiv:1903.07666](https://arxiv.org/abs/1903.07666) find
  19. Bhaskar Mitra, Fernando Diaz, and Nick Craswell. Learning to match using local and distributed representations of text for web search. In Proc. WWW, pages 1291–1299, 2017. find
  20. Bhaskar Mitra, Corby Rosset, David Hawking, Nick Craswell, Fernando Diaz, and Emine Yilmaz. Incorporating query term independence assumption for efficient retrieval and ranking using deep neural networks. arXiv preprint arXiv:1907.03693, 2019. — [arXiv:1907.03693](https://arxiv.org/abs/1907.03693) find
  21. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. Document expansion by query prediction. arXiv preprint arXiv:1904.08375, 2019. find our card
  22. Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends R in Information Retrieval, 3(4):333–389, 2009. find our card
  23. Corby Rosset, Damien Jose, Gargi Ghosh, Bhaskar Mitra, and Saurabh Tiwary. Optimizing query evaluations using reinforcement learning for web search. In Proc. SIGIR, pages 1193–1196. ACM, 2018. find
  24. Trevor Strohman, Donald Metzler, Howard Turtle, and W Bruce Croft. Indri: A language model-based search engine for complex queries. In Proceedings of the International Conference on Intelligent Analysis, volume 2, pages 2–6. Citeseer, 2005. find
  25. Ellen M Voorhees. On building fair and reusable test collections using bandit techniques. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management, pages 407–416, 2018. find
  26. Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 105–114. ACM, 2011. find
  27. Peilin Yang and Jimmy Lin. Reproducing and generalizing semantic term matching in axiomatic information retrieval. In European Conference on Information Retrieval, pages 369–381. Springer, 2019. find
  28. Wei Yang, Kuang Lu, Peilin Yang, and Jimmy Lin. Critically examining the “neural hype”: Weak baselines and the additivity of effectiveness gains from neural ranking models. In Proc. SIGIR, pages 1129–1132. ACM, 2019a. find
  29. Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237, 2019b. find our card
  30. Hamed Zamani, Mostafa Dehghani, W Bruce Croft, Erik Learned-Miller, and Jaap Kamps. From neural re-ranking to neural ranking: Learning a sparse representation for inverted indexing. In Proc. CIKM, pages 497–506. ACM, 2018. find
  31. Zhaohao Zeng and Tetsuya Sakai. Bm25 pseudo relevance feedback using anserini at waseda university. In Proceedings The Open-Source IR Replicability Challenge (OSIRRC) Workshop, 2019. find
  32. Justin Zobel. How reliable are the results of large-scale information retrieval experiments? In Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pages 307–314, 1998. find