Elamur Research
☰
home / papers / 2019 / Document Expansion by Query Prediction

Document Expansion by Query Prediction

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive licence to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/1904.08375

Summary

Search engines often miss relevant texts because users phrase queries with different words than the documents use. Classic fixes rewrite the query. This paper instead enriches the documents before indexing. The method, Doc2query, trains a sequence-to-sequence transformer on (query, relevant passage) pairs from MS MARCO and TREC-CAR, so that it learns to generate questions a passage could answer. For each passage it samples ten such queries and appends them to the text. The expanded corpus is then indexed and searched with ordinary BM25 in Anserini, with optional BERT re-ranking.

On both benchmarks, expansion alone improved BM25 by roughly 15% while adding only modest latency. Combined with BERT, it gave the best published TREC-CAR result and came close to the top of the MS MARCO leaderboard. RM3 query expansion, by contrast, lowered scores. The work matters because it moves costly neural inference from query time to indexing time. It is also simple and reproducible with open-source tools, and the code and models are released.

Key points

  • Setup: The model is a transformer built with OpenNMT. Input passages are cut to 400 tokens and target queries to 100 tokens, with Moses tokenization and BPE. It generates 10 queries per passage using top-k random sampling and appends them without any special marker.
  • MS MARCO results (MRR@10):
    • BM25: 18.4 dev / 18.6 test.
    • BM25 + Doc2query: 21.5 dev / 21.8 test.
    • With BERT re-ranking: 37.5 dev / 36.8 test.
  • TREC-CAR results (MAP):
    • BM25: 15.3, rising to 18.3 with Doc2query.
    • BERT re-ranking: 34.8, rising to 36.5 with Doc2query.
  • Speed and recall:
    • Expansion raises latency from about 50 to 90 ms per query.
    • That is roughly seven times faster than the Duet v2 re-ranker, which scores about 3 MRR points higher.
    • Recall@1000 on MS MARCO dev rose from 85.3 to 89.3.
  • Analysis of generated words (stop words excluded):
    • About 69% of predicted terms already appear in the passage, which acts as term re-weighting; 31% are new.
    • Expanding with only new words gives an MRR@10 of 18.8, with only copied words 19.7, and with both 21.5, so the two types complement each other.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Computing & GPUs

Cited by

125 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 115 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. arXiv 1904.08375

@misc{nogueira2019document,
  title = {{Document Expansion by Query Prediction}},
  author = {Nogueira, Rodrigo and Yang, Wei and Lin, Jimmy and Cho, Kyunghyun},
  year = {2019},
  eprint = {1904.08375},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/1904.08375}
}

References

28 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nasreen Abdul-Jaleel, James Allan, W. Bruce Croft, Fernando Diaz, Leah Larkey, Xiaoyan Li, Donald Metzler, Mark D. Smucker, Trevor Strohman, Howard Turtle, and Courtney Wade. 2004. UMass at TREC 2004: Novelty and HARD. In Proceedings of the Thirteenth Text REtrieval Conference (TREC 2004). Gaithersb… find
  2. Payal Bajaj, Daniel Campos, Nick Craswell, Li Deng, Jianfeng Gao, Xiaodong Liu, Rangan Majumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. arXiv:1611.09268… find our card
  3. Adam Berger and John Lafferty. 1999. Information Retrieval as Statistical Translation. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1999). 222–229. find
  4. Bodo Billerbeck and Justin Zobel. 2005. Document Expansion versus Query Expansion for Ad-hoc Retrieval. In Proceedings of the 10th Australasian Document Computing Symposium. 34–41. find
  5. Scott Deerwester, Susan T. Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990. Indexing by Latent Semantic Analysis. Journal of the Association for Information Science 41, 6 (1990), 391–407. find
  6. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. arXiv:1810.04805 (2018). find
  7. Laura Dietz, Manisha Verma, Filip Radlinski, and Nick Craswell. 2017. TREC Complex Answer Retrieval Overview. In Proceedings of the Twenty-Sixth Text REtrieval Conference (TREC 2017). find
  8. Miles Efron, Peter Organisciak, and Katrina Fenlon. 2012. Improving Retrieval of Short Texts Through Document Expansion. In Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval (SIGIR 2012). 911–920. find
  9. Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. arXiv:1805.04833 (2018). find
  10. Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. arXiv:1412.6980 (2014). find
  11. Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush. 2017. Open-NMT: Open-Source Toolkit for Neural Machine Translation. In Proc. ACL. https://doi.org/10.18653/v1/P17-4012 ↗ our card
  12. Sean MacAvaney, Andrew Yates, and Kai Hui. 2017. Contextualized PACRR for complex answer retrieval. In Proceedings of TREC. find
  13. Bhaskar Mitra and Nick Craswell. 2019a. An Introduction to Neural Information Retrieval. Foundations and Trends in Information Retrieval 13, 1 (2019), 1–126. find
  14. Bhaskar Mitra and Nick Craswell. 2019b. An Updated Duet Model for Passage Re-ranking. arXiv:1903.07666 (2019). — [arXiv:1903.07666](https://arxiv.org/abs/1903.07666) find
  15. Rodrigo Nogueira and Kyunghyun Cho. 2019. Passage Re-ranking with BERT. arXiv:1901.04085 (2019). find our card
  16. Kezban Dilek Onal, Ye Zhang, Ismail Sengor Al-tingovde, Md Mustafizur Rahman, Pinar Karagoz, Alex Braylan, Brandon Dang, Heng-Lu Chang, Henna Kim, Quinten Mcnamara, Aaron Angert, Edward Banner, Vivek Khetan, Tyler Mcdonnell, An Thanh Nguyen, Dan Xu, Byron C. Wallace, Maarten Rijke, and Matthew Lease… find
  17. Jeremy Pickens, Matthew Cooper, and Gene Golovchinsky. 2010. Reverted Indexing for Feedback and Expansion. In Proceedings of the 19th ACM International Conference on Information and Knowledge Management (CIKM 2010). 1049–1058. find
  18. Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. 1994. Okapi at TREC-3. In Proceedings of the 3rd Text REtrieval Conference (TREC-3). Gaithersburg, Maryland, 109–126. find our card
  19. Joseph John Rocchio. 1971. Relevance Feedback in Information Retrieval. In The SMART Retrieval System—Experiments in Automatic Document Processing, Gerard Salton (Ed.). Prentice-Hall, Englewood Cliffs, New Jersey, 313–323. find
  20. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2015. Neural Machine Translation of Rare Words with Subword Units. arXiv:1508.07909 (2015). find
  21. Amit Singhal and Fernando Pereira. 1999. Document Expansion for Speech Retrieval. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1999). 34–41. find
  22. Tao Tao, Xuanhui Wang, Qiaozhu Mei, and ChengXiang Zhai. 2006. Language Model Information Retrieval with Document Expansion. In Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. 407– 414. find
  23. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention Is All You Need. In Advances in Neural Information Processing Systems. 5998–6008. find
  24. Ellen M. Voorhees. 1994. Query Expansion Using Lexical-Semantic Relations. In Proceedings of the 17th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 1994). 61–69. find
  25. Jinxi Xu and W. Bruce Croft. 2000. Improving the Effectiveness of Information Retrieval with Local Context Analysis. ACM Transactions on Information Systems 18, 1 (2000), 79–112. find
  26. Peilin Yang, Hui Fang, and Jimmy Lin. 2017. Anserini: Enabling the Use of Lucene for Information Retrieval Research. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2017). 1253–1256. find
  27. Peilin Yang, Hui Fang, and Jimmy Lin. 2018. Anserini: Reproducible Ranking Baselines Using Lucene. Journal of Data and Information Quality 10, 4 (2018), Article 16. 21.5 find
  28. MRR@10 Beam Search Top-k Random Sampling 20.5 \# of queries produced (beam size) Figure 2: Retrieval effectiveness on the development set of MS MARCO when using different decoding methods to produce queries. On the x-axis, we vary the number of predicted queries that are appended to the original docu… find