Elamur Research⌕
☰
home / papers / 2019 / From doc2query to docTTTTTquery

From doc2query to docTTTTTquery

Paper · 2019 · Rodrigo Nogueira and Jimmy Lin

Summary

This technical report revisits doc2query, a document expansion technique that predicts questions a passage could answer and appends them to the passage before indexing. The authors keep the doc2query pipeline but replace its original sequence-to-sequence transformer, which was trained from scratch, with a pretrained T5-base model. They fine-tune it on MS MARCO passage–query pairs and generate queries with top-k sampling (k = 10). The expanded passages are indexed with Anserini and searched with plain BM25, so all neural computation happens offline. On the MS MARCO passage ranking task, docTTTTTquery clearly outperforms both BM25 and the original doc2query, as well as the BERT-based expansion method DeepCT. Query latency stays close to plain BM25, while a BERT Large reranker needs seconds per query. The generated questions also match reference queries much more closely. The authors credit this improvement to pretraining rather than model size. The result shows that a cheap first-stage retriever can be made substantially stronger without query-time neural inference.

Key points

  • Training and generation setup: T5-base was fine-tuned for 4k iterations with batch size 256, about 2 epochs, at a learning rate of 10⁻⁴. Inputs were capped at 512 tokens and outputs at 64. T5-large brought no retrieval gain, and T5-3B and T5-11B were not tested.
  • Retrieval results: MRR@10 on the dev set rose from 0.184 with BM25 and 0.218 with doc2query to between 0.259 and 0.278 with docTTTTTquery, depending on the number of sampled queries (5 to 80). With 40 samples, the test MRR@10 was 0.272, and Recall@1000 on dev reached up to 0.947 compared with 0.853 for BM25.
  • Speed: Latency was 58–66 ms per query, compared with 55 ms for BM25 and about 3,500 ms for BM25 + BERT Large.
  • Cost: Training took under 1.5 hours on one TPU v3. Generating 5 queries for each of the 8.8M passages took about 40 hours, roughly $96 on preemptible TPUs.
  • Query quality: The generated queries scored 0.21 BLEU against dev queries, versus 0.088 for doc2query. T5-small, which has a parameter count similar to doc2query, scored 0.18 BLEU. Both models copy about 67% of their non-stopword terms from the passage and introduce about 33% new terms. Code and data are at github.com/castorini/docTTTTTquery.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelf 1The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 works

Cite

Cite as Rodrigo Nogueira and Jimmy Lin. 2019. From doc2query to docTTTTTquery. technical report, University of Waterloo.

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{nogueira2019doc2query,
  title = {{From doc2query to docTTTTTquery}},
  author = {Rodrigo Nogueira and Jimmy Lin},
  booktitle = {technical report, University of Waterloo},
  year = {2019},
  url = {https://cs.uwaterloo.ca/~jimmylin/publications/Nogueira_Lin_2019_docTTTTTquery-v2.pdf}
}

References

10 references, 6 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. P. Bajaj, D. Campos, N. Craswell, L. Deng, J. Gao, X. Liu, R. Majumder, A. McNamara, B. Mitra, T. Nguyen, et al. MS MARCO: A human generated MAchine Reading COmprehension dataset. arXiv:1611.09268, 2016.
  2. Z. Dai and J. Callan. Context-aware sentence/passage term importance estimation for first stage retrieval. arXiv:1910.10687, 2019. — [arXiv:1910.10687](https://arxiv.org/abs/1910.10687) find
  3. J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4171–4186, 2019…
  4. A. Fan, M. Lewis, and Y. Dauphin. Hierarchical neural story generation. arXiv:1805.04833, 2018.
  5. S. Hofstätter, N. Rekabsaz, C. Eickhoff, and A. Hanbury. On the effect of low-frequency terms on neural-IR models. arXiv:1904.12683, 2019. find
  6. R. Nogueira, W. Yang, K. Cho, and J. Lin. Multi-stage document ranking with BERT. arXiv:1910.14424, 2019. MRR@10 Dev Test 0.184 0.186 0.218 0.215 R@1000 Dev 0.853 0.891 Latency (ms/query) 55 61 0.259 0.265 0.272 0.277 0.278 - - - 0.272 - 0.929 0.939 0.944 0.947 0.945 58 61 62 64 66 0.243 0.290 0.375… find
  7. R. Nogueira, W. Yang, J. Lin, and K. Cho. Document expansion by query prediction. arXiv:1904.08375, 2019.
  8. C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv:1910.10683, 2019.
  9. A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pages 5998–6008, 2017.
  10. P. Yang, H. Fang, and J. Lin. Anserini: reproducible ranking baselines using Lucene. Journal of Data and Information Quality, 10(4):Article 16, 2018. find

Updated

This page in Markdown → https://md.research.elamur.ai/free/en/papers/nogueira-2019-doc2query-docttttquery/