Elamur Research
☰
home / papers / 2022 / Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2212.14024

Summary

Pairing a frozen language model with a retriever usually follows a fixed recipe: fetch passages for the query, insert them into the prompt, and let the model answer. This fails when no single passage holds the answer, as with multi-hop or conversational questions. The authors propose DSP, a Python framework in which an LM and an RM exchange natural-language text inside short, task-specific programs built from composable operations. DEMONSTRATE builds few-shot examples by running the pipeline on labeled question–answer pairs and keeping only the runs that reach the correct answer. This means intermediate steps, such as search queries, need no manual annotation. SEARCH lets the LM formulate queries across several retrieval hops, resolve conversational context and condense what earlier hops found. PREDICT generates the final answer from the gathered passages, optionally sampling several candidates and picking one by self-consistency. With GPT-3.5 and ColBERTv2 and no fine-tuning, DSP programs achieved new in-context learning bests on Open-SQuAD, HotPotQA and QReCC. This suggests that carefully designed pipelines over off-the-shelf models can be a cheap alternative to task-specific training.

Key points

  • Setup: The LM is GPT-3.5 (text-davinci-002) and the RM is ColBERTv2, both used frozen without fine-tuning. The code is released at github.com/stanfordnlp/dsp.
  • Overall gains: Across tasks, DSP reports relative improvements of:
    • 37–120% over the vanilla LM,
    • 8–39% over a standard retrieve-then-read pipeline,
    • 80–290% over the contemporaneous self-ask pipeline.
  • HotPotQA results: In the open-domain "fullwiki" setting (Wikipedia 2017 abstracts), the multi-hop program reaches 51.4% EM. This beats the vanilla LM by 82%, retrieve-then-read by 39% and self-ask by 80% in EM. It also clearly surpasses concurrent methods such as a Wikipedia-API search system at 35.1% EM and PaLM-540B with self-consistency at 33.8% EM.
  • QReCC evaluation: QReCC is run in an open-domain setting over Wikipedia 2018. Conversations are evaluated auto-regressively, so the system sees its own earlier replies. The metric is novel-F1 (nF1), and noisy conversations are filtered out.
  • Supervision: Only end-task question–answer labels are required. Annotations for intermediate pipeline steps are bootstrapped automatically through weak supervision.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

160 papers of our library cite it (of the 140673 whose bibliographies we read); 17 of them have a page here, 143 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2212.14024 · arXiv 2212.14024

@misc{khattab2022demonstrate,
  title = {{Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP}},
  author = {Khattab, Omar and Santhanam, Keshav and Li, Xiang Lisa and Hall, David and Liang, Percy and Potts, Christopher and Zaharia, Matei},
  year = {2022},
  doi = {10.48550/arxiv.2212.14024},
  eprint = {2212.14024},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2212.14024}
}

References

62 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anantha, R., Vakulenko, S., Tu, Z., Longpre, S., Pulman, S., and Chappidi, S. Open-domain question answering goes conversational via question rewriting. arXiv preprint arXiv:2010.04898, 2020. find
  2. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877–1901, 2020. find
  3. Cao, G., Nie, J.-Y., Gao, J., and Robertson, S. Selecting good expansion terms for pseudo-relevance feedback. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval, pp. 243–250, 2008. find
  4. Chen, D., Fisch, A., Weston, J., and Bordes, A. Reading Wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1870–1879, Vancouver, Canada, 2017. Association for Computational Linguistics. do… find our card
  5. Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. find
  6. Del Tredici, M., Barlacchi, G., Shen, X., Cheng, W., and de Gispert, A. Question rewriting for open-domain conversational qa: Best practices and limitations. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management, pp. 2974–2978, 2021. find
  7. Dohan, D., Xu, W., Lewkowycz, A., Austin, J., Bieber, D., Lopes, R. G., Wu, Y., Michalewski, H., Saurous, R. A., Sohl-Dickstein, J., et al. Language model cascades. arXiv preprint arXiv:2207.10342, 2022. find
  8. Fox, E. A. and Shaw, J. A. Combination of multiple searches. NIST special publication SP, 243, 1994. find
  9. Gao, L., Dai, Z., Pasupat, P., Chen, A., Chaganty, A. T., Fan, Y., Zhao, V. Y., Lao, N., Lee, H., Juan, D.-C., et al. Attributed text generation via post-hoc research and revision. arXiv preprint arXiv:2210.08726, 2022. find our card
  10. Geva, M., Khashabi, D., Segal, E., Khot, T., Roth, D., and Berant, J. Did aristotle use a laptop? a question answering benchmark with implicit reasoning strategies. Transactions of the Association for Computational Linguistics, 9: 346–361, 2021. find
  11. Hofstätter, S., Chen, J., Raman, K., and Zamani, H. Fid-light: Efficient and effective retrieval-augmented text generation. arXiv preprint arXiv:2209.14290, 2022. — [arXiv:2209.14290](https://arxiv.org/abs/2209.14290) find
  12. Huang, J., Gu, S. S., Hou, L., Wu, Y., Wang, X., Yu, H., and Han, J. Large language models can self-improve. arXiv preprint arXiv:2210.11610, 2022. find our card
  13. Ishii, Y., Madotto, A., and Fung, P. Survey of hallucination in natural language generation. ACM Comput. Surv, 1(1), 2022. find our card
  14. Izacard, G. and Grave, E. Leveraging passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282, 2020. find our card
  15. Izacard, G., Lewis, P., Lomeli, M., Hosseini, L., Petroni, F., Schick, T., Dwivedi-Yu, J., Joulin, A., Riedel, S., and Grave, E. Few-shot learning with retrieval augmented language models. arXiv preprint arXiv:2208.03299, 2022. find
  16. Jiang, Y., Bordia, S., Zhong, Z., Dognin, C., Singh, M., and Bansal, M. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 3441–3460, Online, 2020. Association for Computational Linguistics. doi: 10.1865… find our card
  17. Karpukhin, V., Oguz, B., Min, S., Lewis, P., Wu, L., Edunov, S., Chen, D., and Yih, W.-t. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769–6781, Online, 2020. Association for Comput… find our card
  18. Khattab, O. and Zaharia, M. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Huang, J., Chang, Y., Cheng, X., Kamps, J., Murdock, V., Wen, J., and Liu, Y. (eds.), Proceedings of the 43rd International ACM SIGIR conference on research and development i… find our card
  19. ACM, 2020. doi: 10.1145/3397271.3401075. URL https://doi.org/10.1145/3397271.3401075. ↗ our card
  20. Khattab, O., Potts, C., and Zaharia, M. Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021a. find
  21. Khattab, O., Potts, C., and Zaharia, M. Relevance-guided supervision for openqa with ColBERT. Transactions of the Association for Computational Linguistics, 9:929– 944, 2021b. find
  22. Khot, T., Trivedi, H., Finlayson, M., Fu, Y., Richardson, K., Clark, P., and Sabharwal, A. Decomposed prompting: A modular approach for solving complex tasks. arXiv preprint arXiv:2210.02406, 2022. find
  23. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y. Large language models are zero-shot reasoners. arXiv preprint arXiv:2205.11916, 2022. find our card
  24. Krishna, K., Chang, Y., Wieting, J., and Iyyer, M. Rankgen: Improving text generation with large ranking models. arXiv preprint arXiv:2205.09726, 2022. find
  25. Kurland, O. and Culpepper, J. S. Fusion in information retrieval: Sigir 2018 half-day tutorial. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval, pp. 1383–1386, 2018. find
  26. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. Natural questions: A benchmark for question answering research. T… find our card
  27. Lazaridou, A., Gribovskaya, E., Stokowiec, W., and Grigorev, N. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022. find our card
  28. Le, N. T., Bai, F., and Ritter, A. Few-shot anaphora resolution in scientific protocols via mixtures of in-context experts. arXiv preprint arXiv:2210.03690, 2022. find
  29. Lee, K., Chang, M.-W., and Toutanova, K. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6086– 6096, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/… find our card
  30. Levine, Y., Dalmedigos, I., Ram, O., Zeldes, Y., Jannai, D., Muhlgay, D., Osin, Y., Lieber, O., Lenz, B., Shalev-Shwartz, S., et al. Standing on the shoulders of giant frozen language models. arXiv preprint arXiv:2204.10019, 2022. — [arXiv:2204.10019](https://arxiv.org/abs/2204.10019) find
  31. Lewis, P. S. H., Perez, E., Piktus, A., Petroni, F., Karpukhin, V., Goyal, N., Küttler, H., Lewis, M., Yih, W., Rocktäschel, T., Riedel, S., and Kiela, D. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Ad… find
  32. Li, X. L., Holtzman, A., Fried, D., Liang, P., Eisner, J., Hashimoto, T., Zettlemoyer, L., and Lewis, M. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022. find our card
  33. Liu, J., Shen, D., Zhang, Y., Dolan, B., Carin, L., and Chen, W. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804, 2021. find our card
  34. McCann, B., Keskar, N. S., Xiong, C., and Socher, R. The natural language decathlon: Multitask learning as question answering. arXiv:1806.08730, 2018. URL https://arxiv.org/abs/1806.08730. ↗ our card
  35. Min, S., Zhong, V., Zettlemoyer, L., and Hajishirzi, H. Multi-hop reading comprehension through question decomposition and rescoring. arXiv preprint arXiv:1906.02916, 2019. find
  36. Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C. L., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022. find
  37. Paranjape, A., Khattab, O., Potts, C., Zaharia, M., and Manning, C. D. Hindsight: Posterior-guided Training of Retrievers for Improved Open-ended Generation. In International Conference on Learning Representations, 2022. URL https://openreview.net/forum?id=Vr\_BTpw3wz. ↗
  38. Perez, E., Kiela, D., and Cho, K. True few-shot learning with language models. Advances in Neural Information Processing Systems, 34:11054–11070, 2021. find our card
  39. Press, O., Zhang, M., Min, S., Schmidt, L., Smith, N. A., and Lewis, M. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. find
  40. Qi, P., Lee, H., Sido, O., Manning, C. D., et al. Retrieve, rerank, read, then iterate: Answering open-domain questions of arbitrary complexity from text. arXiv preprint arXiv:2010.12527, 2020. URL https://arxiv.org/abs/2010.12527. — [arXiv:2010.12527](https://arxiv.org/abs/2010.12527) ↗
  41. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. find
  42. Rajpurkar, P., Zhang, J., Lopyrev, K., and Liang, P. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383– 2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653/v… find our card
  43. Raposo, G., Ribeiro, R., Martins, B., and Coheur, L. Question rewriting? assessing its importance for conversational question answering. In European Conference on Information Retrieval, pp. 199–206. Springer, 2022. find
  44. Santhanam, K., Khattab, O., Potts, C., and Zaharia, M. PLAID: An Efficient Engine for Late Interaction Retrieval. arXiv preprint arXiv:2205.09707, 2022a. find
  45. Santhanam, K., Khattab, O., Saad-Falcon, J., Potts, C., and Zaharia, M. ColBERTv2: Effective and efficient retrieval via lightweight late interaction. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp… find our card
  46. Shuster, K., Poff, S., Chen, M., Kiela, D., and Weston, J. Retrieval augmentation reduces hallucination in conversation. arXiv preprint arXiv:2104.07567, 2021. find our card
  47. Si, C., Gan, Z., Yang, Z., Wang, S., Wang, J., Boyd-Graber, J., and Wang, L. Prompting gpt-3 to be reliable. arXiv preprint arXiv:2210.09150, 2022. find our card
  48. Sun, Z., Wang, X., Tay, Y., Yang, Y., and Zhou, D. Recitation-augmented language models. arXiv preprint arXiv:2210.01296, 2022. — [arXiv:2210.01296](https://arxiv.org/abs/2210.01296) find
  49. Thorne, J., Vlachos, A., Christodoulopoulos, C., and Mittal, A. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)… find our card
  50. Vakulenko, S., Kiesel, J., and Fröbe, M. SCAI-QReCC shared task on conversational question answering. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 4913–4922, Marseille, France, June 2022. European Language Resources Association. URL https://aclanthology.org/2022… ↗
  51. Wang, X., Macdonald, C., Tonellotto, N., and Ounis, I. Colbert-prf: Semantic pseudo-relevance feedback for dense passage and document retrieval. ACM Transactions on the Web, 2022a. find
  52. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Rationale-augmented ensembles in language models. arXiv preprint arXiv:2207.00747, 2022b. find
  53. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., and Zhou, D. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171, 2022c. find our card
  54. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Chi, E., Le, Q., and Zhou, D. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. find our card
  55. Wiher, G., Meister, C., and Cotterell, R. On decoding strategies for neural text generators. arXiv preprint arXiv:2203.15721, 2022. find
  56. Xiong, W., Li, X. L., Iyer, S., Du, J., Lewis, P., Wang, W. Y., Mehdad, Y., Yih, W.-t., Riedel, S., Kiela, D., et al. Answering complex open-domain questions with multi-hop dense retrieval. arXiv preprint arXiv:2009.12756, 2020. URL https://arxiv.org/abs/2009.12756. — [arXiv:2009.12756](https://arxiv.org/abs/2009.12756) ↗
  57. Xue, X. and Croft, W. B. Modeling reformulation using query distributions. ACM Transactions on Information Systems (TOIS), 31(2):1–34, 2013. find
  58. Yang, Z., Qi, P., Zhang, S., Bengio, Y., Cohen, W. W., Salakhutdinov, R., and Manning, C. D. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600, 2018. find our card
  59. Yao, S., Zhao, J., Yu, D., Du, N., Shafran, I., Narasimhan, K., and Cao, Y. React: Synergizing reasoning and acting in language models. arXiv preprint arXiv:2210.03629, 2022. find our card
  60. Zelikman, E., Wu, Y., and Goodman, N. D. Star: Bootstrapping reasoning with reasoning. arXiv preprint arXiv:2203.14465, 2022. find our card
  61. Zhang, Z., Zhang, A., Li, M., and Smola, A. Automatic chain of thought prompting in large language models. arXiv preprint arXiv:2210.03493, 2022. find our card
  62. Zhong, V., Shi, W., Yih, W.-t., and Zettlemoyer, L. Romqa: A benchmark for robust, multi-evidence, multi-answer question answering. arXiv preprint arXiv:2210.14353, 2022. find