Elamur Research
☰
home / papers / 2023 / FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2310.03214

Summary

LLMs are usually trained once and then frozen, so they struggle with questions whose answers depend on recent events or rest on false assumptions. The authors built FreshQA, a dynamic benchmark of 600 natural questions written by NLP researchers and Upwork freelancers. Questions are labeled as never-changing, slow-changing, fast-changing, or false-premise, and as one-hop or multi-hop. They tested many closed and open-source models and collected over 50K human judgments under two scoring modes: Relaxed, which checks only the main answer, and Strict, which also penalizes any hallucinated claim.

Model size alone did little to help on fast-changing or false-premise questions. To address this, the authors propose FreshPrompt, a few-shot in-context method that requires no training. It formats Google Search results (answer box, organic results, related questions, etc.) with their dates and places them in the prompt. FreshPrompt outperforms Self-Ask and Perplexity.ai. Because the benchmark will be updated regularly, it offers an ongoing test of how factual and current LLMs remain.

Key points

  • FreshQA has a 500-question test set (125 per category), a 100-question dev set, and 15 few-shot demonstrations. It is released at github.com/freshllms/freshqa, and the authors commit to periodic answer updates.
  • The paper reports that GPT-4 + FreshPrompt beats vanilla GPT-4 by 32.6% (Relaxed) and 49.0% (Strict) in accuracy.
  • In the April 2023 Relaxed comparison, overall accuracy was:
    • GPT-4 + FreshPrompt: 77.8
    • Perplexity.ai: 66.2
    • GPT-4 + Self-Ask: 50.4
    • GPT-4 alone: 46.4
  • The number and ordering of retrieved evidences matter. With GPT-4, using only 1 evidence gives 67.2, while using 15 evidences gives 79.0.
  • Asking the model to check the question's premise raised GPT-4 + FreshPrompt's false-premise accuracy from 75.0 to 86.3. Requesting concise answers reduced hallucination compared with verbose ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

123 papers of our library cite it (of the 140673 whose bibliographies we read); 10 of them have a page here, 113 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2310.03214 · arXiv 2310.03214

@misc{vu2023freshllms,
  title = {{FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation}},
  author = {Vu, Tu and Iyyer, Mohit and Wang, Xuezhi and Constant, Noah and Wei, Jerry and Wei, Jason and Tar, Chris and Sung, Yun-Hsuan and Zhou, Denny and Le, Quoc and Luong, Thang},
  year = {2023},
  doi = {10.48550/arxiv.2310.03214},
  eprint = {2310.03214},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2310.03214}
}

References

36 references, 18 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin… find
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chri… find
  3. Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021a. URL https://arxiv. org/abs/2107.03374. find our card
  4. Wenhu Chen, Xinyi Wang, William Yang Wang, and William Yang Wang. A dataset for answering time-sensitive questions. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), volume 1, 2021b. URL https://datasets-benchmarks-proceedings.neurips.cc/paper\_fi… ↗
  5. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022. URL https: //arxiv.org/abs/2204.02311. find
  6. Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022. URL https://arxiv.org/abs/2210.11416. ↗
  7. Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 3558–3567, 2019. URL https://aclanthology.org/P19-1346. ↗ our card
  8. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval augmented language model pre-training. In Proceedings of the 37th International Conference on Machine Learning (ICML), volume 119 of Proceedings of Machine Learning Research (PMLR), pp. 3929– 3938. PMLR, 2020. URL http… find
  9. Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with retrieval augmented language models. 2022. URL https://arxiv.org/abs/2208.03299. ↗
  10. Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1601–1611, 2017. URL https://aclanthology.org/… ↗ our card
  11. Jungo Kasai, Keisuke Sakaguchi, Yoichi Takahashi, Ronan Le Bras, Akari Asai, Xinyan Yu, Dragomir Radev, Noah A Smith, Yejin Choi, and Kentaro Inui. Realtime qa: What’s the answer right now? 2022. URL https://arxiv.org/abs/2207.13332. ↗ our card
  12. Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-search-predict: Composing retrieval and language models for knowledge-intensive nlp. 2022. URL https://arxiv.org/abs/2212.14024. ↗ our card
  13. Najoung Kim, Phu Mon Htut, Samuel R. Bowman, and Jackson Petty. (QA) 2 : Question answering with questionable assumptions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 8466–8487, 2023. URL https://aclanthology.org/2023.acl-long.472. ↗
  14. Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiec, and Nikolai Grigorev. Internet-augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2203.05115, 2022. find our card
  15. Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 3045–3059, November 2021. URL https://aclanthology.org/2021.emnlp-main.243. ↗
  16. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks. In Advances in Neural Information Processing… find
  17. Adam Liska, Tomas Kocisky, Elena Gribovskaya, Tayfun Terzi, Eren Sezener, Devang Agrawal, Cyprien De Masson D’Autume, Tim Scholtes, Manzil Zaheer, Susannah Young, Ellen Gilsenan-Mcmahon, Sophia Austin, Phil Blunsom, and Angeliki Lazaridou. StreamingQA: A benchmark for adaptation to new knowledge ove… find
  18. Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172, 2023a. URL https://arxiv.org/abs/2307.03172. ↗ our card
  19. Nelson F Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. 2023b. URL https://arxiv.org/abs/2304.09848. ↗ our card
  20. Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301.13688, 2023. URL https://arxiv. org/abs/2301.13688. find our card
  21. Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. 2023. URL https://arxiv.org/abs/2305.15225. ↗
  22. Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1906–1919, 2020. URL https://aclanthology.org/2020.acl-main.173. ↗
  23. Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell-Gillingham, Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. arXiv preprint arXiv:2203.11147, 2022. URL https://arxiv.org/ab… ↗ our card
  24. Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser-assisted question-answering with human feedback. 2021. URL https://arxiv.org/abs/2112.09332. ↗ our card
  25. OpenAI. Gpt-4 technical report. arXiv preprint arXiv:2303.0877, 2023. URL https://arxiv.org/abs/2303.0877. ↗ our card
  26. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. Training… find
  27. Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. Measuring and narrowing the compositionality gap in language models. arXiv preprint arXiv:2210.03350, 2022. URL https://arxiv.org/abs/2210.03350. ↗
  28. Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 21(140):1–67, 2020. URL https://jmlr.org/paper… ↗
  29. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761, 2023. URL https://arxiv.org/abs/2302.04761. ↗ our card
  30. Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmented language models. Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), 2023. URL https://openreview.net/forum?id=-cqvvvb-NkI. ↗
  31. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022. URL https://arxiv.org/abs/2201.11903. ↗ our card
  32. Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. 2022. URL https://arxiv.org/abs/2210.03629. ↗ our card
  33. Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), 2023a… find our card
  34. Wenhao Yu, Meng Jiang, Peter Clark, and Ashish Sabharwal. Ifqa: A dataset for open-domain question answering under counterfactual presuppositions. 2023b. URL https://arxiv.org/abs/2305.14010. ↗ our card
  35. Xinyan Yu, Sewon Min, Luke Zettlemoyer, and Hannaneh Hajishirzi. CREPE: Open-domain question answering with false presuppositions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 10457–10480, 2023c. URL https: //aclanthology.org/2023.acl-long.583… find our card
  36. Michael Zhang and Eunsol Choi. SituatedQA: Incorporating extra-linguistic contexts into QA. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7371–7387, 2021. URL https://aclanthology.org/2021.emnlp-main.586. ↗