Elamur Research⌕
☰
home / papers / 2024 / MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries

Summary

Most retrieval-augmented generation (RAG) evaluation uses questions that a single passage can answer. Real users often need several documents combined, for example to compare firms, order events in time, or draw inferences across sources. Existing benchmarks such as RGB and RECALL do not test this. The authors introduce MultiHop-RAG, an English benchmark whose knowledge base is news collected through the mediastack API. The articles are dated after the training cutoff of popular LLMs, so models cannot rely on memorized facts.

GPT-4 builds the queries in stages. It pulls factual sentences from articles, rewrites each as a claim tagged with a bridge-topic and a bridge-entity, and then writes questions that link articles sharing a bridge. A validation stage, combining human effort with the model, checks the output. Each item comes with a gold answer and its supporting evidence.

Two experiments with a LlamaIndex pipeline compare embedding models for evidence retrieval and test several LLMs answering from retrieved text. Both show that current RAG setups handle multi-hop queries poorly. This makes the dataset a useful testbed for better retrieval and reasoning.

Key points

  • The benchmark has four query types: Inference, Comparison, Temporal, and Null. Null queries have no answer in the corpus and test whether the model hallucinates.
  • The corpus is English news from September 26 to December 26, 2023. It spans entertainment, business, sports, technology, health, and science.
  • Retrieval is scored with MAP@K, MRR@K, and Hit@K. Generation is scored by accuracy against short gold answers: yes/no, entity names, or before/after.
  • The LLMs tested are GPT-4, GPT-3.5, PaLM, Claude-2, Llama2-70B, and Mixtral-8x7B.
  • Each query needs at most four evidence pieces. The dataset and RAG code are public at github.com/yixuantt/MultiHop-RAG.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing · AI & machine learning

Cited by

87 papers of our library cite it (of the 140884 whose bibliographies we read); 2 of them have a page here, 85 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2401.15391 · arXiv 2401.15391

@misc{tang2024multihop,
  title = {{MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queries}},
  author = {Tang, Yixuan and Yang, Yi},
  year = {2024},
  doi = {10.48550/arxiv.2401.15391},
  eprint = {2401.15391},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2401.15391}
}

References

28 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2023. Claude 2.1 (May version). https://api.anthropic.com/v1/messages. Claude 2.1. ↗
  2. Akari Asai, Sewon Min, Zexuan Zhong, and Danqi Chen. 2023. Retrieval-based language models and applications. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 6: Tutorial Abstracts), pages 41–46. find
  3. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin… find
  4. Harrison Chase. 2022. LangChain. find
  5. Jiawei Chen, Hongyu Lin, Xianpei Han, and Le Sun. 2023. Benchmarking large language models in retrieval-augmented generation. find our card
  6. Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023. Ragas: Automated evaluation of retrieval augmented generation. find our card
  7. Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. 2023. Enabling large language models to generate text with citations. find our card
  8. Google. 2023. PaLM 2 (May version). https://generativelanguage.googleapis.com/v1beta2/models/. Chat-bison-002. ↗
  9. Significant Gravitas. 2023. Autogpt. https://github.com/Significant-Gravitas/AutoGPT. ↗
  10. Michael Günther, Jackmin Ong, Isabelle Mohr, Alaeddine Abdessalem, Tanguy Abel, Mohammad Kalim Akram, Susana Guzman, Georgios Mastrapas, Saba Sturua, Bo Wang, Maximilian Werk, Nan Wang, and Han Xiao. 2023. Jina embeddings 2: 8192-token general-purpose text embeddings for long documents. find
  11. Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Comm… find our card
  12. Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock,… find
  13. Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  14. Ehsan Kamalloo, Xinyu Zhang, Odunayo Ogundepo, Nandan Thakur, David Alfonso-Hermelo, Mehdi Rezagholizadeh, and Jimmy Lin. 2023. Evaluating embedding apis for information retrieval. arXiv preprint arXiv:2305.06300. find
  15. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking Beyond the Surface: A Challenge Set for Reading Comprehension over Multiple Sentences. In Proc. of the Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAAC… find our card
  16. Jerry Liu. 2022. LlamaIndex. find
  17. Yi Liu, Lianzhe Huang, Shicheng Li, Sishuo Chen, Hao Zhou, Fandong Meng, Jie Zhou, and Xu Sun. 2023. Recall: A benchmark for llms robustness against external counterfactual knowledge. find
  18. OpenAI. 2023. GPT4 (Nov 7 version). https://chat.openai.com/chat. gpt-4-1106-preview. ↗
  19. Jon Saad-Falcon, Omar Khattab, Christopher Potts, and Matei Zaharia. 2023. Ares: An automated evaluation framework for retrieval-augmented generation systems. find
  20. Hongjin Su, Weijia Shi, Jungo Kasai, Yizhong Wang, Yushi Hu, Mari Ostendorf, Wen tau Yih, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2023. One embedder, any task: Instruction-finetuned text embeddings. find our card
  21. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. find our card
  22. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller,… find
  23. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association… find our card
  24. Liang Wang, Nan Yang, Xiaolong Huang, Binxing Jiao, Linjun Yang, Daxin Jiang, Rangan Majumder, and Furu Wei. 2022. Text embeddings by weakly-supervised contrastive pre-training. arXiv preprint arXiv:2212.03533. find our card
  25. Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. 2023. C-pack: Packaged resources to advance general chinese embedding. find
  26. Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
  27. Peitian Zhang, Shitao Xiao, Zheng Liu, Zhicheng Dou, and Jian-Yun Nie. 2023. Retrieve anything to augment large language models. find
  28. Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards a unified multidimensional evaluator for text generation. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tang-2024-multihop-rag-benchmarking-retrieval-augmented/