HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/D18-1259.pdf
Summary
Earlier QA benchmarks such as SQuAD, TriviaQA and SearchQA can mostly be solved by matching a question to one or a few nearby sentences. Multi-hop sets such as QAngaroo and COMPLEXWEBQUESTIONS are limited by the schemas of the knowledge bases used to build them. None of these datasets show a model which evidence supports an answer. HotpotQA fills these gaps with about 113k crowdsourced question-answer pairs based on English Wikipedia. The authors build a graph from the hyperlinks in each article's opening paragraph. They then pair linked paragraphs through hand-picked "bridge" entities. They also draw pairs of entities from curated lists to get comparison questions, some of them yes/no. Mechanical Turk workers write the questions and also mark the sentences their answers depend on. A baseline model trained with these sentence labels gains about 2 F1. With distractor paragraphs added, it still trails humans by roughly 30 points on answer EM and F1. HotpotQA therefore offers a hard, interpretable benchmark for reasoning across documents and for checking whether a model can justify its answers.
Key points
- The dataset has 112,779 examples, collected on Amazon Mechanical Turk through the ParlAI interface. The splits are train-easy (18,089, single-hop), train-medium (56,814), train-hard (15,661), and dev, test-distractor and test-fullwiki (7,405 each).
- There are two question types. Bridge-entity questions (about 75% of collection) come from edges in the Wikipedia hyperlink graph. Comparison questions come from 42 manually curated entity lists, and half of them are yes/no questions.
- Every example has sentence-level supporting facts. These give strong supervision during training and let evaluators score explainability by comparing predicted evidence to gold evidence.
- On 1,000 sampled examples in the distractor setting, the baseline reached 68.99 answer F1 and 52.37 joint F1. Humans reached 91.40 and 82.55, and the human upper bound for answer F1 is 98.77.
- Using the gold supporting facts as the only context adds 10+ F1. This suggests the paper's way of using the supervision (about 2 F1) is far from optimal. Training on all three train splits together gave the best results.
Cited by
- Aya 23: Open Weight Releases to Further Multilingual Progress
- Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering
- LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling
- HybridQA: A Dataset of Multi-Hop Question Answering over Tabular and Textual Data
- FinQA: A Dataset of Numerical Reasoning over Financial Data
- M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation
- BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions
- Transformers as Soft Reasoners over Language
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- Explaining Answers with Entailment Trees
- A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers
- Few-NERD: A Few-shot Named Entity Recognition Dataset
- DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs
- English Machine Reading Comprehension Datasets: A Survey
- ELI5: Long Form Question Answering
- Retrieval-Augmented Generation for Large Language Models: A Survey
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- Evaluating Large Language Models: A Comprehensive Survey
- HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models
- FOLIO: Natural Language Reasoning with First-Order Logic
- Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps
- RULER: What's the Real Context Size of Your Long-Context Language Models?
- Large Language Models Cannot Self-Correct Reasoning Yet
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
- TrustLLM: Trustworthiness in Large Language Models
- Adaptive-RAG: Learning to Adapt Retrieval-Augmented Large Language Models through Question Complexity
- Survey of Hallucination in Natural Language Generation
- PubMedQA: A Dataset for Biomedical Research Question Answering
- Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning
- SpanBERT: Improving Pre-training by Representing and Predicting Spans
- Evaluating Open-Domain Question Answering in the Era of Large Language Models
- CTRL: A Conditional Transformer Language Model for Controllable Generation
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines
- Natural Questions: A Benchmark for Question Answering Research
- Internet-augmented language models through few-shot prompting for open-domain question answering
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
- HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models
- LooGLE: Can Long-Context Language Models Understand Long Contexts?
- Making Language Models Better Reasoners with Step-Aware Verifier
- SnapKV: LLM Knows What You are Looking for Before Generation
- Search-o1: Agentic Search-Enhanced Large Reasoning Models
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning
- LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning
- Query Rewriting in Retrieval-Augmented Large Language Models
- Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools
- Large Language Models: A Survey
- SPARTQA: A Textual Question Answering Benchmark for Spatial Reasoning
- SGPT: GPT Sentence Embeddings for Semantic Search
- Generative Representational Instruction Tuning
- MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answering
- QuALITY: Question Answering with Long Input Texts, Yes!
- KILT: a Benchmark for Knowledge Intensive Language Tasks
- Humanity's Last Exam
- PragmatiCQA: A Dataset for Pragmatic Question Answering in Conversations
- Tool Learning with Foundation Models
- Pre-trained Models for Natural Language Processing: A Survey
- GPQA: A Graduate-Level Google-Proof Q&A Benchmark
- A guide to the dataset explosion in QA, NLI, and commonsense reasoning
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Texts
- Reflexion: Language Agents with Verbal Reinforcement Learning
- Prompting GPT-3 To Be Reliable
- Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuning
- R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
- ASQA: Factoid Questions Meet Long-Form Answers
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings
- oLMpics -- On what Language Model Pre-training Captures
- Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference
- Long Range Arena: A Benchmark for Efficient Transformers
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
- Solving math word problems with process- and outcome-based feedback
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
- A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation
- Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models
- SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer
- Self-Consistency Improves Chain of Thought Reasoning in Language Models
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- Executable Code Actions Elicit Better LLM Agents
- Improving Text Embeddings with Large Language Models
- Multilingual E5 Text Embeddings: A Technical Report
- Break It Down: A Question Understanding Benchmark
- BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysis
- Retrieval meets Long Context Large Language Models
- ReAct: Synergizing Reasoning and Acting in Language Models
- LinkBERT: Pretraining Language Models with Document Links
- The Unreliability of Explanations in Few-shot Prompting for Textual Reasoning
- IfQA: A Dataset for Open-domain Question Answering under Counterfactual Presuppositions
- Big Bird: Transformers for Longer Sequences
- AgentTuning: Enabling Generalized Agent Abilities for LLMs
- ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehension
- mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
- A Survey of Large Language Models
Cite
@inproceedings{yang2018hotpotqa,
title = {{HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering}},
author = {Yang, Zhilin and Qi, Peng and Zhang, Saizheng and Bengio, Yoshua and Cohen, William and Salakhutdinov, Ruslan and Manning, Christopher D.},
booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
year = {2018},
doi = {10.18653/v1/d18-1259},
eprint = {1809.09600},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/d18-1259}
}References
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open-domain questions. In Association for Computational Linguistics (ACL). find our card
- Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (EMNLP). find our card
- Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. In Proceedings of the 55th Annual Meeting of the Association of Computational Linguistics. find our card
- Shimi Salant and Jonathan Berant. 2018. Contextualized word representations for reading comprehension. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
- Matthew Dunn, Levent Sagun, Mike Higgins, Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179. find our card
- Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proceedings of the International Conference on Learning Representations. find our card
- Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. find our card
- Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 16th Annual Conference of the North American Chapter of the Association for Computational Linguistics. find
- Xiaodong Liu, Yelong Shen, Kevin Duh, and Jianfeng Gao. 2018. Stochastic answer networks for machine reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find
- Wenhui Wang, Nan Yang, Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 189–198. find
- Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations, pages 55–60. find our card
- Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association of Computational Linguistics. find our card
- Alexander H Miller, Will Feng, Adam Fisch, Jiasen Lu, Dhruv Batra, Antoine Bordes, Devi Parikh, and Jason Weston. 2017. ParlAI: A dialog research software platform. arXiv preprint arXiv:1705.06476. find our card
- Caiming Xiong, Victor Zhong, and Richard Socher. 2018. DCN+: Mixed objective and deep residual coattention for question answering. In Proceedings of the International Conference on Learning Representations. find
- Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. In Proceedings of the 30th Annual Conference on Neural Information Processing Systems (NIPS). find our card
- Zhilin Yang, Saizheng Zhang, Jack Urbanek, Will Feng, Alexander H Miller, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Mastering the dungeon: Grounded language learning by mechanical turker descent. In Proceedings of the International Conference on Learning Representations. find
- Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for NLG. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. find
- Boyuan Pan, Hao Li, Zhou Zhao, Bin Cao, Deng Cai, and Xiaofei He. 2017. Memen: Multi-layer embedding with memory networks for machine comprehension. arXiv preprint arXiv:1707.09098. — [arXiv:1707.09098](https://arxiv.org/abs/1707.09098) find
- Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don’t know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics. find our card