Elamur Research
☰
home / papers / 2020 / HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2020.findings-emnlp.309.pdf

Summary

Existing fact-checking benchmarks such as FEVER mostly need evidence from one Wikipedia page. Multi-hop QA sets like HotpotQA can often be solved through surface word overlap. HoVer targets claims whose verification requires chaining evidence across up to four English Wikipedia articles, where some supporting documents are hard to find by matching against the claim alone.

The authors had trained Appen annotators convert HotpotQA question–answer pairs into 2-hop claims. Annotators then extended a subset to 3 and 4 hops by replacing an entity with a description drawn from another article, often splitting the result into several sentences linked by coreference. Unsupported variants were produced through human edits and automatic substitutions. Because annotators could not reliably separate REFUTED from NOT ENOUGH INFO, those labels were merged into NOT-SUPPORTED.

A FEVER-style baseline (TF-IDF plus three fine-tuned BERT models) degrades sharply as the hop count grows and falls far below human performance. This makes HoVer a demanding testbed for many-hop retrieval and verification.

Key points

  • The dataset contains roughly 26k claims built from HotpotQA (its dev set plus 5,000 training examples). Each claim needs 2–4 supporting Wikipedia documents, and the reasoning graphs take varied shapes. The data is released at hover-nlp.github.io.
  • Bi-gram TF-IDF top-100 retrieval recovers the full set of supporting documents for 80% of 2-hop claims, 39% of 3-hop claims, and only 15% of 4-hop claims.
  • On SUPPORTED vs NOT-SUPPORTED classification, the baseline pipeline scores 73.7%. With gold evidence (oracle) it reaches 81.2%, and with no evidence at all it gets 63.7%.
  • On the combined HoVer score (full evidence retrieved and correct label), the best pipeline achieves 14.9% on the dev set, compared with 81% for humans.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

116 papers of our library cite it (of the 140673 whose bibliographies we read); 6 of them have a page here, 110 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.findings-emnlp.309

@inproceedings{jiang2020hover,
  title = {{HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification}},
  author = {Jiang, Yichen and Bordia, Shikha and Zhong, Zheng and Dognin, Charles and Singh, Maneesh and Bansal, Mohit},
  booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2020},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.309},
  url = {https://doi.org/10.18653/v1/2020.findings-emnlp.309}
}

References

32 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, and Caiming Xiong. 2020. Learning to retrieve reasoning paths over wikipedia graph for question answering. In ICLR. find our card
  2. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims. In EMNLP. find our card
  3. Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal. Association for Computational Linguistic… find
  4. Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL. find our card
  5. Jifan Chen and Greg Durrett. 2019. Understanding dataset design choices for multi-hop reasoning. In NAACL. find
  6. Sihao Chen, Daniel Khashabi, Wenpeng Yin, Chris Callison-Burch, and Dan Roth. 2019. Seeing things from a different angle: Discovering diverse perspectives about claims. In NAACL. find
  7. Tongfei Chen, Zhengping Jiang, Adam Poliak, Keisuke Sakaguchi, and Benjamin Van Durme. 2020a. Uncertain natural language inference. In ACL. find our card
  8. Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. 2020b. Tabfact: A large-scale dataset for table-based fact verification. In ICLR. find our card
  9. Yixin Nie, Haonan Chen, and Mohit Bansal. 2019a. Combining fact extraction and verification with neural semantic matching networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6859–6866. find
  10. P. Clark, Oyvind Tafjord, and Kyle Richardson. 2020. Transformers as soft reasoners over language. In IJ-CAI. find our card
  11. Yixin Nie, Songhe Wang, and Mohit Bansal. 2019b. Revealing the importance of semantic retrieval for machine reading at scale. In 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP). find
  12. Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2010. Recognizing textual entailment: Rational, evaluation and approaches–erratum. Natural Language Engineering, 16(1):105–105. find
  13. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. find our card
  14. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,… find
  15. Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In AAAI. find
  16. Yuwei Fang, Siqi Sun, Zhe Gan, Rohit Pillai, Shuohang Wang, and Jingjing Liu. 2019. Hierarchical graph network for multi-hop question answering. CoRR, abs/1911.03631. find
  17. Jorge Ramı́rez, Marcos Baez, Fabio Casati, and Boualem Benatallah. 2019. Understanding the impact of text highlighting in crowdsourcing tasks. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 7, pages 144–152. find
  18. William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 conference of the North American chapter of the association for computational linguistics: Human language technologies, pages 1163–1168. find
  19. Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop qa. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. find
  20. Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics:… find our card
  21. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP. find
  22. Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In ACL. find
  23. Peng Qi, Xiaowen Lin, Leo Mehr, Zijian Wang, and Christopher D Manning. 2019. Answering complex open-domain questions through iterative query generation. In EMNLP. find
  24. Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, New Orleans, Louisiana. Associatio… find our card
  25. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020. Winogrande: An adversarial winograd schema challenge at scale. In AAAI. find
  26. Tal Schuster, Darsh J. Shah, Yun Jie Serene Yeo, Daniel Filizzola, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In EMNLP/IJCNLP. find our card
  27. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In NAACL-HLT. find our card
  28. James Thorne, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulos, and Arpit Mittal. 2019. The fever2. 0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), pages 1–6. find
  29. Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. *Figure 2: Baseline system with the 4-stage architecture.* In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science, pages 18–22, Baltimore, MD, USA. Associat… find our card
  30. William Yang Wang. 2017. “liar, liar pants on fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada. Association for Computational Linguistics. find our card
  31. Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302. find our card
  32. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa… find