FEVER: a Large-scale Dataset for Fact Extraction and VERification
Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/N18-1074.pdf
Summary
Before FEVER, datasets for automated fact checking held only a few hundred claims. Many of them also handed systems the source text instead of making them find it. FEVER provides 185,445 claims built from the introductory sections of about 50,000 popular Wikipedia pages (June 2017 dump). Annotators wrote short single-fact claims from sampled sentences. They then altered those claims with six mutation types inspired by natural logic, such as negation, entity substitution and generalization. In a separate pass, other annotators labelled each claim SUPPORTED, REFUTED or NOT ENOUGH INFO. For the first two labels, they also marked the sentences that justify the verdict, sometimes taken from several pages. To measure difficulty, the authors built a baseline pipeline with three steps: DrQA-based document retrieval, sentence selection, and a textual entailment classifier (decomposable attention). On the test set, it scores 31.87% accuracy when the correct evidence is required and 50.91% when evidence is ignored. Oracle runs show that selecting the evidence sentences is the hardest step. FEVER gives verification research a large testbed with evidence annotations.
Key points
- How much evidence claims need: 31.75% of claims have more than one valid evidence set, 16.82% need several sentences combined, and 12.15% need evidence from more than one Wikipedia page.
- Annotation quality: the labels reach a Fleiss κ of 0.6841. Evidence selection has 95.42% precision and 72.36% recall. Claims average 9.4 tokens, close to the hypothesis length in SNLI.
- Baseline pipeline on the test set: top-5 DrQA documents, then 5 retrieved sentences, then a Decomposable Attention model trained with the NEARESTP sampling strategy for NOT ENOUGH INFO examples. It reaches 31.87% when the correct evidence is required and 50.91% when it is not.
- Learning curves: Decomposable Attention is unstable with fewer than about 6,000 training examples, then overtakes an MLP baseline as data grows. This suggests the dataset is large enough to tell models apart.
- Released resources: the data at fever.ai, the annotation interfaces (github.com/awslabs/fever) and the baseline system (github.com/sheffieldnlp/fever-baselines).
Cited by
- Zero-Shot Stance Detection: A Dataset and Model using Generalized Topic Representations
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection
- MultiFC: A Real-World Multi-Domain Dataset for Evidence-Based Fact Checking of Claims
- The Internal State of an LLM Knows When It's Lying
- LLM2Vec: Large Language Models Are Secretly Powerful Text Encoders
- Attributed Question Answering: Evaluation and Modeling for Attributed Large Language Models
- TabFact: A Large-scale Dataset for Table-based Fact Verification
- FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios
- Promptagator: Few-shot Dense Retrieval From 8 Examples
- Explaining Answers with Entailment Trees
- CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
- Enabling Large Language Models to Generate Text with Citations
- RARR: Researching and Revising What Language Models Say, Using Language Models
- Retrieval-Augmented Generation for Large Language Models: A Survey
- CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing
- Annotating and Modeling Fine-grained Factuality in Summarization
- A Survey on Automated Fact-Checking
- Transformer-Patcher: One Mistake worth One Neuron
- Unsupervised Dense Information Retrieval with Contrastive Learning
- Survey of Hallucination in Natural Language Generation
- HoVer: A Dataset for Many-Hop Fact Extraction And Claim Verification
- SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization
- Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP
- Explainable Automated Fact-Checking for Public Health Claims
- Internet-augmented language models through few-shot prompting for open-domain question answering
- NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
- TruthfulQA: Measuring How Models Mimic Human Falsehoods
- RA-DIT: Retrieval-Augmented Dual Instruction Tuning
- WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation
- Fine-Tuning LLaMA for Multi-Stage Text Retrieval
- Teaching language models to support answers with verified quotes
- Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey
- FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
- Generative Representational Instruction Tuning
- A Comprehensive Overview of Large Language Models
- KILT: a Benchmark for Knowledge Intensive Language Tasks
- Automatic Detection of Fake News
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- Towards Debiasing Fact Verification Models
- Get Your Vitamin C! Robust Fact Verification with Contrastive Evidence
- Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy
- Prompting GPT-3 To Be Reliable
- Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
- One Embedder, Any Task: Instruction-Finetuned Text Embeddings
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Fact or Fiction: Verifying Scientific Claims
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
- Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificity
- Improving Text Embeddings with Large Language Models
- Multilingual E5 Text Embeddings: A Technical Report
- ReAct: Synergizing Reasoning and Acting in Language Models
- Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach
- DocNLI: A Large-scale Dataset for Document-level Natural Language Inference
- Making Retrieval-Augmented Language Models Robust to Irrelevant Context
- Generate rather than Retrieve: Large Language Models are Strong Context Generators
- Defending Against Neural Fake News
- AlignScore: Evaluating Factual Consistency with A Unified Alignment Function
- mGTE: Generalized Long-Context Text Representation and Reranking Models for Multilingual Text Retrieval
- AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models
- ToolQA: A Dataset for LLM Question Answering with External Tools
Cite
@inproceedings{thorne2018fever,
title = {{FEVER: a Large-scale Dataset for Fact Extraction and VERification}},
author = {Thorne, James and Vlachos, Andreas and Christodoulopoulos, Christos and Mittal, Arpit},
booktitle = {Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)},
year = {2018},
doi = {10.18653/v1/n18-1074},
eprint = {1803.05355},
archivePrefix = {arXiv},
url = {https://doi.org/10.18653/v1/n18-1074}
}References
- Edward Loper and Steven Bird. 2002. Nltk: The natural language toolkit. In Proceedings of the ACL-02 Workshop on Effective Tools and Methodologies for Teaching Natural Language Processing and Computational Linguistics - Volume 1. Association for Computational Linguistics, Stroudsburg, PA, USA, ETMTN… — [doi:10.3115/1118108.1118117](https://doi.org/10.3115/1118108.1118117) find
- Gabor Angeli and Christopher D. Manning. 2014. Nat-uralLI: Natural logic inference for common sense reasoning. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. pages 534–545. find
- Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. find
- Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL (System Demonstrations). pages 55–60. find our card
- Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, pages 1870–1879. https://d… find our card
- Mausam, Michael Schmitz, Robert Bart, Stephen Soderland, and Oren Etzioni. 2012. Open language learning for information extraction. In Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning. pages 523–534. find
- Ankur Parikh, Oscar Täckström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Austin, Texas, pages 2249–225… find our card
- Ido Dagan, Bill Dolan, Bernardo Magnini, and Dan Roth. 2009. Recognizing textual entailment: Rational, evaluation and approaches. Natural Language Engineering 15(4):i–xvii. https://doi.org/10.1017/S1351324909990209. — [doi:10.1017/s1351324909990209](https://doi.org/10.1017/s1351324909990209) ↗
- Joe Ellis, Jeremy Getman, Dana Fore, Neil Kuster, Zhiyi Song, Ann Bies, and Stephanie Strassel. 2016. Overview of Linguistic Resources for the TAC KBP 2016 Evaluations : Methodologies and Results. Proceedings of TAC KBP 2016 Workshop, National Institute of Standards and Technology, Maryland, USA (Ld… find
- Dean Pomerleau and Delip Rao. 2017. Fake news challenge. http://fakenewschallenge.org/. ↗
- P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Empirical Methods in Natural Language Processing (EMNLP). find our card
- William Ferreira and Andreas Vlachos. 2016. Emergent: a novel data-set for stance classification. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. San Diego, California, pages 1163–1168. find
- Benjamin Riedel, Isabelle Augenstein, George Spithourakis, and Sebastian Riedel. 2017. A simple but tough-to-beat baseline for the Fake News Challenge stance detection task. CoRR abs/1707.03264. http://arxiv.org/abs/1707.03264. ↗
- Joseph L Fleiss. 1971. Measuring nominal scale agreement among many raters. Psychological bulletin 76(5):378. find
- Tim Rocktäschel and Sebastian Riedel. 2017. End-to-end differentiable proving. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, California, United States. volume abs/1705.11040. http://arxiv.org… ↗
- Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2017. AllenNLP: A Deep Semantic Natural Language Processing Platform . find our card
- Álvaro Rodrigo, Anselmo Peñas, and Felisa Verdejo. 2009. Overview of the answer validation exercise 2008. In Carol Peters, Thomas Deselaers, Nicola Ferro, Julio Gonzalo, Gareth J. F. Jones, Mikko Kurimo, Thomas Mandl, Anselmo Peñas, and Vivien Petras, editors, Evaluating Systems for Multilingual… find
- Michael Heilman and Noah A. Smith. 2010. Good Question! statistical ranking for question generation. In Proceedings of the 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics. pages 609–617. find
- Mio Kobayashi, Ai Ishii, Chikara Hoshino, Hiroshi Miyashita, and Takuya Matsuzaki. 2017. Automated historical fact-checking by passage retrieval, word statistics, and virtual question-answering. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Lon… find
- Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. CoRR abs/1611.01603. http://arxiv.org/abs/1611.01603. ↗ our card
- Craig Silverman. 2015. Lies, Damn Lies and Viral Content. http: //towcenter.org/research/ lies-damn-lies-and-viral-content/. find
- Niket Tandon, Gerard de Melo, and Gerhard Weikum. 2011. Deriving a Web-scale common sense fact database. In Proceedings of the 25th AAAI Conference on Artificial Intelligence (AAAI 2011). AAAI Press, Palo Alto, CA, USA, pages 152–157. find
- Andreas Vlachos and Sebastian Riedel. 2014. Fact checking: Task definition and dataset construction. In Proceedings of the ACL 2014 Workshop on Language Technologies and Computational Social Science. ACL. http://www.aclweb.org/anthology/W14-2508. ↗ our card
- William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A new benchmark dataset for fake news detection. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). http://aclweb.org/anthology/P17-2067. ↗ our card