CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims
Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2012.00614
Summary
Automated fact-checking research has relied heavily on FEVER, whose claims were written by annotators and are short and simple. Real statements about climate change, which fuel public misinformation, look quite different. CLIMATE-FEVER adapts the FEVER methodology to such claims. The authors gathered over 3,000 candidate claims from the web, drawn equally from science-oriented and skeptic or denialist sources. Climate scientists then kept 1,535 claims they judged verifiable.
For each claim, a retrieval pipeline searched the full text of English Wikipedia and returned five candidate evidence sentences. Annotators decided whether every retrieved sentence backed the claim, contradicted it, or lacked sufficient information, and these judgments were then aggregated into a single label per claim. Unlike FEVER, the dataset also keeps "disputed" claims, which have evidence on both sides.
Annotator agreement was low, which shows the task is hard even for people. An ALBERT entailment model trained on FEVER performs well on FEVER itself but drops to near-chance accuracy on CLIMATE-FEVER. The dataset is therefore a demanding benchmark for natural language understanding and climate-disinformation tools.
Key points
- The dataset has 1,535 real-world climate claims and 7,675 claim–evidence pairs. It is publicly released via climatefever.ai.
- Claim-level labels are distributed as follows: SUPPORTS 655 (42.67%), REFUTES 253 (16.5%), DISPUTED 153 (9.97%), NOT_ENOUGH_INFO 474 (30.88%).
- The retrieval pipeline has three steps:
- BM25 (Lucene) with entity linking selects the top-10 Wikipedia articles.
- Siamese ALBERT large-v2 sentence embeddings with a FAISS index pick the top-100 sentences.
- An ALBERT base-v2 point-wise re-ranker keeps the top 5.
- Average inter-annotator agreement (Krippendorff's alpha) was 0.334. Trained or expert annotators tended to agree more.
- The FEVER-trained ALBERT large-v2 entailment model reaches 77.68% label accuracy on the FEVER dev set. On CLIMATE-FEVER it gets only 38.78% accuracy (weighted precision 56.49%, weighted F1 32.85%).
Cited by
- A Survey on Automated Fact-Checking
- TrustLLM: Trustworthiness in Large Language Models
- Datasets: A Community Library for Natural Language Processing
- Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
- ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction
- BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models
- Text Embeddings by Weakly-Supervised Contrastive Pre-training
Cite
@misc{diggelmann2020climate,
title = {{CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims}},
author = {Diggelmann, Thomas and Boyd-Graber, Jordan and Bulian, Jannis and Ciaramita, Massimiliano and Leippold, Markus},
year = {2020},
doi = {10.48550/arxiv.2012.00614},
eprint = {2012.00614},
archivePrefix = {arXiv},
url = {https://doi.org/10.48550/arxiv.2012.00614}
}References
- J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: a large-scale dataset for fact extraction and verification,” arXiv preprint arXiv:1803.05355, 2018. find our card
- W. R. Anderegg, J. W. Prall, J. Harold, and S. H. Schneider, “Expert credibility in climate change,” Proceedings of the National Academy of Sciences, vol. 107, no. 27, pp. 12 107–12 109, 2010. find
- D. Ding, E. W. Maibach, X. Zhao, C. Roser-Renouf, and A. Leiserowitz, “Support for climate policy and societal action are linked to perceptions about scientific agreement,” Nature Climate Change, vol. 1, no. 9, pp. 462–466, 2011. find
- S. D. Benegal and L. A. Scruggs, “Correcting misinformation about climate change: The impact of partisanship in an experimental setting,” Climatic change, vol. 148, no. 1-2, pp. 61–80, 2018. To stay compatible with the fever methodology for evaluation we simply excluded disputed claims from our data… find
- S. Van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, p. 1600008, 2017. find
- I. Augenstein, C. Lioma, D. Wang, L. C. Lima, C. Hansen, C. Hansen, and J. G. Simonsen, “Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint C… find our card
- A. Soleimani, C. Monz, and M. Worring, “BERT for evidence retrieval and claim verification,” in European Conference on Information Retrieval. Springer, 2020, pp. 359–366. find
- A. Hanselowski, H. Zhang, Z. Li, D. Sorokin, B. Schiller, C. Schulz, and I. Gurevych, “Ukp-athene: Multi-sentence textual entailment for claim verification,” arXiv preprint arXiv:1809.01479, 2018. find
- Y. Nie, H. Chen, and M. Bansal, “Combining fact extraction and verification with neural semantic matching networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 6859–6866. find
- W. Zhong, J. Xu, D. Tang, Z. Xu, N. Duan, M. Zhou, J. Wang, and J. Yin, “Reasoning over semantic-level graph for fact checking,” arXiv preprint arXiv:1909.03745, 2019. find
- Z. Liu, C. Xiong, M. Sun, and Z. Liu, “Fine-grained fact verification with kernel graph attention network,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7342–7351. find
- J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with gpus,” arXiv preprint arXiv:1702.08734, 2017. find our card
- V. Karpukhin, B. Oğuz, S. Min, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2004.04906, 2020. find our card
- Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” 2019. find
- T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natural language processing,” ArXiv, vol. abs/1910.03771, 2019. find our card
- S. Robertson and H. Zaragoza, The probabilistic relevance framework: BM25 and beyond. 2009. find our card
- N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019. find
- A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv preprint arXiv:1703.07737, 2017. — [arXiv:1703.07737](https://arxiv.org/abs/1703.07737) find
- L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018. find our card
- “Using umap for clustering — umap 0.4 documentation,” https://umap-learn.readthedocs.io/en/latest/clustering.html, (Accessed on 09/24/2020). ↗
- S. Louhichi, M. Gzara, and H. B. Abdallah, “A density based algorithm for discovering clusters with varied density,” in 2014 World Congress on Computer Applications and Information Systems (WCCAIS). IEEE, 2014, pp. 1–6. Now Publishers Inc, find