Elamur Research
☰
home / papers / 2020 / CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims

Written for the Research library (claude-opus-5-5) from the paper text, in our own words: the license "arXiv — non-exclusive license to distribute" does not allow redistributing the text itself. Original — arxiv.org/pdf/2012.00614

Summary

Automated fact-checking research has relied heavily on FEVER, whose claims were written by annotators and are short and simple. Real statements about climate change, which fuel public misinformation, look quite different. CLIMATE-FEVER adapts the FEVER methodology to such claims. The authors gathered over 3,000 candidate claims from the web, drawn equally from science-oriented and skeptic or denialist sources. Climate scientists then kept 1,535 claims they judged verifiable.

For each claim, a retrieval pipeline searched the full text of English Wikipedia and returned five candidate evidence sentences. Annotators decided whether every retrieved sentence backed the claim, contradicted it, or lacked sufficient information, and these judgments were then aggregated into a single label per claim. Unlike FEVER, the dataset also keeps "disputed" claims, which have evidence on both sides.

Annotator agreement was low, which shows the task is hard even for people. An ALBERT entailment model trained on FEVER performs well on FEVER itself but drops to near-chance accuracy on CLIMATE-FEVER. The dataset is therefore a demanding benchmark for natural language understanding and climate-disinformation tools.

Key points

  • The dataset has 1,535 real-world climate claims and 7,675 claim–evidence pairs. It is publicly released via climatefever.ai.
  • Claim-level labels are distributed as follows: SUPPORTS 655 (42.67%), REFUTES 253 (16.5%), DISPUTED 153 (9.97%), NOT_ENOUGH_INFO 474 (30.88%).
  • The retrieval pipeline has three steps:
    • BM25 (Lucene) with entity linking selects the top-10 Wikipedia articles.
    • Siamese ALBERT large-v2 sentence embeddings with a FAISS index pick the top-100 sentences.
    • An ALBERT base-v2 point-wise re-ranker keeps the top 5.
  • Average inter-annotator agreement (Krippendorff's alpha) was 0.334. Trained or expert annotators tended to agree more.
  • The FEVER-trained ALBERT large-v2 entailment model reaches 77.68% label accuracy on the FEVER dev set. On CLIMATE-FEVER it gets only 38.78% accuracy (weighted precision 56.49%, weighted F1 32.85%).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Other

Cited by

135 papers of our library cite it (of the 140673 whose bibliographies we read); 7 of them have a page here, 128 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2012.00614 · arXiv 2012.00614

@misc{diggelmann2020climate,
  title = {{CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims}},
  author = {Diggelmann, Thomas and Boyd-Graber, Jordan and Bulian, Jannis and Ciaramita, Massimiliano and Leippold, Markus},
  year = {2020},
  doi = {10.48550/arxiv.2012.00614},
  eprint = {2012.00614},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2012.00614}
}

References

21 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Thorne, A. Vlachos, C. Christodoulopoulos, and A. Mittal, “FEVER: a large-scale dataset for fact extraction and verification,” arXiv preprint arXiv:1803.05355, 2018. find our card
  2. W. R. Anderegg, J. W. Prall, J. Harold, and S. H. Schneider, “Expert credibility in climate change,” Proceedings of the National Academy of Sciences, vol. 107, no. 27, pp. 12 107–12 109, 2010. find
  3. D. Ding, E. W. Maibach, X. Zhao, C. Roser-Renouf, and A. Leiserowitz, “Support for climate policy and societal action are linked to perceptions about scientific agreement,” Nature Climate Change, vol. 1, no. 9, pp. 462–466, 2011. find
  4. S. D. Benegal and L. A. Scruggs, “Correcting misinformation about climate change: The impact of partisanship in an experimental setting,” Climatic change, vol. 148, no. 1-2, pp. 61–80, 2018. To stay compatible with the fever methodology for evaluation we simply excluded disputed claims from our data… find
  5. S. Van der Linden, A. Leiserowitz, S. Rosenthal, and E. Maibach, “Inoculating the public against misinformation about climate change,” Global Challenges, vol. 1, no. 2, p. 1600008, 2017. find
  6. I. Augenstein, C. Lioma, D. Wang, L. C. Lima, C. Hansen, C. Hansen, and J. G. Simonsen, “Multifc: A real-world multi-domain dataset for evidence-based fact checking of claims,” in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint C… find our card
  7. A. Soleimani, C. Monz, and M. Worring, “BERT for evidence retrieval and claim verification,” in European Conference on Information Retrieval. Springer, 2020, pp. 359–366. find
  8. A. Hanselowski, H. Zhang, Z. Li, D. Sorokin, B. Schiller, C. Schulz, and I. Gurevych, “Ukp-athene: Multi-sentence textual entailment for claim verification,” arXiv preprint arXiv:1809.01479, 2018. find
  9. Y. Nie, H. Chen, and M. Bansal, “Combining fact extraction and verification with neural semantic matching networks,” in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, 2019, pp. 6859–6866. find
  10. W. Zhong, J. Xu, D. Tang, Z. Xu, N. Duan, M. Zhou, J. Wang, and J. Yin, “Reasoning over semantic-level graph for fact checking,” arXiv preprint arXiv:1909.03745, 2019. find
  11. Z. Liu, C. Xiong, M. Sun, and Z. Liu, “Fine-grained fact verification with kernel graph attention network,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 7342–7351. find
  12. J. Johnson, M. Douze, and H. Jégou, “Billion-scale similarity search with gpus,” arXiv preprint arXiv:1702.08734, 2017. find our card
  13. V. Karpukhin, B. Oğuz, S. Min, L. Wu, S. Edunov, D. Chen, and W.-t. Yih, “Dense passage retrieval for open-domain question answering,” arXiv preprint arXiv:2004.04906, 2020. find our card
  14. Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut, “ALBERT: A lite BERT for self-supervised learning of language representations,” 2019. find
  15. T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, and J. Brew, “Huggingface’s transformers: State-of-the-art natural language processing,” ArXiv, vol. abs/1910.03771, 2019. find our card
  16. S. Robertson and H. Zaragoza, The probabilistic relevance framework: BM25 and beyond. 2009. find our card
  17. N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084, 2019. find
  18. A. Hermans, L. Beyer, and B. Leibe, “In defense of the triplet loss for person re-identification,” arXiv preprint arXiv:1703.07737, 2017. — [arXiv:1703.07737](https://arxiv.org/abs/1703.07737) find
  19. L. McInnes, J. Healy, and J. Melville, “Umap: Uniform manifold approximation and projection for dimension reduction,” arXiv preprint arXiv:1802.03426, 2018. find our card
  20. “Using umap for clustering — umap 0.4 documentation,” https://umap-learn.readthedocs.io/en/latest/clustering.html, (Accessed on 09/24/2020). ↗
  21. S. Louhichi, M. Gzara, and H. B. Abdallah, “A density based algorithm for discovering clusters with varied density,” in 2014 World Congress on Computer Applications and Information Systems (WCCAIS). IEEE, 2014, pp. 1–6. Now Publishers Inc, find