Elamur Research⌕
☰
home / papers / 2024 / AmbiFC: Fact-Checking Ambiguous Claims with Evidence

AmbiFC: Fact-Checking Ambiguous Claims with Evidence

Paper · 2024 · Max Glockner, Ieva Staliūnaitė, James Thorne et al.

Summary

Most fact-checking benchmarks expect one veracity verdict per claim. Real evidence, however, often allows several reasonable readings, and human annotators then disagree legitimately. AmbiFC addresses this gap. It is an English dataset of about 10k claims, built by having Mechanical Turk workers turn real yes/no search questions from BoolQ into statements. Each claim is paired with passages from the full Wikipedia page that BoolQ links to it. Annotators labelled each passage and each of its sentences as supporting, refuting or neutral. The authors keep every annotation as a valid perspective and do not merge them into a single label. Models therefore have to predict a soft label distribution for veracity and also select evidence sentences. The authors tagged linguistic phenomena and found that disagreement tracks things like underspecified claims and probabilistic reasoning, so it reflects ambiguity rather than noise. Among the modelling approaches tested, a pipeline that learns label distributions for both sentence selection and veracity prediction (the DISTILL variant) performed best. The resource matters because it pushes fact-checking systems to represent uncertainty instead of hiding it.

Key points

  • Scale: about 10k claims, with evidence annotations covering about 50k passages drawn from about 5k Wikipedia pages.
  • Claim collection: 512 qualified MTurk workers converted the BoolQ questions into claims.
  • Annotation design:
    • Passages were capped at 20 contiguous sentences.
    • Each passage first received two annotations. Three more were added if any initial annotation was non-neutral, giving at least five in those cases.
    • Annotators could flag a passage as "relevant" (for neutral labels) or "uncertain" (for non-neutral labels).
  • Subsets: the data is split into AmbiFC_C and AmbiFC_U. Models perform consistently worse on AmbiFC_U. Models trained on ambiguous instances cope better with the identified linguistic phenomena.
  • Results:
    • Allowing a tolerance of 0.15, the DISTILL pipeline matches human probabilities for about 70% of AmbiFC_C instances.
    • On 1,352 test cases that have both supporting and refuting annotations, giving the model only one side's evidence raises the correct class probability by 11.9% on average.
    • Evidence selection remains the weakest component.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Technische Universität DarmstadtDarmstadt · Germany109 works · 35,920 citations🎓The University of MelbourneMelbourne · Australia100 works · 22,618 citations🎓Korea Advanced Institute of Science and TechnologyDaejeon · South Korea51 works · 3,167 citations🔬Hessian Center for Artificial IntelligenceDarmstadt · Germany12 works · 529 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

Cite

Cite as Max Glockner, Ieva Staliūnaitė, James Thorne, Gisela Vallejo, Andreas Vlachos and Iryna Gurevych. 2024. AmbiFC: Fact-Checking Ambiguous Claims with Evidence. Transactions of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.1162/tacl_a_00629

@article{glockner2024ambifc,
  title = {{AmbiFC: Fact-Checking Ambiguous Claims with Evidence}},
  author = {Max Glockner, Ieva Staliūnaitė, James Thorne, Gisela Vallejo, Andreas Vlachos and Iryna Gurevych},
  journal = {Transactions of the Association for Computational Linguistics},
  year = {2024},
  doi = {10.1162/tacl\_a\_00629},
  url = {https://doi.org/10.1162/tacl_a_00629}
}

References

53 references, 28 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Rami Aly, Zhijiang Guo, Michael Sejr Schlichtkrull, James Thorne, Andreas Vlachos, Christos Christodoulopoulos, Oana Cocarascu, and Arpit Mittal. 2021. The Fact Extraction and Verification Over Unstructured and Structured information (FEVEROUS) Shared Task. In Proceedings of the Fourth Workshop on F…
  2. Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/ no questions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics…
  3. Isabelle Augenstein, Christina Lioma, Dongsheng Wang, Lucas Chaves Lima, Casper Hansen, Christian Hansen, and Jakob Grue Simonsen. 2019. MultiFC: A real-world multi-domain dataset for evidence-based fact checking of claims. In Proceedings of the 2019 Conference on Empirical Methods in Natural Langua…
  4. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Machine Learning Challenges. find
  5. Evaluating Predictive Uncertainty, Visual Object Classification, and Recognising Textual Entailment: First PASCAL Machine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11–13, 2005, Revised Selected Papers, pages 177–190. Springer. https://doi.org/10.1007/11736790 9 ↗
  6. Alexander Philip Dawid and Allan M. Skene. 1979. Maximum likelihood estimation of observer error-rates using the EM algorithm. Journal of the Royal Statistical Society. find
  7. Series Joris Baan, Wilker Aziz, Barbara Plank, and Raquel Fernandez. 2022. Stop measuring calibration when humans disagree. In Proceedings of the 2022 Conference on Empirical C (Applied Statistics), 28(1):20–28. https://doi.org/10.2307/2346806 page 719. https://doi.org/10.1163/9789004368811_003 ↗
  8. Shrey Desai and Greg Durrett. 2020. Calibration of pre-trained transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 295–302, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.emnlp-main.21 ↗
  9. Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q. Weinberger. 2017. On calibration of modern neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1321–1330. PMLR. find
  10. Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2022. A survey on automated fact-checking. Transactions of the Association for Computational Linguistics, 10:178–206. https://doi.org/10.1162/tacl a 00454 ↗
  11. Thomas Diggelmann, Jordan Boyd-Graber, Jannis Bulian, Massimiliano Ciaramita, and Markus Leippold. 2020. CLIMATE-FEVER: A dataset for verification of real-world climate claims. In Tackling Climate Change with Machine Learning workshop at NeurIPS 2020, Online.
  12. Ivan Habernal and Iryna Gurevych. 2017. Argumentation mining in user-generated web discourse. Computational Linguistics, 43(1):125–179. https://doi.org/10.1162/COLI a 00276 ↗
  13. Nouha Dziri, Ehsan Kamalloo, Sivan Milton, Osmar Zaiane, Mo Yu, Edoardo M. Ponti, and Siva Reddy. 2022. FaithDial: A faithful benchmark for information-seeking dialogue. Transactions of the Association for Computational Linguistics, 10:1473–1490. https://doi.org/10.1162/tacl a 00529 ↗
  14. Andreas Hanselowski, Christian Stab, Claudia Schulz, Zile Li, and Iryna Gurevych. 2019. A richly annotated corpus for different tasks in automated fact-checking. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 493–503, Hong Kong, China. Association for…
  15. Eve Fleisig, Rediet Abebe, and Dan Klein. 2023. When the majority is wrong: Leveraging annotator disagreement for subjective tasks. arXiv preprint arXiv:2305.06626v3. — [arXiv:2305.06626](https://arxiv.org/abs/2305.06626) find
  16. Tommaso Fornaciari, Alexandra Uma, Silviu Paun, Barbara Plank, Dirk Hovy, and Massimo Poesio. 2021. Beyond black & white: Leveraging annotator disagreement via soft-label multi-task learning. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Lin…
  17. Casper Hansen, Christian Hansen, and Lucas Chaves Lima. 2021. Automatic fake news detection: Are models learning to reason? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2:… find
  18. Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTaV3: Improving DeBERTa using ELECTRA-style pre-training with gradient-disentangled embedding sharing. arXiv preprint arXiv:2111.09543v3.
  19. Max Glockner, Yufang Hou, and Iryna Gurevych. 2022. Missing counter-evidence renders NLP fact-checking unrealistic for misinformation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5916–5936, Abu Dhabi, United Arab Emirates. Association for Computat… find
  20. Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. 2015. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531.
  21. Jerry R. Hobbs. 1979. Coherence and coreference. Cognitive Science, 3(1):67–90. https://doi.org/10.1207/s15516709cog0301 4 — [doi:10.1207/s15516709cog0301](https://doi.org/10.1207/s15516709cog0301) ↗
  22. H. P. Grice. 1975. Logic and conversation. Foundations of Cognitive Psychology, Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–…
  23. Association for Computational Linguistics. https://doi.org/10.18653/v1/2023 .acl-long.895 — [doi:10.18653/v1/2023](https://doi.org/10.18653/v1/2023) ↗
  24. Elisa Leonardelli, Alexandra Uma, Gavin Abercrombie, Dina Almanea, Valerio Basile, Tommaso Fornaciari, Barbara Plank, Verena Rieser, and Massimo Poesio. 2023. SemEval-2023 Task 11: Learning With Disagreements (LeWiDi). arXiv preprint arXiv:2304.14803v1. https://doi.org/10.18653/v1/2023 .semeval-1.31… ↗
  25. Nan-Jiang Jiang and Marie-Catherine de Marneffe. 2022. Investigating reasons for disagreement in natural language inference. Transactions of the Association for Computational Linguistics, 10:1357–1374. https://doi.org/10.1162/tacl_a_00523 ↗
  26. Chloe Lim. 2018. Checking how fact-checkers check. Research & Politics, 5(3):2053168018786848. https://doi.org/10.1177/2053168018786848 — [doi:10.1177/2053168018786848](https://doi.org/10.1177/2053168018786848) ↗
  27. Yichen Jiang, Shikha Bordia, Zheng Zhong, Charles Dognin, Maneesh Singh, and Mohit Bansal. 2020. HoVer: A dataset for many-hop fact extraction and claim verification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 3441–3460, Online. Association for Computational Ling…
  28. Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5783–5797, Online. Association for Computational Linguistics. https… find
  29. Ryo Kamoi, Tanya Goyal, Juan Diego Rodriguez, and Greg Durrett. 2023. WiCE: Real-world entailment for claims in Wikipedia. arXiv preprint arXiv:2303.01432v1.
  30. Yixin Nie, Xiang Zhou, and Mohit Bansal. 2020. What can we learn from collective human opinions on natural language inference data? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9131–9143, Online. Association for Computational Linguistics. h…
  31. Lauri Karttunen. 1974. Presupposition and linguistic context. Theoretical Linguistics, 1(1-3):181–194. https://doi.org/10.1515/thli.1974.1.1-3.181 ↗
  32. Rosanna Kenney and Peter Smith. 1997. Vagueness: A Reader. The MIT Press. https://doi.org/10.7551/mitpress/7064.001.0001 — [doi:10.7551/mitpress/7064.001.0001](https://doi.org/10.7551/mitpress/7064.001.0001) ↗
  33. Kashif Khan, Ruizhe Wang, and Pascal Poupart. 2022. WatClaimCheck: A new dataset for claim entailment and inference. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1293–1304, Dublin, Ireland. Association for Computational Lin… find
  34. Jungsoo Park, Sewon Min, Jaewoo Kang, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022. FaVIQ: FAct verification from information-seeking questions. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5154–5166, Dublin, Ireland. As… find
  35. Jiho Kim, Sungjin Park, Yeonsu Kwon, Yohan Jo, James Thorne, and Edward Choi. 2023. FactKG: Fact verification via reasoning on knowledge graphs. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16190–16206, Toronto, Canada. find
  36. Ellie Pavlick and Tom Kwiatkowski. 2019. Inherent disagreements in human textual inferences. Transactions of the Association for Computational Linguistics, 7:677–694. https://doi.org/10.1162/tacl_a_00293 ↗
  37. Joshua C. Peterson, Ruairidh M. Battleday, Thomas L. Griffiths, and Olga Russakovsky. 2019. Human uncertainty makes classification more robust. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9617–9626. https://doi.org/10 .1109/ICCV.2019.00971 checking of claims fro… find
  38. Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. AVeriTeC: A dataset for real-world claim verification with evidence from the Web. arXiv preprint arXiv:2305 .13117v2. find
  39. Barbara Plank. 2022. The ‘‘problem’’ of human label variation: On ground truth in data, modeling and evaluation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10671–10682, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. h…
  40. Tal Schuster, Adam Fisch, and Regina Barzilay. 2021. Get your vitamin C! Robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 624–643, Online. Assoc…
  41. Vinodkumar Prabhakaran, Aida Mostafazadeh Davani, and Mark Diaz. 2021. On releasing annotator-level labels and information in datasets. In Proceedings of the Joint 15th Linguistic Annotation Workshop (LAW) and 3rd Designing Meaning Representations (DMR) Workshop, pages 133–138, Punta Cana, Dominican…
  42. Tal Schuster, Darsh Shah, Yun Jie Serene Yeo, Daniel Roberto Filizzola Ortiz, Enrico Santus, and Regina Barzilay. 2019. Towards debiasing fact verification models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on…
  43. Arkadiy Saakyan, Tuhin Chakrabarty, and Smaranda Muresan. 2021. COVID-fact: Fact extraction and verification of real-world claims on COVID-19 pandemic. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural La…
  44. Neset Tan, Trung Nguyen, Josh Bensemann, Alex Peng, Qiming Bao, Yang Chen, Mark Gahegan, and Michael Witbrock. 2023. Multi2Claim: Generating scientific claims from multi-choice questions for scientific fact-checking. In Proceedings of the 17th Conference of the European Chapter of the Association fo… find
  45. Mourad Sarrouti, Asma Ben Abacha, Yassine Mrabet, and Dina Demner-Fushman. 2021. Evidence-based fact-checking of health-related claims. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 3499–3512, Punta Cana, Dominican Republic. Association for Computational Linguistics…
  46. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A Large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Aa…
  47. Empirical Methods in Natural Language Processing (EMNLP), pages 7534–7550, Online. Association for Computational Linguistics. https://doi.org/10.18653/v1/2020 .emnlp-main.609 — [doi:10.18653/v1/2020](https://doi.org/10.18653/v1/2020) ↗
  48. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2020. A case for soft loss functions. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 8, pages 173–177. https://doi.org/10.1609/hcomp.v8i1.7478 ↗
  49. Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pa…
  50. Alexandra Uma, Tommaso Fornaciari, Dirk Hovy, Silviu Paun, Barbara Plank, and Massimo Poesio. 2022. Learning from disagreement: A survey. Journal of Artificial Intelligence Research, 72:1385–1470. https://doi.org/10.1613/jair.1.12752 — [doi:10.1613/jair.1.12752](https://doi.org/10.1613/jair.1.12752) ↗
  51. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lho… — Transformers: State-of-the-Art Natural Language Processing
  52. Dmitry Ustalov, Nikita Pavlichenko, Vladimir Losev, Iulian Giliazev, and Evgeny Tulin. 2021. A general-purpose crowdsourcing computational quality control toolkit for Python. In The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track, HCOMP 2021. find
  53. David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020. Fact or fiction: Verifying scientific claims. In Proceedings of the 2020 Conference on

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/glockner-2024-ambifc-fact-checking-ambiguous-claims/