Elamur Research⌕
☰
home / papers / 2026 / Multimodal Claim Extraction for Fact-Checking

Multimodal Claim Extraction for Fact-Checking

Paper · 2026 · Joycelyn Teo, Rui Cao, Zhenyun Deng et al.

Summary

Claim extraction is the first stage of automated fact-checking, but existing approaches mostly handle text alone. Much social media misinformation pairs short, informal text with photos, screenshots or memes. The authors build MMCE, described as the first benchmark for multimodal claim extraction. It contains real image-text posts linked to claims written by professional fact-checkers and focuses on out-of-context images.

They also propose a three-part LLM-judged evaluation:

  • semantic alignment with the gold claim,
  • entailment, meaning faithfulness to the post,
  • decontextualization, meaning whether the claim stands alone.

Their tests show that state-of-the-art MLLMs capture surface content but often read posts too literally, missing rhetorical intent and context. To address this, they introduce MICE. It first adds structured visual descriptions, then infers the post's intent, tone, context and image context, and finally extracts claims with retrieved in-context examples. MICE helps on intent-critical posts and improves decontextualization, but it loses some entailment. Across the full dataset its gains are not consistent. The work offers a resource and a baseline method for multimodal fact-checking pipelines.

Key points

  • MMCE dataset: 732 post–claim pairs (618 train / 114 dev) drawn from AVerImaTeC, released on Hugging Face (joycelynt/MMCE). It includes a curated subset of 50 intent-critical examples.
  • MICE pipeline:
    • Azure Vision API supplies dense captions, OCR text and tags.
    • An MLLM produces a contextual breakdown of the post.
    • Claim extraction uses ICL, with demonstrations retrieved by BM25 under a leave-one-out strategy.
  • Evaluation setup: Gemini 2.5 Flash Lite acts as the judge. Image-text inputs beat text-only inputs, and ICL adds further gains. MICE raised decontextualization scores for Gemini 2.0 Flash and GPT-4o Mini.
  • Human validation: 4 expert annotators scored the outputs, and their agreement with the LLM judge was measured with Krippendorff's α. Agreement was moderate for reference-based scores (α = 0.59) and decontextualization (α = 0.54), but very low for entailment (α = 0.07).
  • Temporal leakage check: On 50 posts from July to December 2024, Gemini 2.5 Flash showed no significant gain over Gemini 2.0 Flash. This gave no clear sign that the models rely on newer pre-training knowledge.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CambridgeCambridge · United Kingdom243 works · 121,396 citations🎓Queen Mary University of LondonLondon · United Kingdom54 works · 3,542 citations🏛️Defence Science and Technology AgencySingapore · Singapore2 works · 2 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksSociology and Political Science · 577 works

🧭 Topics 1Research subjects and languages the work is about.Vision & language · 657 works

Cite

Cite as Joycelyn Teo, Rui Cao, Zhenyun Deng, Zifeng Ding, Michael Sejr Schlichtkrull and Andreas Vlachos. 2026. Multimodal Claim Extraction for Fact-Checking. The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.wassa-1.22

@inproceedings{teo2026multimodal,
  title = {{Multimodal Claim Extraction for Fact-Checking}},
  author = {Joycelyn Teo, Rui Cao, Zhenyun Deng, Zifeng Ding, Michael Sejr Schlichtkrull and Andreas Vlachos},
  booktitle = {The Proceedings for the 15th Workshop on Computational Approaches to Subjectivity, Sentiment Social Media Analysis (WASSA 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.wassa-1.22},
  url = {https://doi.org/10.18653/v1/2026.wassa-1.22}
}

References

30 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023. Multimodal automated fact-checking: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5430–5448, Singapore. Association for Computational Linguisti… find
  2. Mubashara Akhtar, Michael Schlichtkrull, and Andreas Vlachos. 2025. Ev2r: Evaluating evidence retrieval in automated fact-checking. find
  3. Firoj Alam, Julia Maria Struß, Tanmoy Chakraborty, Stefan Dietze, Salim Hafid, Katerina Korre, Arianna Muti, Preslav Nakov, Federico Ruggeri, Sebastian Schellhammer, Vinay Setty, Megha Sundriyal, Konstantin Todorov, and Venktesh V. 2025. The clef-2025 checkthat! lab: Subjectivity, fact-checking, cla… find
  4. Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Ha… — Qwen2.5-VL Technical Report
  5. Tobias Braun, Mark Rothermel, Marcus Rohrbach, and Anna Rohrbach. 2025. DEFAME: Dynamic evidence-based FAct-checking with multimodal experts. In Forty-second International Conference on Machine Learning. find
  6. System Demonstrations, pages 150–158, St. Julians, Malta. Association for Computational Linguistics. find
  7. Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2024. GPTScore: Evaluate as you desire. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 6556–6576, Mexico City, M… find
  8. Jiahui Geng, Jonathan Tonglet, and Iryna Gurevych. 2026. M4fc: a multimodal, multilingual, multicultural, multitask real-world fact-checking dataset. find
  9. Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo. 2025. A survey on llm-as-a-judge. find
  10. Rui Cao, Zifeng Ding, Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2025. Averimatec: A dataset for automatic verification of image-text claims with evidence from the web. find
  11. Naeemul Hassan, Chengkai Li, and Mark Tremayne. 2015. Detecting check-worthy factual claims in presidential debates. In Proceedings of the 24th ACM International on Conference on Information and Knowledge Management, CIKM ’15, page 1835–1838, New York, NY, USA. Association for Computing Machinery. find
  12. Gullal Singh Cheema, Sherzod Hakimov, Abdul Sittar, Eric Müller-Budack, Christian Otto, and Ralph Ewerth. 2022. MM-claims: A dataset for multimodal claim detection in social media. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 962–979, Seattle, United States. Associ… find
  13. Mahmoud Khademi, Ziyi Yang, Felipe Frujeri, and Chenguang Zhu. 2023. MM-reasoner: A multimodal knowledge-aware framework for knowledge-based visual question answering. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6571–6581, Singapore. Association for Computational… find
  14. DeepMind. 2024. Gemini 2.0 flash model card. https://modelcards.withgoogle.com/assets/ documents/gemini-2-flash.pdf. ↗
  15. K. Krippendorff. 2013. Content Analysis: An Introduction to Its Methodology. SAGE Publications. find
  16. DeepMind. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities.
  17. Zhenyun Deng, Michael Schlichtkrull, and Andreas Vlachos. 2024. Document-level claim extraction and decontextualisation for fact-checking. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11943–11954, Bangkok, Thailand. Associa… find
  18. Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-eval: NLG evaluation using gpt-4 with better human alignment. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2511–2522, Singapore. Association for Computational Lin…
  19. Dasha Metropolitansky and Jonathan Larson. 2025. Towards effective extraction and evaluation of factual claims. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6996–7045, Vienna, Austria. Association for Computational Linguist…
  20. Nicholas Dufour, Arkanath Pathak, Pouya Samangouei, Nikki Hariri, Shashi Deshetti, Andrew Dud-field, Christopher Guess, Pablo Hernández Escayola, Bobby Tran, Mevan Babakar, and Christoph Bregler. 2024. Ammeba: A large-scale survey and dataset of media-based misinformation in-the-wild. find
  21. OpenAI. 2024. Gpt-4o system card.
  22. Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RAGAs: Automated evaluation of retrieval augmented generation. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Anku Rani, Dwip Dalal, Shreya Gautam, Pankaj Gupta,…
  23. Computational Linguistics (Volume 4: Student Research Workshop), pages 97–128, Vienna, Austria. Association for Computational Linguistics. find
  24. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389. find
  25. C Spearman. 2010. The proof and measurement of association between two things. International Journal of Epidemiology, 39(5):1137–1150. find
  26. Herbert Ullrich, Tomáš Mlynář, and Jan Drchal. 2025. Claim extraction for fact-checking: Data, models, and automated metrics. find
  27. Michiel Van Der Meer, Pavel Korshunov, Sébastien Marcel, and Lonneke Van Der Plas. 2025. HintsOfTruth: A multimodal checkworthiness detection dataset with real and synthetic claims. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pa… find
  28. Haoran Wang, Aman Rangapur, Xiongxiao Xu, Yueqing Liang, Haroon Gharwi, Carl Yang, and Kai Shu. 2025. Piecing it all together: Verifying multi-hop multimodal claims. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7453–7469, Abu Dhabi, UAE. Association for Com…
  29. Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2023. End-to-end multimodal fact-checking and explanation generation: A challenging dataset and models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR… find
  30. Fengzhu Zeng, Wenqian Li, Wei Gao, and Yan Pang. 2024. Multimodal misinformation detection by learning from synthetic data with multimodal LLMs. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 10467–10484, Miami, Florida, USA. Association for Computational Linguistics… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/teo-2026-multimodal-claim-extraction-fact-checking/