Elamur Research⌕
☰
home / papers / 2026 / VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking

Paper · 2026 · Mark Rothermel, Marcus Kornmann, Marcus Rohrbach et al.

Summary

Benchmarks for automated fact-checking (AFC) are usually frozen in time. Their claims and verdicts gradually enter the web data used to pretrain LLMs, which inflates measured performance. Many are also limited in modality, language or domain, or depend on invented claims.

The authors present VeriTaS, a dynamic multimodal benchmark. It is built only from real claims already reviewed by professional fact-checkers and covers text, images and video in dozens of languages. A seven-stage automated pipeline adds new material every quarter. It rewrites claims into a normalized form, recovers the original media, and translates the varied expert rulings into a new rating scheme. Instead of one coarse label, separate claim and media properties each get an uncertainty-aware score with a written justification. This allows MSE-based evaluation, which tolerates small errors but strongly penalizes reversed verdicts.

Human evaluation confirms that the automatic labels agree closely with human judgment. Baseline tests show that current multimodal LLMs remain far from reliable fact-checkers. Their scores on claims published before their knowledge cutoff overstate their real ability, which is the core argument for leakage-resistant evaluation.

Key points

  • Scale: about 25,000 real-world claims from 104 fact-checking organizations in 54 languages, covering textual, image and video content.
  • Scoring: properties such as claim veracity, claim integrity, video authenticity and video contextualization are rated on a −1 to 1 scale. Each rating comes with a justification, and the benchmark reports 5 scores rather than a single class.
  • Leakage evidence: models perform worse on claims that appeared after their knowledge cutoff (Fig. 5 of the paper).
  • Update commitment: the authors promise quarterly extensions until at least Q4 2028; a Q1 2026 extension has already been demonstrated.
  • Cost and access: the benchmark has cost about $16.5K so far (about $660 per quarterly split), plus roughly 2.9K GPU hours on 8 NVIDIA H100s for the in-house LLAMA 4 MAVERICK. Access for researchers is gated via veritas.mai.informatik.tu-darmstadt.de.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

Cite

Cite as Mark Rothermel, Marcus Kornmann, Marcus Rohrbach and Anna Rohrbach. 2026. VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking. Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1948

@inproceedings{rothermel2026veritas,
  title = {{VeriTaS: The First Dynamic Benchmark for Multimodal Automated Fact-Checking}},
  author = {Mark Rothermel, Marcus Kornmann, Marcus Rohrbach and Anna Rohrbach},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1948},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1948}
}

References

73 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023. find
  2. Multimodal Automated Fact-Checking: A Survey. In The 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP): Findings. arXiv. find
  3. Shivangi Aneja, Chris Bregler, and Matthias Nießner. 2021. COSMOS: Catching Out-of-Context Misinformation with Self-Supervised Learning. Preprint, arXiv:2101.06278. — [arXiv:2101.06278](https://arxiv.org/abs/2101.06278) find
  4. Anthropic. 2025. Claude Sonnet 4.5 System Card. System Card. Accessed on Jan 5, 2026. find
  5. Alessandro Bondielli, Pietro Dell’Oglio, Alessandro Lenci, Francesco Marcelloni, and Lucia Passaro. 2024. Dataset for multimodal fake news detection and verification tasks. Data in Brief, 54:110440. find
  6. Tobias Braun, Mark Rothermel, Marcus Rohrbach, and Anna Rohrbach. 2025. DEFAME: Dynamic Evidence-based FAct-checking with Multimodal Experts. In Proceedings of the 42nd International Conference on Machine Learning, pages 5383–5417. PMLR. find
  7. Grégoire Burel, Martino Mensio, Youri Peskine, Raphael Troncy, Paolo Papotti, and Harith Alani. 2025. CimpleKG: A Continuously Updated Knowledge Graph on Misinformation, Factors and Fact-Checks. In The Semantic Web – ISWC 2024, pages 97–114, Cham. Springer Nature Switzerland. find
  8. Rui Cao, Zifeng Ding, Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2025. AVer-ImaTeC: A Dataset for Automatic Verification of Image-Text Claims with Evidence from the Web. In NeurIPS 2025. find
  9. Megha Chakraborty, Khushbu Pahwa, Anku Rani, Shreyas Chatterjee, Dwip Dalal, Harshit Dave, Ritvik G, Preethi Gurumurthy, Adarsh Mahor, Samahriti Mukherjee, Aditya Pakala, Ishan Paul, Janvita Reddy, Arghya Sarkar, Kinjal Sensharma, Aman Chadha, Amit Sheth, and Amitava Das. 2023. FACTIFY3M: A benchmar… find
  10. Gullal Singh Cheema, Sherzod Hakimov, Abdul Sittar, Eric Müller-Budack, Christian Otto, and Ralph Ewerth. 2022. MM-Claims: A Dataset for Multimodal Claim Detection in Social Media. In Findings of the Association for Computational Linguistics: NAACL 2022, pages 962–979, Seattle, United States. Associ… find
  11. Alphaeus Dmonte, Roland Oruche, Marcos Zampieri, Prasad Calyam, and Isabelle Augenstein. 2025. Claim Verification in the Age of Large Language Models: A Survey. Preprint, arXiv:2408.14317.
  12. Nicholas Dufour, Arkanath Pathak, Pouya Samangouei, Nikki Hariri, Shashi Deshetti, Andrew Dud-field, Christopher Guess, Pablo Hernández Escay-ola, Bobby Tran, Mevan Babakar, and Christoph Bregler. 2024. AMMeBa: A Large-Scale Survey and Dataset of Media-Based Misinformation In-The-Wild. Preprint, arX… — [arXiv:2405.11697](https://arxiv.org/abs/2405.11697) find
  13. Mark Elsner, Grace Atkinson, and Saadia Zahidi. 2025. find
  14. Global Risks Report 2025. Technical report, World Economic Forum. find
  15. Farima Fatahi Bayat, Lechen Zhang, Sheza Munir, and Lu Wang. 2025. FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 33090–33110, Vienna, Austr…
  16. Jiahui Geng, Jonathan Tonglet, and Iryna Gurevych. 2025. M4FC: A Multimodal, Multilingual, Multicultural, Multitask Real-World Fact-Checking Dataset. Preprint, arXiv:2510.23508. find
  17. Max Glockner, Yufang Hou, and Iryna Gurevych. 2022. Missing Counter-Evidence Renders NLP Fact-Checking Unrealistic for Misinformation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 5916– 5936, Abu Dhabi, United Arab Emirates. Association for Computa… find
  18. Max Glockner, Ieva Staliūnaitė, James Thorne, Gisela Vallejo, Andreas Vlachos, and Iryna Gurevych. 2024. AmbiFC: Fact-Checking Ambiguous Claims with Evidence. Transactions of the Association for Computational Linguistics, pages 1–18.
  19. Google. 2025a. Gemini 3 Pro Model Card. Model Card. Accessed on Jan 5, 2026. find
  20. Google. 2025b. Introducing Nano Banana Pro. https://blog.google/technology/ai/nano-banana-pro/. Accessed on Jan 5, 2026. ↗
  21. Rainer Greifeneder, Mariela Jaffe, Eryn Newman, and Norbert Schwarz. 2020. The Psychology of Fake News: Accepting, Sharing, and Correcting Misinformation, 1 edition. Routledge, London. find
  22. M. Hameleers, T. E. Powell, T. G. L. A. Van Der Meer, and L. Bos. 2020. A picture paints a thousand lies? The effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Political Communication, 37. 42076 11 find
  23. Naeemul Hassan, Bill Adair, J. Hamilton, Chengkai Li, Mark Tremayne, Jun Yang, and Cong Yu. 2015. The Quest to Automate Fact-Checking. find
  24. Xuming Hu, Zhijiang Guo, Junzhe Chen, Lijie Wen, and Philip S. Yu. 2023. MR2: A Benchmark for Multimodal Retrieval-Augmented Rumor Detection in Social Media. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’23, pages 2901–2912… find
  25. Yiqiao Jin, Minje Choi, Gaurav Verma, Jindong Wang, and Srijan Kumar. 2024. MM-SOC: Benchmarking Multimodal Large Language Models in Social Media Platforms. In Findings of the Association for Computational Linguistics: ACL 2024, pages 6192–6210, Bangkok, Thailand. Association for Computational Lingu… find
  26. Prerana Khatiwada, Qile Wang, Kenneth E. Barner, and Matthew Louis Mauriello. 2025. Towards a Multimodal Multi-Label Election-Context Repository for Classifying Misinformation. Workshop Proceedings of the 19th International AAAI Conference on Web and Social Media, 2025:26. find
  27. Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams. 2021. Dyna… find
  28. Haiyang Li, Yaxiong Wang, Shengeng Tang, Lianwei Wu, Lechao Cheng, and Zhun Zhong. 2025a. Towards Unified Multimodal Misinformation Detection in Social Media: A Benchmark Dataset and Baseline. Preprint, arXiv:2509.25991. — [arXiv:2509.25991](https://arxiv.org/abs/2509.25991) find
  29. Haonan Li, Xudong Han, Hao Wang, Yuxia Wang, Minghan Wang, Rui Xing, Yilin Geng, Zenan Zhai, Preslav Nakov, and Timothy Baldwin. 2025b. Loki: An Open-Source Tool for Fact Verification. In Proceedings of the 31st International Conference on Computational Linguistics: System Demonstrations, pages 28–3… find
  30. Yiyi Li and Ying Xie. 2020. Is a Picture Worth a Thousand Words? An Empirical Study of Image Content and Social Media Engagement. Journal of Marketing Research, 57(1):1–19. find
  31. Xuannan Liu, Zekun Li, Peipei Li, Huaibo Huang, Shuhan Xia, Xing Cui, Linzhi Huang, Weihong Deng, and Zhaofeng He. 2025. MMFakeBench: A Mixed-Source Multimodal Misinformation Detection Benchmark for LVLMs. Preprint, arXiv:2406.08772. — [arXiv:2406.08772](https://arxiv.org/abs/2406.08772) find
  32. Grace Luo, Trevor Darrell, and Anna Rohrbach. 2021. NewsCLIPpings: Automatic Generation of Out-of-Context Multimodal Media. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 6801–6817, Online and Punta Cana, Dominican Republic. Association for Computati… find
  33. Meta AI. 2025. The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation. https://ai.meta.com/blog/llama-4-multimodal-intelligence/. Accessed on Jan 5, 2026. ↗
  34. Shreyash Mishra, S. Suryavardan, Amrit Bhaskar, P. Chopra, Aishwarya N. Reganti, Parth Patwa, Amitava Das, Tanmoy Chakraborty, A. Sheth, and Asif Ekbal. 2022. FACTIFY: A Multi-Modal Fact Verification Dataset. In DE-FACTIFY@AAAI. find
  35. Kai Nakamura, Sharon Levy, and William Yang Wang. 2020. Fakeddit: A New Multimodal Benchmark Dataset for Fine-grained Fake News Detection. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 6149–6157, Marseille, France. European Language Resources Association. find
  36. Eryn J. Newman, Maryanne Garry, Daniel M. Bernstein, Justin Kantner, and D. Stephen Lindsay. 2012. Non-probative photographs (or words) inflate truthiness. Psychonomic Bulletin & Review, 19(5):969–974. find
  37. Dan S. Nielsen and Ryan McConville. 2022. MuMiN: A Large-Scale Multilingual Multimodal Fact-Checked Misinformation Social Network Dataset. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’22, pages 3141–3153, New York, NY, USA… find
  38. Kaipeng Niu, Danni Xu, Bingjian Yang, Wenxuan Liu, and Zheng Wang. 2025. Pioneering Explainable Video Fact-Checking with a New Dataset and Multi-role Multimodal Model Approach. Proceedings of the AAAI Conference on Artificial Intelligence, 39(27):28276–28283. find
  39. OpenAI. 2025a. GPT-5 System Card. System Card, OpenAI. Accessed on Jan 5, 2026. find
  40. OpenAI. 2025b. Sora https://openai.com/index/sora-2/. Jan 5, 2026. is here. Accessed on Stefanos-Iordanis Papadopoulos, Ivana Beňová, Sebastian Kula, Michal Gregor, George Karantaidis, Tomáš Javůrek, Marián Šimko, and Symeon Papadopoulos. 2025. Multimodal and Multilingual Fact-Checked Article Retr… ↗
  41. Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, and Panagiotis C. Petran-tonakis. 2024. VERITE: A Robust benchmark for multimodal misinformation detection accounting for 42077 12 unimodal bias. International Journal of Multimedia Information Retrieval, 13(1):4. find
  42. Dina Pisarevskaya and Arkaitz Zubiaga. 2025. Zero-shot and Few-shot Learning with Instruction-following LLMs for Claim Matching in Automated Fact-checking. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9721–9736, Abu Dhabi, UAE. Association for Computational…
  43. Aman Rangapur, Haoran Wang, Ling Jian, and Kai Shu. 2025. Fin-Fact: A Benchmark Dataset for Multimodal Financial Fact-Checking and Explanation Generation. In Companion Proceedings of the ACM on Web Conference 2025, WWW ’25, pages 785–788, New York, NY, USA. Association for Computing Machinery. find
  44. Shaina Raza, Ashmal Vayani, Aditya Jain, Aravind Narayanan, Vahid Reza Khazaie, S. Bashir, Elham Dolatabadi, Gias Uddin, Christos Emmanouilidis, Rizwan Qureshi, and Mubarak Shah. 2025. VLD-Bench: Vision Language Models Disinformation Detection Benchmark. Information Fusion Journal. find
  45. Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos. 2024. The Automated Verification of Textual Claims (AVeriTeC) Shared Task. FEVER Workshop at EM… find
  46. Michael Schlichtkrull, Zhijiang Guo, and Andreas Vlachos. 2023. AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Web. Advances in Neural Information Processing Systems, 36:65128–65167.
  47. Rui Shao, Tianxing Wu, and Ziwei Liu. 2023. find
  48. Detecting and Grounding Multi-Modal Media Manipulation. 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6904–6913. find
  49. Ali Shirali, Rediet Abebe, and Moritz Hardt. 2023. A Theory of Dynamic Benchmarks. In ICLR 2023. find
  50. Anastasios Skoularikis, Stefanos-Iordanis Papadopoulos, Symeon Papadopoulos, and Panagiotis C. Petran-tonakis. 2025. ’Humor, Art, or Misinformation?’: A Multimodal Dataset for Intent-Aware Synthetic Image Detection. In Proceedings of the 2nd International Workshop on Diffusion of Harmful Content on… find
  51. Ieva Staliunaite and Andreas Vlachos. 2025. Dis2Dis: Explaining Ambiguity in Fact-Checking. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 246–267, Albuquerque, New Mexico. Association for Computational Linguistics. find
  52. Ruiran Su, Jiasheng Si, Zhijiang Guo, and Janet B. Pierrehumbert. 2025. ClimateViz: A Benchmark for Statistical Reasoning and Fact Verification on Scientific Charts. Preprint, arXiv:2506.08700. find
  53. Megha Sundriyal, Tanmoy Chakraborty, and Preslav Nakov. 2023. From Chaos to Clarity: Claim Normalization to Empower Fact-Checking. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 6594–6609, Singapore. Association for Computational Linguistics. find
  54. Chia-Wei Tang, Ting-Chih Chen, Kiet A. Nguyen, Kazi Sajeed Mehrab, Alvi Md Ishmam, and Chris Thomas. 2024. M3D: MultiModal MultiDocument Fine-Grained Inconsistency Detection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 22270–22293, Miami, Florida,… find
  55. Andon Tchechmedjiev, Pavlos Fafalios, Katarina Boland, Malo Gasquet, Matthäus Zloch, Benjamin Zapilko, Stefan Dietze, and Konstantin Todorov. 2019. ClaimsKG: A Knowledge Graph of Fact-Checked Claims. In The Semantic Web – ISWC 2019: 18th International Semantic Web Conference, Auckland, New Zealand,… find
  56. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: A Large-scale Dataset for Fact Extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vo…
  57. Jonathan Tonglet, Marie-Francine Moens, and Iryna Gurevych. 2024. “Image, Tell me your story!” Predicting the original meta-context of visual misinformation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7845–7864, Miami, Florida, USA. Association f… find
  58. Jonathan Tonglet, Gabriel Thiem, and Iryna Gurevych. 2025. COVE: COntext and VEracity prediction for out-of-context images. Preprint, arXiv:2502.01194. find
  59. Michiel van der Meer, Pavel Korshunov, Sébastien Marcel, and Lonneke van der Plas. 2025. HintsOfT-ruth: A Multimodal Checkworthiness Detection Dataset with Real and Synthetic Claims. Preprint, arXiv:2502.11753. — [arXiv:2502.11753](https://arxiv.org/abs/2502.11753) find
  60. Haoran Wang, Aman Rangapur, Xiongxiao Xu, Yueqing Liang, Haroon Gharwi, Carl Yang, and Kai Shu. 2025. Piecing It All Together: Verifying Multi-Hop Multimodal Claims. In Proceedings of the 31st International Conference on Computational Linguistics, pages 7453–7469, Abu Dhabi, UAE. Association for Com…
  61. William Yang Wang. 2017. “Liar, Liar Pants on Fire”: A New Benchmark Dataset for Fake News Detection. In Proceedings of the 55th Annual Meeting of the 42078 13
  62. Association for Computational Linguistics (Volume 2: Short Papers), pages 422–426, Vancouver, Canada. Association for Computational Linguistics. find
  63. Yuping Wang, Fatemeh Tahmasbi, Jeremy Blackburn, Barry Bradlyn, Emiliano De Cristofaro, David Magerman, Savvas Zannettou, and Gianluca Stringhini. 2021. Understanding the Use of Fauxtography on Social Media. Proceedings of the International AAAI Conference on Web and Social Media, 15:776–786. find
  64. Yuzhuo Xiao, Zeyu Han, Yuhan Wang, and Huaizu Jiang. 2025. XFacta: Contemporary, Real-World Dataset and Evaluation for Multimodal Misinformation Detection with Multimodal LLMs. Preprint, arXiv:2508.09999. — [arXiv:2508.09999](https://arxiv.org/abs/2508.09999) find
  65. Communities. In Proceedings of the Internet Measurement Conference 2018, IMC ’18, pages 188–202, New York, NY, USA. Association for Computing Machinery. find
  66. Dimitrina Zlatkova, Preslav Nakov, and Ivan Koychev. 2019. Fact-Checking Meets Fauxtography: Verifying Claims About Images. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNL… find
  67. Qingzheng Xu, Huiqiang Chen, Heming Du, Hu Zhang, Szymon Łukasik, Tianqing Zhu, and Xin Yu. 2024a. M3A: A multimodal misinformation dataset for media authenticity analysis. Computer Vision and Image Understanding, 249:104205. find
  68. Qingzheng Xu, Heming Du, Huiqiang Chen, Bo Liu, and Xin Yu. 2024b. MMOOC: A Multimodal Misinformation Dataset for Out-of-Context News Analysis. In Information Security and Privacy, pages 444–459, Singapore. Springer Nature. find
  69. Qingzheng Xu, Heming Du, Szymon Łukasik, Tianqing Zhu, Sen Wang, and Xin Yu. 2025. MDAM3: A Misinformation Detection and Analysis Framework for Multitype Multimodal Media. In Proceedings of the ACM on Web Conference 2025, WWW ’25, pages 5285–5296, New York, NY, USA. Association for Computing Machine… find
  70. Bingjian Yang, Danni Xu, Kaipeng Niu, Wenxuan Liu, Zheng Wang, and Mohan Kankanhalli. 2025a. A New Dataset and Benchmark for Grounding Multimodal Misinformation. In Proceedings of the 33rd ACM International Conference on Multimedia, MM ’25, pages 12571–12577, New York, NY, USA. Association for Compu… find
  71. Shuo Yang, Yuqin Dai, Guoqing Wang, Xinran Zheng, Jinfeng Xu, Jinze Li, Zhenzhe Ying, Weiqiang Wang, and Edith C. H. Ngai. 2025b. RealFactBench: A Benchmark for Evaluating Large Language Models in Real-World Fact-Checking. In Proceedings of the 33rd ACM International Conference on Multimedia, MM ’25… find
  72. Barry Menglong Yao, Aditya Shah, Lichao Sun, Jin-Hee Cho, and Lifu Huang. 2023. End-to-End Multimodal Fact-Checking and Explanation Generation: A Challenging Dataset and Models. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages… find
  73. Savvas Zannettou, Tristan Caulfield, Jeremy Blackburn, Emiliano De Cristofaro, Michael Sirivianos, Gianluca Stringhini, and Guillermo Suarez-Tangil. 2018. On the Origins of Memes by Means of Fringe Web 42079 14 find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/rothermel-2026-veritas-first-dynamic-benchmark-multimodal/