Elamur Research⌕
☰
home / papers / 2026 / Selective Multimodal Retrieval for Automated Verification of Image–Text Claims

Selective Multimodal Retrieval for Automated Verification of Image–Text Claims

Paper · 2026 · Yoana Tsoneva, Paul-Conrad Feig, Jiaao Li et al.

Summary

Text-based fact-checking pipelines often fail when misinformation pairs a statement with an image. Fully multimodal systems handle such claims but are expensive and can behave unpredictably. For the FEVER-9 shared task, the authors built a pipeline that decides deliberately where visual information is used, instead of passing it through every component. Qwen3-VL-8B-Instruct, used zero-shot, splits each AVerImaTeC claim into three to five verification questions and tags each one as text-related or image-related. Text questions go to a hybrid retriever: BM25 candidate selection followed by gte-base dense reranking. Image questions use SigLIP embeddings that combine the query with the referenced picture. The same model then assigns a verdict and writes a brief justification based on the collected textual evidence. The system ranked fourth overall and clearly outperformed the official baseline on evidence, verdict and justification metrics. The results suggest that targeted, modality-aware retrieval can give solid image–text verification without heavy multimodal fusion, which matters for teams with limited compute.

Key points

  • Data: AVerImaTeC contains 1,297 real-world image–text claims, each assigned one of four labels: Refuted, Supported, Conflicting Evidence, or Not Enough Evidence. It comes with a knowledge store of scraped web text and images.
  • Text retrieval:
    • The store is cut into four-sentence blocks.
    • BM25 keeps the top 2,000 blocks.
    • gte-base cosine similarity selects the final 10.
  • Image retrieval:
    • google/siglip-base-patch16-384 embeds chunks of at most 60 tokens.
    • The query vector averages the text and image embeddings.
    • Each URL is scored by the mean of its three best chunk similarities, and the system returns 5 URLs with 3 chunks each.
  • Results vs. baseline:
    • Evidence Score: 0.2703 vs. 0.1707 (about 58% relative gain).
    • Verdict Accuracy: 0.2557 vs. 0.1136.
    • Justification Score: 0.198 vs. 0.1322.
    • Overall rank: 4th in FEVER-9.
  • Limitations and code:
    • The system uses one small model with zero-shot prompting only and was developed on the validation set.
    • Forcing questions into text/image types made them less fluent.
    • The effect of images on verdicts was not tested.
    • Code: GitHub XplaiNLP/FEVER-9-AVerImaTeC-XxP.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🔬German Research Centre for Artificial IntelligenceKaiserslautern · Germany77 works · 29,479 citations🎓Technische Universität BerlinBerlin · Germany22 works · 957 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksComputer Vision and Pattern Recognition · 2,869 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 657 worksInformation retrieval & search · 473 works

Cite

Cite as Yoana Tsoneva, Paul-Conrad Feig, Jiaao Li, Veronika Solopova, Neda Foroutan, Arthur Hilbert and Vera Schmitt. 2026. Selective Multimodal Retrieval for Automated Verification of Image–Text Claims. Proceedings of the Ninth Fact Extraction and VERification Workshop (FEVER).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.fever-1.10

@inproceedings{tsoneva2026selective,
  title = {{Selective Multimodal Retrieval for Automated Verification of Image–Text Claims}},
  author = {Yoana Tsoneva, Paul-Conrad Feig, Jiaao Li, Veronika Solopova, Neda Foroutan, Arthur Hilbert and Vera Schmitt},
  booktitle = {Proceedings of the Ninth Fact Extraction and VERification Workshop (FEVER)},
  year = {2026},
  doi = {10.18653/v1/2026.fever-1.10},
  url = {https://doi.org/10.18653/v1/2026.fever-1.10}
}

References

15 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Mubashara Akhtar, Michael Schlichtkrull, Zhijiang Guo, Oana Cocarascu, Elena Simperl, and Andreas Vlachos. 2023. Multimodal automated fact-checking: A survey. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 5430–5448, Singapore. Association for Computational Linguisti… find
  2. Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, Wenbin Ge, Zhifang Guo, Qidong Huang, Jie Huang, Fei Huang, Binyuan Hui, Shutong Jiang, Zhaohai Li, Mingsheng Li, and 45 others. 2025. Qwen3-vl technical report. Preprint, ar…
  3. Rui Cao, Zifeng Ding, Zhijiang Guo, Michael Schlichtkrull, and Andreas Vlachos. 2025. Averimatec: A dataset for automatic verification of image-text claims with evidence from the web. arXiv preprint arXiv:2505.17978. — [arXiv:2505.17978](https://arxiv.org/abs/2505.17978) find
  4. Recep Firat Cekinel, Pinar Karagoz, and Çagrı Cöltekin. 2025. Multimodal fact-checking with vision language models: A probing classifier based solution with embedding strategies. In Proceedings of COLING. find
  5. Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollar, and C. Lawrence Zitnick. 2015. Microsoft coco captions: Data collection and evaluation server. Preprint, arXiv:1504.00325.
  6. Michael Hameleers, Thomas E. Powell, Toni G.L.A. Van Der Meer, and Lieke Bos. 2020. A picture paints a thousand lies? the effects and mechanisms of multimodal disinformation and rebuttals disseminated via social media. Political Communication, 37(2):281– 301. find
  7. Thomas Hetzner, Veronika Solopova, Vera Schmitt, and Dorothea Kolossa. 2025. Integrating video, text, and images for multimodal disinformation detection. In Proceedings of the 4th ACM International Workshop on Multimedia AI against Disinformation (MAD’25). ACM. find
  8. Stephen Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3:333–389. find
  9. Michael Schlichtkrull, Yulong Chen, Chenxi Whitehouse, Zhenyun Deng, Mubashara Akhtar, Rami Aly, Zhijiang Guo, Christos Christodoulopoulos, Oana Cocarascu, Arpit Mittal, James Thorne, and Andreas Vlachos. 2024. The automated verification of textual claims (AVeriTeC) shared task. In Proceedings of th… find
  10. Sahar Tahmasebi, Eric Müller-Budack, and Ralph Ewerth. 2024. Multimodal misinformation detection using large vision-language models. In Proceedings of the 33rd ACM International Conference on Information and Knowledge Management, CIKM ’24, page 2189–2199, New York, NY, USA. Association for Computing… find
  11. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. Fever: a large-scale dataset for fact extraction and verification. In Proceedings of NAACL-HLT.
  12. Max Upravitelev, Premtim Sahitaj, Arthur Hilbert, Veronika Solopova, Jing Yang, Nils Feldhus, Tatiana Anikina, Simon Ostermann, and Vera Schmitt. 2025. Exploring semantic filtering heuristics for efficient claim verification. In Proceedings of the Eighth Fact Extraction and VERification Workshop (FE… find
  13. Shengkang Wang, Hongzhan Lin, Ziyang Luo, Zhen Ye, Guang Chen, and Jing Ma. 2025. Mfc-bench: Benchmarking multimodal fact-checking with large vision-language models. In Proceedings of the Building Trust Workshop at ICLR. ArXiv:2406.11288. — [arXiv:2406.11288](https://arxiv.org/abs/2406.11288) find
  14. Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975–11986.
  15. Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. 2024. Long-clip: Unlocking the long-text capability of clip. In European conference on computer vision, pages 310–325. Springer. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tsoneva-2026-selective-multimodal-retrieval-automated-verification/