Elamur Research⌕
☰
home / papers / 2026 / Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection

Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection

Paper · 2026 · Ivan Vykopal, Antonia Karamolegkou, Jaroslav Kopčan et al.

Summary

The paper examines two fairness problems in multilingual Previously Fact-Checked Claim Detection (PFCD), the task of matching a social media post to fact-checks that already exist. The first is language bias, where models perform better on English and other high-resource languages. The second is a newly formalized retrieval bias, where retrievers return some debunks far more often than their relevance justifies.

For language bias, the authors test six open-source LLMs on AMC-16K. They translate the task instructions into every input language and compare the results with English-only prompting, using four prompting strategies. For retrieval bias, they use multilingual embedding models and count how often each fact-checked claim is retrieved across different posts.

Most models do best in English and lose accuracy on lower-resource languages. Few-shot prompts and translated task descriptions reduce part of this gap. A small set of generic, popular or malformed claims dominates retrieval results, which inflates scores. LLM-based re-ranking partly reduces this effect.

The work is mainly diagnostic. It gives a measurement basis for building more equitable fact-checking tools for underrepresented languages.

Key points

  • Data: AMC-16K, a subset of MultiClaim (which covers 39 languages), with 16,000 post–claim pairs. Half (8K) are monolingual across 20 languages, and half (8K) are cross-lingual across 20 language combinations.
  • Models: Qwen3 8B/14B, Llama3.1 Instruct 8B/70B and Gemma3 IT 12B/27B. "Thinking" mode was enabled only for Qwen3 8B, for compute reasons.
  • Prompting: Instructions were translated into the 20 dataset languages plus Chinese. Four strategies were tested: zero-shot, zero-shot with task description, few-shot and chain-of-thought.
  • Retrieval bias findings: Noisy corpus entries such as URLs and truncated snippets are retrieved repeatedly because of surface similarity. Topic skew, possibly inherited from MultiClaim, also contributes.
  • Code and data: Available at github.com/kinit-sk/llms-biases. The authors list as limitations that the study covers only open-source models and does not test code-switching.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of CopenhagenCopenhagen · Denmark115 works · 15,292 citations🎓Brno University of TechnologyBrno · Czechia11 works · 32,326 citations🤝Kempelen Institute of Intelligent TechnologiesBratislava · Slovakia11 works · 82 citations

 Fields, topics and concepts

📚 Shelves 7The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksSocial Sciences · 1,772 worksInformation Systems · 1,437 worksSociology and Political Science · 577 works

🧭 Topics 2Research subjects and languages the work is about.Multilingual & cross-lingual NLP · 828 worksInformation retrieval & search · 473 works

Cite

Cite as Ivan Vykopal, Antonia Karamolegkou, Jaroslav Kopčan, Qiwei Peng, Tomáš Javůrek, Michal Gregor and Marian Simko. 2026. Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection. Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.eacl-long.240

@inproceedings{vykopal2026investigating,
  title = {{Investigating Language and Retrieval Bias in Multilingual Previously Fact-Checked Claim Detection}},
  author = {Ivan Vykopal, Antonia Karamolegkou, Jaroslav Kopčan, Qiwei Peng, Tomáš Javůrek, Michal Gregor and Marian Simko},
  booktitle = {Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.eacl-long.240},
  url = {https://doi.org/10.18653/v1/2026.eacl-long.240}
}

References

30 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher, and Edward Malthouse. 2021. User-centered evaluation of popularity bias in recommender systems. In Proceedings of the 29th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’21, page 119–129, New York, NY, USA.… find
  2. Canyu Chen and Kai Shu. 2024. Combating misinformation in the age of llms: Opportunities and challenges. AI Magazine, 45(3):354–368. find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. find
  4. Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, and 542 o… — The Llama 3 Herd of Models
  5. Michael A. Hedderich, Lukas Lange, Heike Adel, Jannik Strötgen, and Dietrich Klakow. 2021. A survey on recent approaches for natural language processing in low-resource scenarios. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu…
  6. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in LLMs: Improving multilingual capability by cross-lingual-thought prompting. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 12365– 1239…
  7. Pratik Joshi, Sebastin Santy, Amar Budhiraja, Kalika Bali, and Monojit Choudhury. 2020. The state and fate of linguistic diversity and inclusion in the NLP world. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6282–6293, Online. Association for Comp…
  8. Márton Kardos, Kenneth C. Enevoldsen, Jan Kostkan, Ross Deans Kristensen-McLachlan, and Roberta Rocca. 2025. Turftopic: Topic modelling with contextual representations from sentence transformers. Journal of Open Source Software, 10(111):8183. find
  9. Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Onli…
  10. Ashkan Kazemi, Kiran Garimella, Devin Gaffney, and Scott A. Hale. 2021. Claim matching beyond English to scale global fact-checking. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (… find
  11. Anastasiia Klimashevskaia, Dietmar Jannach, Mehdi Elahi, and Christoph Trattner. 2024. A survey on popularity bias in recommender systems. User Modeling and User-Adapted Interaction, 34(5):1777–1834. find
  12. Chunxi Liu, Qiaochu Zhang, Xiaohui Zhang, Kritika Singh, Yatharth Saraf, and Geoffrey Zweig. 2020. Multilingual graphemic hybrid ASR with massive data augmentation. In Proceedings of the 1st Joint Workshop on Spoken Language Technologies for Under-resourced languages (SLTU) and Collaboration and Com… find
  13. Preslav Nakov and 1 others. 2022. Overview of the CLEF-2022 checkthat! lab. In Working Notes of CLEF 2022-Conference and Labs of the Evaluation Forum. find
  14. Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vincent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. 2022. Large dual encoders are generalizable retrievers. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 9844–…
  15. Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podrouzek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. 2025a. SemEval-2025 task 7: Multilingual and crosslingual fact-checked claim retrieval. In Proceedings of the 19th International Workshop on Seman… find
  16. Qiwei Peng, Robert Moro, Michal Gregor, Ivan Srba, Simon Ostermann, Marian Simko, Juraj Podroužek, Matúš Mesarčík, Jaroslav Kopčan, and Anders Søgaard. 2025b. SemEval-2025 task 7: Multilingual and crosslingual fact-checked claim retrieval. In Proceedings of the 19th International Workshop on Seman… find
  17. Matúš Pikuliak, Ivan Srba, Robert Moro, Timo Hro-madka, Timotej Smoleň, Martin Melišek, Ivan Vykopal, Jakub Simko, Juraj Podroužek, and Maria Bielikova. 2023a. Multilingual previously fact-checked claim retrieval. In Proceedings of the 2023 find
  18. Conference on Empirical Methods in Natural Language Processing, pages 16477–16500, Singapore. Association for Computational Linguistics. Matúš Pikuliak and 1 others. 2023b. Multilingual previously fact-checked claim retrieval. In Proceedings of the 2023 Conference on Empirical Methods in Natural Lan… find
  19. Dina Pisarevskaya and Arkaitz Zubiaga. 2025. Zero-shot and few-shot learning with instruction-following LLMs for claim matching in automated fact-checking. In Proceedings of the 31st International Conference on Computational Linguistics, pages 9721–9736, Abu Dhabi, UAE. Association for Computational…
  20. Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982–3992,…
  21. Shaden Shaar, Nikolay Babulkov, Giovanni Da San Martino, and Preslav Nakov. 2020. That is a known lie: Detecting previously fact-checked claims. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3607– 3618, Online. Association for Computational Linguis…
  22. Shaden Shaar, Nikola Georgiev, Firoj Alam, Giovanni Da San Martino, Aisha Mohamed, and Preslav Nakov. 2022. Assisting the human fact-checkers: Detecting all previously fact-checked claims in a document. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2069–2080, Abu Dh… find
  23. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, Louis Rouillard, Thomas Mesnard, Geoffrey Cideron, Jean bastien Grill, Sabela Ramos, Edouard Yvinec, Michelle Casbon, Etienne Pot, Ivo Penche… find
  24. Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, Tatiana Anikina, Michal Gregor, and Marian Simko. 2025. Large language models for multilingual previously fact-checked claim detection. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 15741–15765, Suzhou, China. Associati… find
  25. Ivan Vykopal, Matúš Pikuliak, Simon Ostermann, and Marián Šimko. 2024. Generative Large Language Models in Automated Fact-Checking: A Survey. Preprint, arXiv:2407.02351. — [arXiv:2407.02351](https://arxiv.org/abs/2407.02351) find
  26. Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. 2024. Multilingual e5 text embeddings: A technical report. arXiv preprint arXiv:2402.05672.
  27. Yuemei Xu, Ling Hu, Jiayi Zhao, Zihan Qiu, Kexin Xu, Yuqi Ye, and Hanwen Gu. 2025. A survey on multilingual large language models: corpora, alignment, and bias. Front. Comput. Sci., 19(11). find
  28. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 41 others. 2025. Qwen3 technical report. Preprint, arXiv:2505.09388.
  29. Jinrui Yang, Fan Jiang, and Timothy Baldwin. 2024. Language bias in multilingual information retrieval: The nature of the beast and mitigation methods. In Proceedings of the Fourth Workshop on Multilingual Representation Learning (MRL 2024), pages 280– 292, Miami, Florida, USA. Association for Compu… find
  30. Liwen Zheng, Chaozhuo Li, Xi Zhang, Yu-Ming Shang, Feiran Huang, and Haoran Jia. 2024. Evidence retrieval is almost all you need for fact verification. In Findings of the Association for Computational Linguistics: ACL 2024, pages 9274–9281, Bangkok, Thailand. Association for Computational Linguistic… find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vykopal-2026-investigating-language-retrieval-bias-multilingual/