Elamur Research⌕
☰
home / papers / 2026 / FIBER: A Multilingual Evaluation Resource for Factual Inference Bias

FIBER: A Multilingual Evaluation Resource for Factual Inference Bias

Summary

Most factual-knowledge probing benchmarks cover only English and only questions with a single correct answer. FIBER (Factual Inference Bias Evaluation Resource) fills this gap with a dataset in English, Italian and Turkish. It covers single-entity topics and multi-entity topics, where one subject maps to several valid objects, such as a country's official languages. It includes three task formats: question answering, sentence completion and object-count prediction.

Models do not generate free text. Instead, they score a fixed pool of candidate answers by cumulative log-probability. The resulting ranking is evaluated with Average Precision and with a new Factual Inference Bias score. This score checks whether models favour entities tied to the country of the prompt language.

The study finds that prompt language does shift predictions, but the effect depends on the topic and is strongest for language-related topics. Turkish prompts produce more bias than Italian ones. Multi-entity questions are harder than single-entity ones, and English yields the best accuracy. These results show that prompt language is a real source of unreliability that multilingual evaluations should account for.

Key points

  • Dataset design: 8 single-entity and 8 multi-entity topics in English, Italian and Turkish, built around country- and language-related facts. Data and code are released at github.com/metunlp/fiber.
  • Bias results: 31% of topics have a Factual Inference Bias score above 0.5. Turkish prompts show higher bias than Italian prompts in 83% of topics.
  • Language results: Averaged across all settings, English reaches about 38% MAP, Italian 32% and Turkish 31%. The extracted text also contains a different breakdown (English 35%, Italian 27%, Turkish 25%), apparently from another aggregation. Llama-3.1-8B is an exception, scoring higher in Turkish than in Italian.
  • Model results: Llama-3.1-8B and Qwen-2.5-7B are the strongest models, each at about 38% MAP. Gemma-3-4B is the weakest at 26%.
  • Entity count: MAP rises as the average number of correct entities per question falls, so multi-entity questions are clearly harder.
  • Stated limitations: only 3B–8B models are tested, only three languages are covered, and the candidate answers are limited to entities included in the dataset.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/32j8kpkoevin

@inproceedings{munis2026fiber,
  title = {{FIBER: A Multilingual Evaluation Resource for Factual Inference Bias}},
  author = {Munis, Evren Ayberk and Yilmaz, Deniz and Muti, Arianna and Toraman, Cagri},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/32j8kpkoevin},
  url = {https://doi.org/10.63317/32j8kpkoevin}
}

References

33 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Abdaljalil, S., Kurban, H., and Serpedin, E. (2025). Halluverse25: Fine-grained multilingual benchmark dataset for llm hallucinations. arXiv preprint arXiv:2503.07833. — [arXiv:2503.07833](https://arxiv.org/abs/2503.07833) find
  2. Artetxe, M., Ruder, S., and Yogatama, D. (2020). On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  3. Bowserinator (2025). Periodic-table-json: A json of the entire periodic table. https://github.com/Bowserinator/Periodic-Table-JSON. ↗
  4. Cao, B., Lin, H., Han, X., Sun, L., Yan, L., Liao, M., Xue, T., and Xu, Z. (2021). Knowledgeable or educated guess? revisiting language models as knowledge bases. arXiv preprint arXiv:2106.09231. find
  5. Chataigner, C., Taïk, A., and Farnadi, G. (2024). Multilingual hallucination gaps in large language models. arXiv preprint arXiv:2410.18270. — [arXiv:2410.18270](https://arxiv.org/abs/2410.18270) find
  6. CoAStaL NLP Group (2025). mpararel: Multilingual paraphrase-based relational probes for language models. GitHub repository. Retrieved October 24, 2025. find
  7. Elazar, Y., Kassner, N., Ravfogel, S., Ravichander, A., Hovy, E., Schütze, H., and Goldberg, Y. (2021). Measuring and improving consistency https://chatgpt.com/https://gemini.google.com in pretrained language models. arXiv preprint arXiv:2102.01017. ↗
  8. Farquhar, S., Kossen, J., Kuhn, L., and Gal, Y. (2024). Detecting hallucinations in large language models using semantic entropy. Nature, 630(8017):625–630. find
  9. Fierro, C. and Søgaard, A. (2022). Factual consistency of multilingual pretrained language models. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3046–3052. find
  10. Huang, L., Yu, W., Ma, W., Zhong, W., Feng, Z., Wang, H., Chen, Q., Peng, W., Feng, X., Qin, B., and Liu, T. (2024). A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Office Information Systems, 43(2). find our card
  11. Joshi, M., Choi, E., Weld, D., and Zettlemoyer, L. (2017). Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). find our card
  12. Kassner, N., Dufter, P., and Schütze, H. (2021). Multilingual lama: Investigating knowledge in multilingual pretrained language models. arXiv preprint arXiv:2102.00894. find
  13. Kim, S. and Kim, H. J. (2025). A dual-layered evaluation of geopolitical and cultural bias in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, volume 4, pages 580– 595, Bangkok, Thailand. Association for Computational Lingui… find
  14. Kwiatkowski, T., Palomaki, J., Redfield, O., Collins, M., Parikh, A., Alberti, C., Epstein, D., Polosukhin, I., Devlin, J., Lee, K., Toutanova, K., Jones, L., Kelcey, M., Chang, M.-W., Dai, A. M., Uszkoreit, J., Le, Q., and Petrov, S. (2019). Natural questions: A benchmark for question answering res… find our card
  15. Lewis, P., Oguz, B., Rinott, R., Riedel, S., and Schwenk, H. (2020). Mlqa: Evaluating cross-lingual extractive question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. find our card
  16. Li, B., Haider, S., and Callison-Burch, C. (2024a). This land is {Your, My} land: Evaluating geopolitical bias in language models through territorial disputes. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technol… find
  17. Li, H., Tomko, M., Vasardani, M., and Baldwin, T. (2022). Multispanqa: A dataset for multi-span question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1250–1260, Seattle, Washington.… find
  18. Li, J., Chen, J., Ren, R., Cheng, X., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. (2024b). The dawn after the dark: An empirical study on factuality hallucination in large language models. arXiv preprint arXiv:2401.03205. find
  19. Lin, S., Hilton, J., and Evans, O. (2021). Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958. find our card
  20. Muti, A., Bassignana, E., Moscato, E., and Nozza, D. (2025). Probing feminist representations: A study of bias in LLMs and word embeddings. In Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025), Cagliari, Italy. AILC / Italian Association for Computational Lin… find
  21. Petroni, F., Rocktäschel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. (2019). Language models as knowledge bases? arXiv preprint arXiv:1909.01066. find our card
  22. Priola, M. P. (2025). Addressing hallucinations with rag and nmiss in italian healthcare llm chatbots. arXiv preprint arXiv:2412.04235. — [arXiv:2412.04235](https://arxiv.org/abs/2412.04235) find
  23. Qiu, Y., Ziser, Y., Korhonen, A., Ponti, E., and Cohen, S. (2023). Detecting and mitigating hallucinations in multilingual summarisation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 8914–8932, Singapore. Association for Computational Linguistics. find
  24. QS Top Universities (2025). 2026 qs world university rankings [excel file]. https://www.topuniversities.com/sites/default/files/2025-07/2026%20QS% 20World%20University%20Rankings% 201.2%20(For%20qs.com).xlsx. ↗
  25. Rai, D., Zhou, Y., Feng, S., Saparov, A., and Yao, Z. (2024). A practical review of mechanistic interpretability for transformer-based language models. arXiv preprint arXiv:2407.02646. find
  26. Taş, S., Huseyni, M. E., Özay Ezerceli, Bayraktar, R., and Terzioğlu, F. B. (2025). Turk-lettucedetect: A hallucination detection model for turkish rag applications. arXiv preprint arXiv:2509.17671. — [arXiv:2509.17671](https://arxiv.org/abs/2509.17671) find
  27. Umutlu, E. E., Cengiz, A. A., Sever, A. K., Erdem, S., Aytan, B., Tufan, B., Topraksoy, A., Darıcı, E., and Toraman, C. (2025). Evaluating the quality of benchmark datasets for low-resource languages: A case study on Turkish. In Arviv, O., Clinciu, M., Dhole, K., Dror, R., Gehrmann, S., Habba, E., I… find
  28. UNESCO World Heritage Centre (2025). World heritage list: Statistics. https://whc.unesco.org/en/list/stat/. ↗
  29. Wikipedia contributors (2025). Wikipedia: The free encyclopedia. Wikipedia. Retrieved October 23, 2025, from Wikipedia: The Free Encyclopedia. find
  30. Youssef, P., Koraş, O., Li, M., Schlötterer, J., and Seifert, C. (2023). Give me the facts! a survey on factual knowledge probing in pre-trained language models. Technical report, Findings of the Association for Computational Linguistics: EMNLP 2023. p. 15588. find
  31. Yu, L., Cao, M., Cheung, J. C. K., and Dong, Y. (2024). Mechanisms of non-factual hallucinations in language models. arXiv preprint arXiv:2403.18167. — [arXiv:2403.18167](https://arxiv.org/abs/2403.18167) find
  32. Zhao, X., Yoshinaga, N., and Oba, D. (2024). What matters in memorizing and recalling facts? multifaceted benchmarks for knowledge probing in language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 13186–13214, Miami, Florida. Association for Computational Li… find
  33. Zhu, M. (2004). Recall, precision and average precision. Technical report, Department of Statistics and Actuarial Science, University of Waterloo. find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/munis-2026-fiber-multilingual-evaluation-resource-factual/