Elamur Research⌕
☰
home / papers / 2026 / A multilingual hallucination benchmark

A multilingual hallucination benchmark

Paper · 2026 · Freja Thoresen and Dan Saattrup Smart

Summary

Most hallucination research is done in English, so it is unclear whether its conclusions hold for less-resourced languages. This paper targets faithfulness hallucinations, where an answer sounds plausible but is not supported by the given context. The authors adapt the LettuceDetect framework to the MultiWikiQA reading-comprehension dataset. They use GPT-5-mini, via RAGFactChecker, to rewrite gold answers into subtly wrong versions with character-level spans marking each change. The result is a synthetic hallucination dataset for 306 languages. Pairing correct and corrupted answers, they fine-tune token-level hallucination detectors for 30 European languages, choosing mmBERT-small after comparing it with Ettin and EuroBERT. The detectors are then used to measure hallucination in five open LLMs on English, German, Danish and Icelandic. Hallucination rises as language resources shrink, and Icelandic is worst. The smallest model, Qwen3-0.6B, fails most often, while the Cogito models generally do best. The work provides a common, openly released benchmark for comparing context-grounded hallucination across many languages.

Key points

  • Data setup:
    • Each language has 4000 training and 1000 test context–question–answer triples from MultiWikiQA.
    • Hallucination intensity is sampled from a beta distribution (mean 0.2, SD 0.15), which biases the edits toward subtle errors.
    • The injected errors are of three types: factual, temporal and numerical.
  • Detector choice: mmBERT-small (140m) beat Ettin-17m and EuroBERT-210m on both languages tested.
    • Danish: accuracy 0.8963, unsupported-token F1 0.8689.
    • German: accuracy 0.8948, unsupported-token F1 0.8627.
    • LettuceDetect gains Danish and Icelandic as newly supported languages.
  • Models evaluated: Qwen3-0.6B, Qwen3-14B, Gemma-3-12B-IT, cogito-v1-preview-qwen-32B and cogito-v1-preview-llama-70B.
  • Main results:
    • Qwen3-0.6B produces one or more hallucinated tokens in as many as 60% of its answers, peaking in Icelandic.
    • Qwen3-14B scored best on German.
    • The authors conclude that model size alone does not explain hallucination behaviour.
  • Released resources: the 306-language dataset and 30 classifiers are on HuggingFace, and the code is on GitHub. The work was funded by EU Horizon TrustLLM.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢Alexandra InstituteAarhus · Denmark2 works · 73 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Freja Thoresen and Dan Saattrup Smart. 2026. A multilingual hallucination benchmark. Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/3g94ek9qx85o

@inproceedings{thoresen2026multilingual,
  title = {{A multilingual hallucination benchmark}},
  author = {Freja Thoresen and Dan Saattrup Smart},
  booktitle = {Proceedings of the Fourth Workshop on the Role of Resources in the Age of Large Language Models (RESOURCEFUL 2026)},
  year = {2026},
  doi = {10.63317/3g94ek9qx85o},
  url = {https://doi.org/10.63317/3g94ek9qx85o}
}

References

23 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. HalluLens: LLM hallucination benchmark. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24128–24156, Vi…
  2. Nicolas Boizard, Hippolyte Gisserot-Boukhlef, Duarte M. Alves, André Martins, Ayoub Hammal, Caio Corro, Céline Hudelot, Emmanuel Malherbe, Etienne Malaboeuf, Fanny Jourdan, Gabriel Hautreux, João Alves, Kevin El-Haddad, Manuel Faysse, Maxime Peyrard, Nuno M. Guerreiro, Patrick Fernandes, Ricardo Rei… — EuroBERT: Scaling Multilingual Encoders for European Languages
  3. Deep Cogito. 2025. Cogito v1 preview. Model release. find
  4. Gemma Team. 2025. Gemma 3 technical report. find
  5. Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitel, Sumit Sahrawat, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Matias. 2022. TRUE: Re-evaluating factual consistency evaluation. In Proceedings of the 2022 Conference of the North American Chapter of the Association…
  6. Lianjin Huang, Weize Yu, Weiguang Ma, Wei Zhong, Zhen Feng, Haoran Wang, and Ting Liu. 2025. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1–55.
  7. Akos Kovacs and Gabor Recski. 2025. Lettucedetect: A hallucination detection framework for rag applications. find
  8. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. HaluEval: A large-scale hallucination evaluation benchmark for large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 6449–6464.
  9. Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. TruthfulQA: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics, pages 3214–3252.
  10. Potsawee Manakul, Adian Liusie, and Mark J. F. Gales. 2023. Selfcheckgpt: Zero-resource black-box hallucination detection for generative large language models. find
  11. Matt Marone, Oren Weller, William Fleshman, Eric Yang, Dawn Lawrie, and Benjamin Van Durme. 2025. mmbert: A modern multilingual encoder with annealed language learning.
  12. Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Wei Koh, Mohit Iyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2023. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Proceedings of the 2023 Conference on Empirical Methods in Natur…
  13. OpenAI. 2024. Gpt-4o mini: Advancing cost-efficient intelligence. Blog post. find
  14. Donghao Ru, Liang Qiu, Xiaoyang Hu, Tong Zhang, Peng Shi, Shiyu Chang, and Zheng Zhang. 2024. Ragchecker: A fine-grained framework for diagnosing retrieval-augmented generation. Advances in Neural Information Processing Systems, 37:21999–22027. find
  15. Aohan Yang, An Li, Bo Yang, Bingchao Zhang, Bin Hui, Bo Zheng, and Zheng Qiu. 2025. Qwen3 technical report.
  16. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. 9. find
  17. Language Resource References Cheng Niu and Yuanhao Wu and Juno Zhu and Siliang Xu and Kashun Shum and Randy Zhong and Juntong Song and Tong Zhang. 2024. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models.
  18. Dan Saattrup Smart. 2025. MultiWikiQA: A Reading Comprehension Benchmark in 300+ Languages.
  19. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint…
  20. James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018. FEVER: a large-scale dataset for fact extraction and VERification. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pa…
  21. Binjie Wang, Steffi Chern, Ethan Chern, and Pengfei Liu. 2024. Halu-j: Critique-based hallucination judge. arXiv preprint arXiv:2407.12943.
  22. Jason Wei, Nanyun Karina, Hyung Won Chung, Yao Jie Jiao, Stuart Papay, Aidan Glaese, and William Fedus. 2024. Measuring short-form factuality in large language models.
  23. Oren Weller, Kaleab Ricci, Matt Marone, Alexander Chaffin, Dawn Lawrie, and Benjamin Van Durme. 2025. Seq vs seq: An open suite of paired encoders and decoders. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/thoresen-2026-multilingual-hallucination-benchmark/