Elamur Research⌕
☰
home / papers / 2026 / HalluScore: Large Language Model Hallucination Question Answering Benchmark

HalluScore: Large Language Model Hallucination Question Answering Benchmark

Summary

Arabic has few resources for measuring how often LLMs hallucinate. Existing Arabic sets are either keyword-conditioned generation tasks (Halwasa) or limited to religious content (Aftina, IslamicEval). HalluScore fills this gap with an Arabic question answering benchmark of 827 curated items. The questions cover many knowledge domains, different reasoning difficulties, historical events, culturally specific Arabic scenarios and adversarial phrasings.

The authors built the set in stages: collecting candidates, filtering them for clarity and factual soundness, keeping mainly those that reliably make models hallucinate, and then refining them by hand. Every item has a verified evidence source, an explanation of the answer and several labels. Seventeen Arabic, multilingual and reasoning-oriented LLMs were run on the benchmark. Humans labelled each answer as hallucinated, non-hallucinated or partially hallucinated.

GPT-5 and Claude hallucinate least. The other models are easily misled by adversarial wording, false presuppositions and cultural knowledge. The broader lesson is that hallucination in Arabic involves cultural competence, linguistic analysis and logical checking, not only missing facts.

Key points

  • The benchmark has 827 Arabic QA pairs. Each pair comes with ground-truth evidence, an answer explanation and labels for question type and knowledge domain. It also carries binary flags for whether it needs reasoning, is adversarial, is culturally Arabic or depends on history.
  • Human annotators judged the answers of 17 LLMs as hallucinated, non-hallucinated or partially hallucinated. "Partial" means the core answer is right but invented extra facts were added.
  • Multilingual models stumble on:
    • idioms;
    • historical terminology;
    • Arabic grammatical parsing (I'rab), for example spotting a na't sababi;
    • poetic meter, for example recognising the Al-Basit meter.
  • Arabic LLMs additionally do poorly on numerical reasoning and on pseudoscientific claims.
  • Recurring failure patterns are reality violation, cultural misunderstanding, paraphrase sensitivity and grammar-analysis errors. All evaluation is closed-book with one fixed prompt setup.

Interesting ideas

  • I1. Questions were selected by keeping the ones that consistently cause models to fail, so the benchmark concentrates on weak spots instead of easy items. — For Mova (learn): Build the evaluation set from sentences where the witness parsers disagree or where the LLM teacher tiers fail, so that progress is measured on the hard cases.
  • I2. A separate "partial hallucination" label catches answers that are correct at the core but pad themselves with invented details. — For Mova (answer): The fable and tale readers could check every extra claim in an answer against the world of the text and penalise any claim the text does not support.
  • I3. Each item carries independent binary tags (reasoning, adversarial, cultural, historical), so error rates can be broken down along several axes at once. — For Mova (answer): Tag school-test and word-problem items the same way to see whether failures cluster in reasoning items or in items needing background knowledge.
  • I4. Questions built on false presuppositions turn out to be one of the most reliable ways to trigger fabrication. — For Mova (world): Before answering, check the presuppositions of a question against the entities and states in the text world, and reject the question explicitly when a presupposition is false.
  • I5. Paraphrase sensitivity is treated as a failure pattern of its own. — For Mova (learn): Add paraphrased versions of test questions as negative controls; a correct reader should give the same answer to every variant.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2605.17007 · arXiv 2605.17007

@misc{alansari2026halluscore,
  title = {{HalluScore: Large Language Model Hallucination Question Answering Benchmark}},
  author = {Alansari, Aisha and Luqman, Hamzah},
  year = {2026},
  doi = {10.48550/arxiv.2605.17007},
  eprint = {2605.17007},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2605.17007}
}

References

14 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. J. Maynez, S. Narayan, B. Bohnet, and R. Mc-Donald, “On faithfulness and factuality in abstractive summarization,” in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 1906–1919. find
  2. L. Huang, W. Yu, W. Ma, W. Zhong, Z. Feng, H. Wang, Q. Chen, W. Peng, X. Feng, B. Qin et al., “A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions,” ACM Transactions on Information Systems, vol. 43, no. 2, pp. 1–55, 2025. find our card
  3. A. Alansari and H. Luqman, “Arahallueval: A fine-grained hallucination evaluation framework for arabic llms,” in Proceedings of The Third Arabic Natural Language Processing Conference, 2025, pp. 148–161. find
  4. Z. Ji, N. Lee, R. Frieske, T. Yu, D. Su, Y. Xu, E. Ishii, Y. J. Bang, A. Madotto, and P. Fung, “Survey of hallucination in natural language generation,” ACM computing surveys, vol. 55, no. 12, pp. 1–38, 2023. find our card
  5. S. Qi, L. Gui, Y. He, and Z. Yuan, “A survey of automatic hallucination evaluation on natural language generation,” arXiv preprint arXiv:2404.12041, 2024. ↗
  6. A. Alansari and H. Luqman, “Large language [15] S. Abdaljalil, H. Kurban, and E. Serpedin, “Halluverse25: Fine-grained multilingual benchmark models hallucination: A comprehensive survey,” dataset for llm hallucinations,” arXiv preprint arXiv preprint arXiv:2510.06265, 2025. arXiv:2503.07833, 2025. ↗
  7. M. S. Bari, Y. Alnumay, N. A. Alzahrani, N. M. Alotaibi, H. A. Alyahya, S. AlRashed, F. A. [16] M. Y. Mohammed, S. A. Ali, S. K. Ali, A. A. Majeed, and E. H. Mohamed, “Aftina: enhanc-Mirza, S. Z. Alsubaie, H. A. Alahmed, G. Aling stability and preventing hallucination in ai-abduljabbar et al., “Allam: Large language based islamic fatwa generation using llms and models for arabic and english,” arXiv preprint rag,” Neural Computing and Applications, pp. arXiv:2407.15390, 2024. 1–26, 2025. ↗ our card
  8. N. Sengupta, S. K. Sahu, B. Jia, S. Katipomu, H. Li, F. Koto, W. Marshall, G. Gosal, [17] H. Mubarak, R. Malhas, W. Mansour, A. Mohamed, M. Fawzi, M. Hawasly, T. Elsayed, K. M. C. Liu, Z. Chen et al., “Jais and jais-chat: Darwish, and W. Magdy, “Islamiceval 2025: The Arabic-centric foundation and instruction-tuned first shared task of capturing llms hallucination open generative large language models,” arXiv in islamic content,” in Proceedings of The Third preprint arXiv:2308.16149, 2023. Arabic Natural Language Processing Conference: ↗ our card
  9. F. Team, U. Abbas, M. S. Ahmad, F. Alam, Shared Tasks, 2025, pp. 480–493. E. Altinisik, E. Asgari, Y. Boshmaf, S. Boughorbel, S. Chawla, S. Chowdhury et al., “Fanar: [18] J. Li, X. Cheng, W. X. Zhao, J.-Y. Nie, and J.-R. Wen, “Halueval: A large-scale hallucination An arabic-centric multimodal generative ai plat-evaluation benchmark for large language mod-form,” arXiv preprint arXiv:2501.13944, 2025. els,” arXiv preprint arXiv:2305.11747, 2023. ↗ our card
  10. M. Mashaabi, S. Al-Khalifa, and H. Al-Khalifa, “A survey of large language models for ara- [19] S. Ramprasad, E. Ferracane, and Z. C. Lipton, “Analyzing llm behavior in dialogue sum-bic language and its dialects,” arXiv preprint marization: Unveiling circumstantial hallucina-arXiv:2410.20238, 2024. tion trends,” in Proceedings of the 62nd Annual ↗ our card
  11. A. Alzubaidi, S. Alsuwaidi, B. E. A. Bous-Meeting of the Association for Computational saha, L. AlQadi, O. Alkaabi, M. Alyafeai, Linguistics (Volume 1: Long Papers), 2024, pp. H. Alobeidli, and H. Hacid, “Evaluating ara-12 549–12 561. bic large language models: A survey of benchmarks, methods, and gaps,” arXiv preprint [20] Q. Cheng, T. Sun, W. Zhang, S. Wang, X. Liu, M. Zhang, J. He, M. Huang, Z. Yin, arXiv:2510.13430, 2025. K. Chen et al., “Evaluating hallucinations in ↗ our card
  12. A. Farghaly and K. Shaalan, “Arabic natural chinese large language models,” arXiv preprint language processing: Challenges and solutions,” arXiv:2310.03368, 2023. ACM Transactions on Asian Language Information Processing (TALIP), vol. 8, no. 4, pp. 1–22, [21] X. Liang, S. Song, S. Niu, Z. Li, F. Xiong, B. Tang, Y. Wang, D. He, C. Peng, Z. Wang 2009. et al., “Uhgeval: Benchmarking the hallucina- ↗
  13. N. Y. Habash, Introduction to Arabic natural tion of chinese large language models via uncon-language processing. Morgan & Claypool Pub-strained generation,” in Proceedings of the 62nd lishers, 2010. Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), find
  14. H. Mubarak, H. Al-Khalifa, and K. S. Alkhale-2024, pp. 5266–5293. fah, “Halwasa: Quantify and analyze hallucinations in large language models: Arabic as a [22] X. Zhang, Z. Liu, J. Wang, H. Zhang, F. Xu, J. Zhang, and X. Wan, “C-faith: A chinese case study,” in Proceedings of the 2024 Joint Infine-grained benchmark for automated halluci-ternational Conference on Computational Lin-nation evaluation,” in Proceedings of the 34th guistics, Language Resources and Evaluation ACM International Conference on Information (LREC-COLING 2024), 2024, pp. 8008–8015. and Knowledge Management, 2025, pp. 6575– [31] P. Manakul, A. Liusie, and M. Gales, “Selfcheckgpt: Zero-resource black-box hallucination de-6579. tection for generative large language models,” [23] H. Ding, L. Pang, Z. Wei, H. Shen, and in Proceedings of the 2023 Conference on Em-X. Cheng, “Retrieve only when it needs: Adap-pirical Methods in Natural Language Processing, tive retrieval augmentation for hallucination 2023, pp. 9004–9017. mitigation in large language models,” arXiv [32] J. Zhang, Z. Li, K. Das, B. Malin, and S. Kumar, preprint arXiv:2402.10612, 2024. “Sac3: reliable hallucination detection in black-[24] S. AboulEla, P. Zabihitari, N. Ibrahim, M. Af-box language models via semantic-aware cross-shar, and R. Kashef, “Exploring rag solu-check consistency,” in Findings of the Associ-tions to reduce hallucinations in llms,” in ation for Computational Linguistics: EMNLP 2025 IEEE International systems Conference 2023, 2023, pp. 15 445–15 458. (SysCon). IEEE, 2025, pp. 1–8. [33] K. Jiang, Q. Zhang, D. Guo, D. Huang, S. Zhang, Z. Wei, F. Ning, and R. Li, “Ai-[25] S. Farquhar, J. Kossen, L. Kuhn, and Y. Gal, generated news articles based on large language “Detecting hallucinations in large language models,” in Proceedings of the 2023 Interna-models using semantic entropy,” Nature, vol. tional Conference on Artificial Intelligence, Sys-630, no. 8017, pp. 625–630, 2024. tems and Network Security, 2023, pp. 82–87. [26] T. Zhang, L. Qiu, Q. Guo, C. Deng, Y. Zhang, Z. Zhang, C. Zhou, X. Wang, and L. Fu, “En- [34] M. Kim, H. Jung, and M.-W. Koo, “Self-expertise: knowledge-based instruction dataset hancing uncertainty-based hallucination detec-augmentation for a legal expert language tion with stronger focus,” in Proceedings of the model,” in Findings of the Association for Com-2023 Conference on Empirical Methods in Nat-putational Linguistics: NAACL 2024, 2024, pp. ural Language Processing, 2023, pp. 915–932. 1098–1112. [27] D. Dale, E. Voita, L. Barrault, and M. R. Costa-Jussà, “Detecting and mitigating hallucinations [35] M. Arslan, H. Ghanem, S. Munawar, and C. Cruz, “A survey on rag with llms,” Proce-in machine translation: Model internal workings dia computer science, vol. 246, pp. 3781–3790, alone do well, sentence similarity even better,” 2024. in Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Vol- [36] J. Wei, X. Wang, D. Schuurmans, M. Bosma, ume 1: Long Papers), 2023, pp. 36–50. F. Xia, E. Chi, Q. V. Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large [28] N. Nonkes, S. Agaronian, E. Kanoulas, and language models,” Advances in neural informa-R. Petcu, “Leveraging graph structures to de-tion processing systems, vol. 35, pp. 24 824– tect hallucinations in large language models,” 24 837, 2022. in Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, 2024, [37] S. Dhuliawala, M. Komeili, J. Xu, R. Raileanu, pp. 93–104. X. Li, A. Celikyilmaz, and J. Weston, “Chain-of-verification reduces hallucination in large lan-[29] L. Kong, Y. Zhang, X. Zhong, H. Fu, Y. Wang, guage models,” in Findings of the association for and H. Liu, “Halugnn: Hallucination detection computational linguistics: ACL 2024, 2024, pp. in large language models using graph neural 3563–3578. network,” Expert Systems with Applications, p. 130857, 2025. [38] M. Hu, B. He, Y. Wang, L. Li, C. Ma, and I. King, “Mitigating large language model hallu-[30] S. Dasgupta, S. Nath, A. Basu, P. Shamsolmoali, cination with faithful finetuning,” arXiv preprint and S. Das, “Hallushift: Measuring distribution arXiv:2406.11267, 2024. shifts towards hallucination detection in llms,” in 2025 International Joint Conference on Neu- [39] Y.-S. Chuang, Y. Xie, H. Luo, Y. Kim, J. Glass, and P. He, “Dola: Decoding by contrasting lay-ral Networks (IJCNN). IEEE, 2025, pp. 1–8. ers improves factuality in large language models,” arXiv preprint arXiv:2309.03883, 2023. [40] E. Durmus, H. He, and M. Diab, “Feqa: A question answering evaluation framework for faith- [48] M. F. Alghifari, M. Kartiwi, M. B. A. Zaim, fulness assessment in abstractive summariza-and D. O. D. Handayani, “Mitigating llm hal-tion,” in Proceedings of the 58th Annual Meet-lucinations in quranic content: An agentic aping of the Association for Computational Lin-proach using deployable language models,” in guistics, 2020, pp. 5055–5070. 2025 10th International Conference on Information and Communication Technology for the [41] W. Kryściński, B. McCann, C. Xiong, and Muslim World (ICT4M). IEEE, 2025, pp. 1–6. R. Socher, “Evaluating the factual consistency of abstractive text summarization,” in Proceed- [49] R. Vázquez, T. Mickus, E. Zosa, T. Vah-ings of the 2020 Conference on Empirical Meth-tola, J. Tiedemann, A. Sinha, V. Segonne, ods in Natural Language Processing (EMNLP), F. Sánchez-Vega, A. Raganato, J. Libovickỳ 2020, pp. 9332–9346. et al., “Semeval-2025 task 3: Mu-shroom, the multilingual shared task on hallucinations and [42] S. Min, K. Krishna, X. Lyu, M. Lewis, W.-t. Yih, related observable overgeneration mistakes,” P. Koh, M. Iyyer, L. Zettlemoyer, and H. Ha-arXiv preprint arXiv:2504.11975, 2025. jishirzi, “Factscore: Fine-grained atomic evaluation of factual precision in long form text gen- [50] D. Dale, E. Voita, J. Lam, P. Hansanti, C. Rop-eration,” in Proceedings of the 2023 Conference ers, E. Kalbassi, C. Gao, L. Barrault, and on Empirical Methods in Natural Language Pro-M. Costa-jussà, “Halomi: A manually anno-cessing, 2023, pp. 12 076–12 100. tated benchmark for multilingual hallucination and omission detection in machine translation,” [43] E. Doostmohammadi, O. Holmström, and in Proceedings of the 2023 Conference on Em-M. Kuhlmann, “How reliable are automatic eval-pirical Methods in Natural Language Processing, uation methods for instruction-tuned llms?” in 2023, pp. 638–653. Findings of the Association for Computational Linguistics: EMNLP 2024, 2024, pp. 6321–6336. [51] H. Zhang, S. Anjum, H. Fan, W. Zheng, Y. Huang, and Y. Feng, “Poly-fever: A multi-[44] S. Lin, J. Hilton, and O. Evans, “Truthfulqa: lingual fact verification benchmark for hallucina-Measuring how models mimic human false-tion detection in large language models,” arXiv hoods,” in Proceedings of the 60th Annual Meet-preprint arXiv:2503.16541, 2025. ing of the Association for Computational Linguistics (Volume 1: Long Papers), 2022, pp. [52] Z. Yang, P. Qi, S. Zhang, Y. Bengio, W. Cohen, 3214–3252. R. Salakhutdinov, and C. D. Manning, “Hotpotqa: A dataset for diverse, explainable multi-[45] T. Vu, M. Iyyer, X. Wang, N. Constant, J. Wei, hop question answering,” in Proceedings of the J. Wei, C. Tar, Y.-H. Sung, D. Zhou, Q. Le et al., 2018 conference on empirical methods in natu-“Freshllms: Refreshing large language models ral language processing, 2018, pp. 2369–2380. with search engine augmentation,” in Findings of the Association for Computational Linguistics [53] M. Joshi, E. Choi, D. S. Weld, and L. Zettle-ACL 2024, 2024, pp. 13 697–13 720. moyer, “Triviaqa: A large scale distantly supervised challenge dataset for reading comprehen-[46] B. Goodrich, V. Rao, P. J. Liu, and M. Saleh, sion,” in Proceedings of the 55th Annual Meeting “Assessing the factual accuracy of generated of the Association for Computational Linguistics text,” in proceedings of the 25th ACM SIGKDD (Volume 1: Long Papers), 2017, pp. 1601–1611. international conference on knowledge discovery & data mining, 2019, pp. 166–175. [54] S. Pandit, J. Xu, J. Hong, Z. Wang, T. Chen, K. Xu, and Y. Ding, “Medhallu: A comprehen-[47] A. El Ganadi, S. Aftar, L. Gagliardelli, F. Ruozzi sive benchmark for detecting medical hallucina-et al., “Generative ai for islamic texts: The eman tions in large language models,” in Proceedings framework for mitigating gpt hallucinations,” in roceedings of the 17th International Conference on Agents and Artificial Intelligence-ICAART, vol. 3, 2025, pp. 1221–1228. “Qwen3-235b-a22b-instruct-2507-fp8,” of the 2025 Conference on Empirical Methods in [64] ——, 2025, together AI model cata-Natural Language Processing, 2025, pp. 2858– log. Accessed: 2026-03-18. [Online]. 2873. Available: https://www.together.ai/models/[55] A. A. Rahman, S. Anwar, M. Usman, I. Ahmad, qwen3-235b-a22b-instruct-2507-fp8 and A. Mian, “Defan: Definitive answer dataset for llm hallucination evaluation,” Information, [65] Anthropic, “System card: Claude opus 4 and claude sonnet 4,” Anthropic, Tech. vol. 16, no. 11, p. 937, 2025. Rep., 2025, accessed: 2026-03-18. [Online]. Available: https://www-cdn.anthropic.com/[56] Naseej for Technology, “Naseej launches its 4263b940cabb546aa0e3283f35b686f4f3b2ff47. innovative arabic ai language model “noon” pdf as an open-source initiative,” Jun. 19 2023, accessed: 2025-07-02. [Online]. Available: https: [66] D. Guo, D. Yang, H. Zhang, J. Song, R. Zhang, //naseej.com/news/2023/06/ R. Xu, Q. Zhu, S. Ma, P. Wang, X. Bi et al., “Deepseek-r1: Incentivizing reasoning capabil-[57] Anthropic, “Introducing claude sonnet 4.5,” ity in llms via reinforcement learning,” arXiv 2025, accessed: 2026-03-18. [Online]. preprint arXiv:2501.12948, 2025. Available: https://www.anthropic.com/news/claude-sonnet-4-5 [67] OpenAI, “Openai o3 and o4-mini system card,” OpenAI, Tech. Rep., 2025, accessed: 2026-03-18. [58] A. Liu, B. Feng, B. Xue, B. Wang, B. Wu, C. Lu, [Online]. Available: https://cdn.openai.com/C. Zhao, C. Deng, C. Zhang, C. Ruan et al., pdf/2221c875-02dc-4789-800b-e7758f3722c1/ “Deepseek-v3 technical report,” arXiv preprint o3-and-o4-mini-system-card.pdf arXiv:2412.19437, 2024. [68] M. G. Reinecke, F. Ting, J. Savulescu, and [59] xAI, “Grok 4,” 2025, accessed: 2026-03-18. I. Singh, “The double-edged sword of anthro-[Online]. Available: https://x.ai/news/grok-4 pomorphism in llms,” in Proceedings, vol. 114, no. 1. MDPI, 2025, p. 4. [60] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, [69] C. Sypherd, W. Tang, and V. Belle, “Breaking the illusion: Revisiting llm anthropomorphism,” J. Altenschmidt, S. Altman, S. Anadkat in The 4th International Conference on Human et al., “Gpt-4 technical report,” arXiv preprint and Artificial Rationalities. Springer Nature, arXiv:2303.08774, 2023. 2025, pp. 1–19. [61] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram et al., “Openai gpt-5 system card,” arXiv preprint arXiv:2601.03267, 2025. [62] Meta AI, “Llama-4-maverick-17b-128e-instruct-fp8,” https://ai.azure.com/catalog/models/Llama-4-Maverick-17B-128E-Instruct-FP8, 2025, azure AI Foundry model catalog. Accessed: 2026-03-18. [63] Alibaba Qwen Team, “Qwen3-next-80b-a3b-instruct,” 2025, qwen official blog. Accessed: 2026-03-18. [Online]. Available: https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/alansari-2026-halluscore-large-language-model-hallucination/