Elamur Research⌕
☰
home / papers / 2024 / Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study

Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study

Paper · 2024 · Hamdy Mubarak, Hend Al-Khalifa and Khaloud Suliman Alkhalefah

Summary

The paper addresses the lack of any resource for measuring LLM hallucination in Arabic. The authors built Halwasa, a 10K-sentence dataset generated by ChatGPT (gpt-3.5-turbo-0301) and GPT-4 (gpt-4-0314), with each model contributing half.

To get topical variety, they sampled 1,000 random words from the SAMER Arabic readability lexicon. For each word, they prompted the models for ten checkable factual sentences and kept five unique ones at random. Fifty trained final-year students then labelled every sentence on five fields:

  • whether it contains a verifiable fact,
  • whether that fact is correct,
  • whether it has linguistic errors,
  • a corrected version,
  • a link to the web page used for verification.

Quality was controlled with hidden test items. About a quarter of the generated sentences turned out to be factually wrong. The authors also catalogue typical factual and linguistic error types and check whether word difficulty affects correctness; they found no clear link.

The dataset matters because it is the first Arabic factuality benchmark for LLMs. It also offers a reusable annotation design that includes human corrections and evidence links.

Key points

  • 10,000 sentences: 5,000 each from ChatGPT and GPT-4, seeded by 1,000 words from SAMER, a lexicon of 26,578 words with readability levels 1–5.
  • 50 annotators from Al-Imam University each labelled 200 sentences. Fifty author-made test questions were hidden among them, and the authors' initial agreement on those items was 94% before consolidation.
  • Roughly 25% of generated sentences are factually incorrect. Annotators marked approximate numbers as wrong, for example 9.63 s instead of 9.58 s for the 100 m record.
  • GPT-4 errors fall into recurring types: ordering or superlatives (largest vs. second largest), dates, names, and quantities such as areas and fractions.
  • Annotators cited 800+ websites as evidence. The top five account for 64% of citations, and Arabic Wikipedia alone covers 55.6%. No clear correlation was found between word readability level and fact correctness.

Interesting ideas

  • I1. Seeding generation with words drawn from a graded lexicon gives diverse prompts and lets you study whether difficulty or rarity relates to errors.
  • I2. Hiding known-answer test items inside each annotator's batch measures annotator reliability without them noticing.
  • I3. The labelling schema keeps "contains a checkable fact" separate from "the fact is correct" and from "the language is flawed", and it adds a corrected sentence plus an evidence link.
  • I4. A sentence holding several facts counts as wrong if any single fact fails, so every claim must be checked.
  • I5. The error taxonomy shows that superlatives, dates, names and numbers are where fluent models slip most. Strict numeric matching treats approximations as errors.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Probe sentences could be sampled by lexicon frequency or level to test whether parser or reader accuracy drops on rare vocabulary.
  • I2 (learn): Planting gold items inside LLM-teacher batches would act as negative controls and score each teacher tier before its silver labels are trusted.
  • I3 (world): Each world-graph assertion could carry a verifiability flag, a truth status and a provenance link, so answers can explain where a fact came from.
  • I4 (frames): Sentences could be split into atomic who-did-what frames, with each frame verified or contradicted separately rather than judging the sentence as a whole.
  • I5 (answer): Test suites could include targeted items on superlatives, dates and exact quantities, scored with strict numeric comparison.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Fields, topics and concepts

📚 Shelves 6The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksNeuroscience · 165 worksCognitive Neuroscience · 164 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksSemitic languages · 2,908 works

Cite

Cite as Hamdy Mubarak, Hend Al-Khalifa and Khaloud Suliman Alkhalefah. 2024. Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study. Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{mubarak2024halwasa,
  title = {{Halwasa: Quantify and Analyze Hallucinations in Large Language Models: Arabic as a Case Study}},
  author = {Hamdy Mubarak, Hend Al-Khalifa and Khaloud Suliman Alkhalefah},
  booktitle = {Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)},
  year = {2024},
  doi = {10.63317/4ttete762fbt},
  url = {https://doi.org/10.63317/4ttete762fbt}
}

References

17 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. % Muhamed Al Khalil, Nizar Habash, and Zhengyang Jiang. 2020. A large-scale leveled readability lexicon for standard arabic. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3053–3062.
  2. Shiqi Chen, Yiran Zhao, Jinghan Zhang, I-Chun Chern, Siyang Gao, Pengfei Liu, and Junxian He. 2023. Felm: Benchmarking factuality evaluation of large language models. find
  3. David Dale, Elena Voita, Janice Lam, Prangthip Hansanti, Christophe Ropers, Elahe Kalbassi, Cynthia Gao, Loïc Barrault, and Marta Costa-jussà. 2023. Halomi: A manually annotated benchmark for multilingual hallucination and omission detection in machine translation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 638–653. ↗
  4. Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. Chain-of-verification reduces hallucination in large language models. ↗
  5. Nuno M. Guerreiro, Duarte Alves, Jonas Waldendorf, Barry Haddow, Alexandra Birch, Pierre Colombo, and André F. T. Martins. 2023. Hallucinations in large multilingual translation models. ↗
  6. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023a. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. ↗
  7. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023b. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1–38. ↗
  8. Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen. 2023. Halueval: A large-scale hallucination evaluation benchmark for large language models.
  9. Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. 2023. Mitigating hallucination in large multi-modal models via robust instruction tuning. ↗
  10. Nick McKenna, Tianyi Li, Liang Cheng, Mohammad Javad Hosseini, Mark Johnson, and Mark Steedman. 2023. Sources of hallucination by large language models on inference tasks.
  11. Hamdy Mubarak and Kareem Darwish. 2014. Using twitter to collect a multi-dialectal corpus of arabic. In Proceedings of the EMNLP 2014 Workshop on Arabic Natural Language Processing (ANLP), pages 1–7.
  12. Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. 2023. Med-halt: Medical domain hallucination test for large language models.
  13. Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S. M Towhidul Islam Tonmoy, Aman Chadha, Amit P. Sheth, and Amitava Das. 2023a. The troubling emergence of hallucination in large language models – an extensive definition, quantification, and prescriptive remediations. ↗
  14. Vipula Rawte, Amit Sheth, and Amitava Das. 2023b. A survey of hallucination in large foundation models. ↗
  15. Neha Sengupta, Sunil Kumar Sahu, Bokang Jia, Satheesh Katipomu, Haonan Li, Fajri Koto, William Marshall, Gurpreet Gosal, Cynthia Liu, Zhiming Chen, Osama Mohammed Afzal, Samta Kamboj, Onkar Pandit, Rahul Pal, Lalit Pradhan, Zain Muhammad Mujahid, Massa Baali, Xudong Han, Sondos Mahmoud Bsharat, Alham Fikri Aji, Zhiqiang Shen, Zhengzhong Liu, Natalia Vassilieva, Joel Hestness, Andy Hock, Andrew Feldman, Jonathan Lee, Andrew Jackson, Hector Xuguang Ren, Preslav Nakov, Timothy Baldwin, and Eric Xing. 2023. Jais and jais-chat: Arabic-centric foundation and instruction-tuned open generative large language models. ↗
  16. Khaled Shaalan, Sanjeera Siddiqui, Manar Alkhatib, and Azza Abdel Monem. 2019. Challenges in arabic natural language processing. In Computational linguistics, speech and image processing for arabic language, pages 59–83. World Scientific. ↗
  17. Yuchen Yang, Houqiang Li, Yanfeng Wang, and Yu Wang. 2023. Improving the reliability of large language models by leveraging uncertainty-aware in-context learning. ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/mubarak-2024-halwasa-quantify-analyze-hallucinations-large/