Elamur Research⌕
☰
home / papers / 2026 / Unmasking the Factual-Conceptual Gap in Persian Language Models

Unmasking the Factual-Conceptual Gap in Persian Language Models

Paper · 2026 · Alireza Sakhaeirad, Ali Ma'manpoosh and Arshia Hemmat

Summary

Persian NLP benchmarks mostly test linguistic skill or factual knowledge. They seldom check whether a model can apply implicit cultural norms in context, such as taarof etiquette, night-time taboos, or rituals against the evil eye. The authors built DIVANBENCH, a hand-written diagnostic set of 315 questions covering 81 Persian superstitions and customs in seven categories. It has three task types: factual MCQ, binary belief verification using matched appropriate/violating scenarios, and scenario-based MCQ requiring multi-step social inference. The paired design lets them measure whether models simply say "yes" to anything that sounds culturally themed.

They tested seven open LLMs of 7–12B parameters. Most models readily accepted correct behavior but often missed violations. Further Persian pretraining made this tendency worse. Every model scored lower on scenario reasoning than on fact recall. The authors argue that knowing cultural facts and reasoning with cultural schemas are separate skills, and that adding more monolingual data alone will not close the gap. The dataset is public on Hugging Face (divanbench/divanbench).

Key points

  • Acquiescence bias: in 6 of 7 models, accuracy on appropriate behaviors was 84–92%, but accuracy on rejecting violations was only 19–48%.
  • Persian pretraining effect: Llama 3.1-8B correctly rejected 73% of violations. Its Persian-adapted version, Dorna2-8B, rejected only 30%, a 43-point drop, while its acceptance rose by 28 points.
  • Factual–conceptual gap: scores fell by about 21% on average from factual to scenario tasks. The gap held even for the strongest model.
  • Model size: Gemma3-12B had the best factual score (87%). It also tied for the worst bias, accepting 92% of positive items but rejecting just 31% of negative ones.
  • Limitations: the study covers only 7–12B models, and the items were written manually by the authors without crowdsourced validation or inter-annotator agreement.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of OxfordOxford · United Kingdom122 works · 160,277 citations🎓École Polytechnique Fédérale de LausanneLausanne · Switzerland48 works · 7,352 citations🎓University of IsfahanIsfahan · Iran2 works

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputer Vision and Pattern Recognition · 3,486 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,894 works

Cite

Cite as Alireza Sakhaeirad, Ali Ma'manpoosh and Arshia Hemmat. 2026. Unmasking the Factual-Conceptual Gap in Persian Language Models. The Proceedings of the First Workshop on NLP and LLMs for the Iranian Language Family.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.silkroadnlp-1.1

@inproceedings{sakhaeirad2026unmasking,
  title = {{Unmasking the Factual-Conceptual Gap in Persian Language Models}},
  author = {Alireza Sakhaeirad, Ali Ma'manpoosh and Arshia Hemmat},
  booktitle = {The Proceedings of the First Workshop on NLP and LLMs for the Iranian Language Family},
  year = {2026},
  doi = {10.18653/v1/2026.silkroadnlp-1.1},
  url = {https://doi.org/10.18653/v1/2026.silkroadnlp-1.1}
}

References

25 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Nikta Gohari Sadr, Sahar Heidariasl, Karine Megerdoomian, Laleh Seyyed-Kalantari, and Ali Emami. 2025. We politely insist: Your llm must learn the persian art of taarof. Preprint, arXiv:2509.01035. find
  2. Hossein Amirkhani, Mohammad AzariJafari, Zohreh Pourjafari, Soroush Faridan-Jahromi, Zeinab Kouhkan, and Azadeh Amirak. 2021. Farstail: A persian natural language inference dataset. Preprint, arXiv:2009.08820. — [arXiv:2009.08820](https://arxiv.org/abs/2009.08820) find
  3. Shehenaz Hossain and Haithem Afli. 2025. Craft: An explanation-based framework for evaluating cultural reasoning in multilingual language models. Preprint, arXiv:2510.14014. find
  4. Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, and 1 others. 2020. Experience grounds language. Proceedings of EMNLP.
  5. Daniel Khashabi, Arman Cohan, Siamak Shakeri, Pedram Hosseini, Pouya Pezeshkpour, Graham Neubig, and 1 others. 2020. Parsinlu: A suite of language understanding challenges for persian. arXiv preprint arXiv:2012.06154.
  6. Yong Cao, Li Zhou, Seolhwa Lee, Laura Cabello, Min Chen, and Daniel Hershcovich. 2023. Assessing cross-cultural alignment between chatgpt and human societies: An empirical study. arXiv preprint arXiv:2303.17466.
  7. Kyuhee Kim and Sangah Lee. 2025. Nunchi-bench: Benchmarking language models on cultural reasoning with a focus on Korean superstition. In Findings of the Association for Computational Linguistics: ACL 2025, pages 15328–15342, Vienna, Austria. Association for Computational Linguistics. find
  8. Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, and 1 others. 2024. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology.
  9. Cohere For AI. 2024. Aya model: An instruction finetuned open-access multilingual language model. arXiv preprint arXiv:2402.07827.
  10. Yoonjoo Ko, Hyungjoo Lee, Taehwan Lee, Jaewook Jung, Jinhyeok Park, and Jaihoon Choi. 2023. From language modeling to instruction following: Understanding the behavior shift in llms after instruction tuning. arXiv preprint arXiv:2310.00492. find
  11. Xunlian Dai, Li Zhou, Benyou Wang, and Haizhou Li. 2025. From word to world: Evaluate and mitigate culture bias in llms via word association test. Preprint, arXiv:2505.18562. find
  12. Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The winograd schema challenge. In Proceedings of KR. find
  13. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  14. Erfan Moosavi Monazzah, Vahid Rahimzadeh, Yadollah Yaghoobzadeh, Azadeh Shakery, and Mohammad Taher Pilehvar. 2025. Percul: A story-driven cultural evaluation of llms in persian. Preprint, arXiv:2502.07459. find
  15. Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, and 1 others. 2024. Towards measuring the representation of subjective global opinions in language models. Preprint, arXiv:2306.16388.
  16. OpenPersian Team. 2024. Openpersian leaderboard: Benchmarking persian language models. https://huggingface.co/spaces/ OpenPersian/leaderboard. ↗
  17. Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, and 1 others. 2023. Towards measuring the representation of subjective global opinions in language models. arXiv preprint arXiv:2306.16388.
  18. PartAI Team. 2024. Dorna: Persian-optimized language models. https://huggingface.co/ PartAI/Dorna-Llama3-8B-Instruct. ↗
  19. Gemini Team. 2023. Gemini: A family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
  20. Zahra Pourbahman, Fatemeh Rajabi, Mohammadhossein Sadeghi, Omid Ghahroodi, Somaye Bakhshaei, Arash Amini, Reza Kazemi, and Mahdieh Soleymani Baghshah. 2025. Elab: Extensive llm alignment benchmark in persian language. Preprint, arXiv:2504.12553. find
  21. Omid Ghahroodi, Arshia Hemmat, Marzia Nouri, Seyed Mohammad Hadi Hosseini, Doratossadat Dastgheib, Mohammad Vali Sanian, Alireza Sahebi, Reihaneh Zohrabi, Mohammad Hossein Rohban, Ehsaneddin Asgari, and Mahdieh Soleymani Baghshah. 2025. Meena (persianmmmu): Multimodal-multilingual educational exams… find
  22. Qwen Team. 2024. Qwen2 technical report. Preprint, arXiv:2407.10671. find
  23. Omid Ghahroodi, Marzia Nouri, Mohammad Vali Sanian, Alireza Sahebi, Doratossadat Dastgheib, Ehsaneddin Asgari, Mahdieh Soleymani Baghshah, and Mohammad Hossein Rohban. 2024. Khayyam challenge (persianmmlu): Is your llm truly wise to the persian language? Preprint, arXiv:2404.06644. — [arXiv:2404.06644](https://arxiv.org/abs/2404.06644) find
  24. Mohammad Javad Ranjbar Kalahroodi, Amirhossein Sheikholselami, Sepehr Karimi, Sepideh Ranjbar Kalahroodi, Heshaam Faili, and Azadeh Shakery. 2025. Persianmedqa: Evaluating large language models on a persian-english bilingual medical question answering benchmark. Preprint, arXiv:2506.00250. find
  25. Maarten Sap, Ronan LeBras, Daniel Fried, and Yejin Choi. 2022. Neural theory-of-mind? on the limits of social intelligence in large lms. arXiv preprint arXiv:2210.13312. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sakhaeirad-2026-unmasking-factual-conceptual-gap-persian/