Elamur Research⌕
☰
home / papers / 2026 / Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants

Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants

Summary

LLMs are often asked about local customs, food, history and etiquette, but they are rarely tested on culturally specific Arabic content, especially in dialects. Most Arabic benchmarks also use multiple-choice formats, which let models score well by guessing or by favouring certain option labels. The authors started from PalmX-GC, an MSA multiple-choice set on Arab culture. With GPT-4.1 they translated every item into four dialects and English, then rewrote the items as free-answer questions, so all six varieties stay aligned item by item. Native speakers post-edited and checked much of the test portion.

They evaluated open, Arabic-centric and closed models in both formats. They also fine-tuned ALLaM-7B with and without generated reasoning chains. Three findings stand out:

  • Dialects score below MSA.
  • Arabic-centric models do well on multiple choice but lose that edge on open answers.
  • Reasoning-chain training raises judge-rated correctness while lowering n-gram overlap.

The released resource, ArabicCulturalQA, appears to be the first parallel dialectal open-ended Arabic cultural QA benchmark.

Key points

  • The source data is PalmX-GC: MSA, 22 Arab countries, split into 2,000 train, 500 dev and 2,000 test MCQs. It is released as QCRI/ArabicCulturalQA.

  • The varieties are MSA, Egyptian, Levantine, Gulf, Maghrebi and English. GPT-4.1 did both translation and MCQ→OEQ conversion, with prompts that required meaning to stay the same.

  • On OEQ, GPT-5 reached the best BERTScore F1 (0.55), with GPT-4.1 close behind (0.54). Qwen2.5-7B beat the other open models, Arabic-centric ones included.

  • Results for ALLaM-7B (averages):

    SettingJudge scoreF1Rouge-L
    Base4.940.480.17
    Fine-tuned5.870.720.49
    Fine-tuned with CoT6.010.670.42

  • The reported experiments ran on the automatically generated data, before human post-editing. The scores may therefore understate the quality of the released version.

Interesting ideas

  • I1. When turning a multiple-choice item into an open question, the wrong options are shown to the converter only as context and then removed, and items that only make sense with visible alternatives are discarded; this keeps open questions faithful to the verified original instead of freshly invented. — For Mova (answer): School-course tests with options could be converted the same way into open questions for Mova's readers, while skipping items like "which of the following" that cannot stand alone.
  • I2. Keeping every question parallel across varieties, with English as a shared anchor, lets the authors separate language-handling failures from missing knowledge. — For Mova (answer): Asking the same fable question in several phrasings or registers would show whether a wrong answer comes from parsing/frames or from the world model.
  • I3. Running the identical item set in both a choice format and a free-answer format exposes how much of a model's score comes from the format rather than from knowing the answer. — For Mova (learn): This could serve as a negative control in Mova's evaluation, where a large gap between the two formats signals option-exploiting shortcuts.
  • I4. Reporting overlap metrics next to a judge score revealed a systematic disagreement: short but correct answers get penalised by F1/Rouge-L, and a manual look at low-scoring cases explained it. — For Mova (learn): Mova's evaluation should pair lexical metrics with a semantic check and audit cases where they diverge, rather than trusting a single score.
  • I5. Training on generated reasoning chains improved judged correctness but changed answer length and style, so the authors argue rationale length and task type decide whether chains help. — For Mova (learn): When LLM teacher tiers supply explanations, Mova could test short versus long rationales separately instead of assuming longer explanations always help.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cited by

3 papers of our library cite it (of the 140884 whose bibliographies we read); 1 of them have a page here, 2 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.63317/2smjp2wega4e

@inproceedings{bhatti2026beyond,
  title = {{Beyond MCQ: An Open-Ended Arabic Cultural QA Benchmark with Dialect Variants}},
  author = {Bhatti, Hunzalah Hassan and Alam, Firoj},
  booktitle = {Proceedings of the Fifteenth Language Resources and Evaluation Conference},
  year = {2026},
  doi = {10.63317/2smjp2wega4e},
  url = {https://doi.org/10.63317/2smjp2wega4e}
}

References

40 references, 15 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ahmed Abdelali, Hamdy Mubarak, Shammur Chowdhury, Maram Hasanain, Basel Mousi, Sabri Boughorbel, Samir Abdaljalil, Yassine El Kheir, Daniel Izham, Fahim Dalvi, Majd Hawasly, Nizi Nazar, Youssef Elshahawy, Ahmed Ali, Nadir Durrani, Natasa Milic-Frayling, and Firoj Alam. 2024. LAraBench: Benchmarking Arabic AI with large language models. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages 487– 520, St. Julian’s, Malta. Association for Computational Linguistics. find our card
  2. Firoj Alam, Md Arid Hasan, Sahinur Rahman Laskar, Mucahid Kutlu, and Shammur Absar Chowdhury. 2025a. NativQA Framework: Enabling llms with native, local, and everyday knowledge. arXiv preprint arXiv:2504.05995. Firoj Alam, Ali Ezzat Shahroor, Md. ↗
  3. Arid Hasan, Zien Sheikh Ali, Hunzalah Hassan Bhatti, Mohamed Bayan Kmainasi, Shammur Absar Chowdhury, Basel Mousi, Fahim Dalvi, Nadir Durrani, and Natasa Milic-Frayling. 2025b. EverydayMMQA: A multilingual and multimodal framework for culturally grounded spoken visual qa. arXiv preprint arXiv:2510.06371. ↗ our card
  4. Fakhraddin Alwajih, Abdellah El Mekki, Samar Mohamed Magdy, AbdelRahim A. Elmadany, Omer Nacar, El Moatez Billah Nagoudi, Reem Abdel-Salam, Hanin Atwany, Youssef Nafea, Abdulfattah Mohammed Yahya, Rahaf Alhamouri, Hamzah A. Alsayadi, Hiba Zayed, Sara Shatnawi, Serry Sibaee, Yasir Ech-chammakhy, Walid Al-Dhabyani, Marwa Mohamed Ali, Imen Jarraya, Ahmed Oumar El-Shangiti, Aisha Alraeesi, Mohammed Anwar AL-Ghrawi, Abdulrahman S. Al-Batati, Elgizouli Mohamed, Noha Taha Elgindi, Muhammed Saeed, Houdaifa Atou, Issam Ait Yahia, Abdelhak Bouayad, Mohammed Machrouh, Amal Makouar, Dania Alkawi, Mukhtar Mohamed, Safaa Taher Abdelfadil, Amine Ziad Ounnoughene, Anfel Rouabhia, Rwaa Assi, Ahmed Sorkatti, Mohamedou Cheikh Tourad, Anis Koubaa, Ismail Berrada, Mustafa Jarrar, Shady Shehata, and Muhammad Abdul-Mageed. 2025a. Palm: A culturally inclusive and linguistically diverse dataset for Arabic LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 32871–32894, Vienna, Austria. Association for Computational Linguistics. find
  5. Fakhraddin Alwajih, Abdellah El Mekki, Hamdy Mubarak, Majd Hawasly, Abubakr Mohamed, and Muhammad Abdul-Mageed. 2025b. PalmX 2025: The first shared task on benchmarking llms on arabic and islamic culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), Suzhou, China. Association for Computational Linguistics. Co-located with EMNLP 2025, November 5–9. find
  6. Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. AraBERT: Transformer-based model for Arabic language understanding. In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9–15, Marseille, France. European Language Resource Association. find our card
  7. Ge Bai, Jie Liu, Xingyuan Bu, Yancheng He, Jiaheng Liu, Zhanhui Zhou, Zhuoran Lin, Wenbo Su, Tiezheng Ge, Bo Zheng, and Wanli Ouyang. 2024. MT-bench-101: A fine-grained benchmark for evaluating large language models in multi-turn dialogues. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7421– 7454, Bangkok, Thailand. Association for Computational Linguistics. M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Dr. find our card
  8. Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al-Twairesh, Areeb Alowisheq, and Haidar Khan. 2025. ALLam: Large language models for arabic and english. In The Thirteenth International Conference on Learning Representations. find our card
  9. Hunzalah Hassan Bhatti, Youssef Ahmed, Md Arid Hasan, and Firoj Alam. 2025. Cultranai at palmx 2025: Data augmentation for cultural knowledge representation. find
  10. Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023. Sparks of artificial general intelligence: Early experiments with GPT-4. Technical report, Microsoft Research. find our card
  11. Nikhil Chandak, Shashwat Goel, Ameya Prabhu, Moritz Hardt, and Jonas Geiping. 2025. Answer matching outperforms multiple choice for language model evaluation. arXiv preprint arXiv:2507.02856. Md. ↗
  12. Arid Hasan, Maram Hasanain, Fatema Ahmad, Sahinur Rahman Laskar, Sunaya Upadhyay, Vrunda N Sukhadia, Mucahid Kutlu, Shammur Absar Chowdhury, and Firoj Alam. 2025. NativQA: Multilingual culturally-aligned natural query for LLMs. In Findings of the Association for Computational Linguistics: ACL 2025, pages 14886–14909, Vienna, Austria. Association for Computational Linguistics. find
  13. Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3. find
  14. Mohamed Bayan Kmainasi, Rakif Khan, Ali Ezzat Shahroor, Boushra Bendou, Maram Hasanain, and Firoj Alam. 2025. Native vs non-native language prompting: A comparative analysis. In Web Information Systems Engineering – WISE 2024, pages 406–420, Singapore. Springer Nature Singapore. find
  15. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35:22199–22213. ↗ our card
  16. Cheng Li, Damien Teney, Linyi Yang, Qingsong Wen, Xing Xie, and Jindong Wang. 2024a. CulturePark: Boosting cross-cultural understanding in large language models. In Advances in Neural Information Processing Systems, volume 37, pages 65183–65216. find
  17. Wangyue Li, Liangzhi Li, Tong Xiang, Xiao Liu, Wei Deng, and Noa Garcia. 2024b. Can multiple-choice questions really be useful in detecting the abilities of llms? In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 2819–2834. find
  18. Yuetai Li, Xiang Yue, Zhangchen Xu, Fengqing Jiang, Luyao Niu, Bill Yuchen Lin, Bhaskar Ramasubramanian, and Radha Poovendran. 2025. Small models struggle to learn from strong reasoners. arXiv preprint arXiv:2502.12143. ↗
  19. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out: Proceedings of the ACL Workshop. find
  20. Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, et al. 2024. Falcon2-11b technical report. arXiv preprint arXiv:2407.14885. ↗
  21. Abdellah El Mekki, Houdaifa Atou, Omer Nacar, Shady Shehata, and Muhammad Abdul-Mageed. 2025. Nilechat: Towards linguistically diverse and culturally aware llms for local communities. arXiv preprint arXiv:2505.18383. ↗
  22. Francesco Maria Molfese, Luca Moroni, Luca Gioffré, Alessandro Scirè, Simone Conia, and Roberto Navigli. 2025. Right answer, wrong score: Uncovering the inconsistencies of llm evaluation in multiple-choice question answering. arXiv preprint arXiv:2503.14996. ↗
  23. Basel Mousi, Nadir Durrani, Fatema Ahmad, Md Arid Hasan, Maram Hasanain, Tameem Kabbani, Fahim Dalvi, Shammur Absar Chowdhury, and Firoj Alam. 2025. AraDiCE: Benchmarks for dialectal and cultural capabilities in LLMs. In Proceedings of the 31st International Conference on Computational Linguistics, pages 4186–4218, Abu Dhabi, UAE. Association for Computational Linguistics. find
  24. Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023. Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15991–16111, Toronto, Canada. Association for Computational Linguistics. find our card
  25. Aidar Myrzakhan, Sondos Mahmoud Bsharat, and Zhiqiang Shen. 2024. Open-llm-leaderboard: From multi-choice to open-style questions for llms evaluation, benchmark, and arena. arXiv preprint arXiv:2406.07545. ↗
  26. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Dimosthenis Antypas, Hsuvas Borkakoty, Eunsu Kim, Carla Perez-Almendros, Abinew Ali Ayele, et al. 2024. BLEnD: A benchmark for llms on everyday knowledge in diverse cultures and languages. In Proceedings of the 38th Conference on Neural Information Processing Systems (NeurIPS), Vancouver, Canada. find our card
  27. OpenAI. 2025. Gpt-5 technical overview. https://openai.com/research/gpt-5 . ↗
  28. Siddhesh Pawar, Junyeong Park, Jiho Jin, Arnav Arora, Junho Myung, Srishti Yadav, Faiz Ghifari Haznitrama, Inhwa Song, Alice Oh, and Isabelle Augenstein. 2025. Survey of cultural awareness in language models: Text and beyond. Computational Linguistics, pages 1–96. find
  29. Haritz Puerto, Tilek Chubakov, Xiaodan Zhu, Harish Tayyar Madabushi, and Iryna Gurevych. 2025. Fine-tuning on diverse reasoning chains drives within-inference CoT refinement in LLMs. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 3789–3808, Vienna, Austria. Association for Computational Linguistics. find
  30. Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. 2023. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 2695– 2709, Singapore. Association for Computational Linguistics. find our card
  31. Narun Raman, Taylor Lundy, and Kevin Leyton-Brown. 2025. Reasoning models are test exploiters: Rethinking multiple-choice. arXiv preprint arXiv:2507.15337. ↗
  32. Abdelrahman Sadallah, Junior Cedric Tonga, Khalid Almubarak, Saeed Almheiri, Farah Atif, Chatrine Qwaider, Karima Kadaoui, Sara Shatnawi, Yaser Alesh, and Fajri Koto. 2025. Commonsense reasoning in Arab culture. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 7695–7710, Vienna, Austria. Association for Computational Linguistics. find
  33. Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An arabic-centric multimodal generative ai platform. find our card
  34. Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. 2024. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191. ↗ our card
  35. Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023. Aligning large language models with human: A survey. arXiv preprint arXiv:2307.12966. ↗ our card
  36. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed H Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems. find our card
  37. Ping Yu, Jack Lanchantin, Tianlu Wang, Weizhe Yuan, Olga Golovneva, Ilia Kulikov, Sainbayar Sukhbaatar, Jason Weston, and Jing Xu. 2025. Cot-self-instruct: Building high-quality synthetic data for reasoning and non-reasoning tasks. arXiv preprint arXiv:2507.23751. ↗
  38. Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. 2022. STaR: Bootstrapping reasoning with reasoning. Advances in Neural Information Processing Systems, 35:15476–15488. ↗ our card
  39. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with bert. In Proceedings of ICLR 2020. find
  40. Chiwei Zhu, Benfeng Xu, An Yang, Junyang Lin, Quan Wang, Chang Zhou, and Zhendong Mao. 2025. Rationales are not silver bullets: Measuring the impact of rationales on model performance and reliability. In Findings of the Association for Computational Linguistics: ACL 2025, pages 5808–5835, Vienna, Austria. Association for Computational Linguistics. 9. Appendix 9.1. Annotation Guidelines To ensure the quality and reliability of the generated dataset, we conducted structured human annotation and post-editing using an in-house annotation platform. This appendix provides an overview of the annotation setup and summarizes the instructions provided to annotators. 9.1.1. MCQ→OEQ Annotation find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/bhatti-2026-beyond-mcq-open-ended-arabic/