Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth

Summary

The paper asks whether frontier LLMs can replace native-speaker experts when grading answers about Arabic culture and dialect usage, where right and wrong depend on cultural judgment. The authors built 103 prompt–rubric pairs (70 Egyptian, 33 Iraqi; 53 Cultural, 50 Linguistic). Native-speaker SMEs wrote each pair, and each rubric combines weighted positive requirements with answer-specific penalty criteria. Three target models answered the prompts, and SMEs scored 302 responses to create the ground truth. Five frontier LLMs then graded the same responses against the rubrics, and no judge was allowed to grade output from its own provider. Judges were compared to humans with two measures: Mean Absolute Deviation for overall distance and a Signed Mean Error for direction. GPT-5.4 tracked human scores most closely with almost no bias. Most other judges were too generous. Cultural items were harder to grade than linguistic ones. The work matters because it shows where automated grading can stand in for scarce cultural experts and where it fails, mainly on implicit native-speaker judgment.

Key points

  • The dataset has 103 validated prompt–rubric pairs, and the study covers 302 SME-graded prompt–response pairs and 1,307 judge evaluations from five judges.
  • GPT-5.4 was the best judge, with MAD of 10.21 pp and Signed Error of −1.12%. Four of five judges were lenient, at +2.01% to +6.56%.
  • Cultural tasks had higher judge MAD than Linguistic tasks for every judge, by 1.83–4.78 pp. The main failure was implicit cultural reasoning, as opposed to checks that can be verified lexically.
  • Grading difficulty did not rise or fall steadily with answer quality:
    • Muse Spark, a mid-quality model, posed the least grading difficulty, with a Target MAD of 10.05 pp.
    • gemini-3.1-pro-preview was harder to grade (12.00 pp).
    • grok-4.3 was harder still (12.77 pp).
    • Muse Spark often found the correct dialect term but then added extra, error-prone material.
  • Egyptian prompts scored higher than Iraqi ones, but Egyptian SMEs graded more leniently. The authors therefore make no causal claim about model knowledge. Each sample also had only one SME, so inter-annotator agreement is unknown.

Interesting ideas

  • I1. Measure grader deviation twice, once as an absolute distance (noise) and once as a signed mean (systematic leniency or strictness), so each grader's failure type is clear. — For Mova (learn): Each witness parser or LLM teacher tier could be scored against gold with both numbers, showing whether a tier over-accepts silver labels or is just noisy.
  • I2. Ban self-grading at the provider level, not only per model version, because related models share blind spots and may favour each other. — For Mova (learn): A teacher from one model family should not validate silver data produced by a sibling. The same rule could apply to witness parsers trained on the same treebank.
  • I3. Use rubrics with fixed positive credit plus explicit penalty criteria (hallucination, irrelevant addition, misattribution). Under such rubrics, extra elaboration creates more chances to lose points, which rewards precise, minimal answers. — For Mova (answer): School-test and fable-QA readers could be scored with penalties for unsupported additions, pushing the answer step to state only what the world of the text supports.
  • I4. Split criteria into explicit, lexically checkable ones and implicit, judgment-dependent ones, and report each group separately, since automatic graders fail mostly on the latter. — For Mova (frames): Evaluation of implicature, irony and speech acts could tag each check as explicit or implicit, keeping easy surface cues from inflating pragmatics scores.
  • I5. Before comparing subsets labelled by different annotator groups, consider that the groups may grade with different strictness. Hold back causal claims until a shared calibration subset exists. — For Mova (tree): When comparing parser accuracy across treebanks or after UD dialect conversion, first check annotation-convention differences on overlapping material before blaming the parser.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Topics: Semitic languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.48550/arxiv.2607.00139 · arXiv 2607.00139

@misc{abdoli2026benchmarking,
  title = {{Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Framework with Human SME Ground Truth}},
  author = {Abdoli, Sajjad and Al-Sumaidaee, Ghassan and ElShiekh, Ahmad and Taylor, Clayton W. and Rashad, Ahmed},
  year = {2026},
  doi = {10.48550/arxiv.2607.00139},
  eprint = {2607.00139},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.48550/arxiv.2607.00139}
}

References

24 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Muhammad Farid Adilazuarda, Sagnik Mukherjee, Pradhyumna Lavania, Siddhant Singh, Alham Fikri Aji, Jacki O’Neill, Ashutosh Modi, and Monojit Choudhury. Towards measuring and modeling “culture” in LLMs: A survey. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 15763–15784, 2024. find
  2. Dhruv Agarwal et al. Fluent but foreign: Even regional LLMs lack cultural alignment. arXiv preprint arXiv:2505.21548, 2026. ↗
  3. Marwah Alwajih et al. PalmX 2025: The first shared task on benchmarking LLMs on arabic and islamic culture. In Proceedings of the Third Arabic Natural Language Processing Conference (ArabicNLP 2025), 2025. find
  4. Xiyan Fu and Wei Liu. How reliable is multilingual LLM-as-a-judge? In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11040–11053, Suzhou, China, 2025. Association for Computational Linguistics. find
  5. Lester Hayes Zhang, Smitha Milli, et al. Cultivating pluralism in algorithmic monoculture: The community alignment dataset. arXiv preprint arXiv:2507.09650, 2025. ↗
  6. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. Challenges and strategies in cross-cultural NLP. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6997–7013, 2022. find our card
  7. Jing Huang and Diyi Yang. Culturally aware natural language inference. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7591–7609, 2023. find our card
  8. Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, and Minjoon Seo. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Representations, 2024. find our card
  9. Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 2024. find our card
  10. Sunzhu Li, Jiale Zhao, Miteto Wei, Huimin Ren, Yang Zhou, Jingwen Yang, Shunyu Liu, Kaike Zhang, and Wei Chen. RubricHub: A comprehensive and highly discriminative rubric dataset via automated coarse-to-fine generation. arXiv preprint arXiv:2601.08430, 2026. Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Frame- work with Human SME 17 Ground Truth ↗
  11. Junho Myung, Nayeon Lee, Yi Zhou, Jiho Jin, Rifki Afina Putri, Alice Oh, et al. BLEnD: A benchmark for LLMs on everyday knowledge in diverse cultures and languages. In Advances in Neural Information Processing Systems, 2024. find our card
  12. Juhyun Oh, Inha Cha, Michael Saxon, Hyunseung Lim, Shaily Bhatt, and Alice Oh. Culture is everywhere: A call for intentionally cultural evaluation. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 19156–19168, 2025. find
  13. Alicia Parrish, Angelica Chen, Nikita Nangia, Vishakh Padmakumar, Jason Phang, Jana Thompson, Phu Mon Htut, and Samuel R. Bowman. BBQ: A hand-built bias benchmark for question answering. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2086–2105, 2022. find our card
  14. Zhaozhi Qian, Faroq Altam, Muhammad Alqurishi, and Riad Souissi. CamelEval: Advancing culturally aligned arabic language models and benchmarks. arXiv preprint arXiv:2409.12623, 2024. ↗ our card
  15. Abhinav Rao, Akhila Yerukola, Vishwa Shah, Katharina Reinecke, and Maarten Sap. NormAd: A framework for measuring the cultural adaptability of large language models. In Proceedings of the 2025 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 2373–2403, 2025. find
  16. Jonathan Rystrøm, Hannah Rose Kirk, and Scott Hale. Multilingual ̸ = multicultural: Evaluating gaps between multilingual capabilities and cultural alignment in LLMs. In Proceedings of the OMMM Workshop 2025, pages 74–85, 2025. find
  17. Ananya Sahu, Mehrnaz Mofakhami, Daniel D’Souza, Thomas Euyang, Julia Kreutzer, and Marzieh Fadaee. The culture funnel: You can’t align what isn’t in the data. arXiv preprint arXiv:2606.13808, 2026. ↗
  18. Weiyan Shi, Ryan Li, Yutong Zhang, Caleb Ziems, Chunhua Yu, Raya Horesh, Rogério Abreu de Paula, and Diyi Yang. CultureBank: An online community-driven knowledge base towards culturally aware language technologies. In Findings of the Association for Computational Linguistics: EMNLP 2024, 2024. find
  19. Yi Su, Dian Yu, Linfeng Song, Juntao Li, Haitao Mi, Zhaopeng Tu, Min Zhang, and Dong Yu. Crossing the reward bridge: Expanding RL with verifiable rewards across diverse domains. arXiv preprint arXiv:2503.23829, 2025. ↗
  20. Binchi Zhang et al. Mind the gap in cultural alignment: Task-aware culture management for large language models. arXiv preprint arXiv:2602.22475, 2026. ↗
  21. Jinghao Zhang et al. CultureScope: A dimensional lens for probing cultural understanding in LLMs. arXiv preprint arXiv:2509.16188, 2025. ↗
  22. Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong, Victor Veitch, Wei Wang, Yunzhong He, Bing Liu, and Lifeng Jin. Chasing the tail: Effective rubric-based reward modeling for large language model post-training. In The Thirteenth International Conference on Learning Representations, 2026. find
  23. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as-a-judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, 2023. find
  24. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. MT-Bench: A multi-turn benchmark for evaluating chat LLMs. In Advances in Neural Information Processing Systems, volume 36, 2023. Benchmarking Frontier LLMs on Arabic Cultural and Sociolinguistic Knowledge: A Cross-Evaluation Frame- work with Human SME 18 Ground Truth find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/abdoli-2026-benchmarking-frontier-llms-arabic-cultural/