Elamur Research⌕
☰
home / papers / 2026 / MonCulture-Eval: A Hierarchical Benchmark for Evaluating Mongolian Cultural Capabilities of Large Language Models across Scripts and Regions

MonCulture-Eval: A Hierarchical Benchmark for Evaluating Mongolian Cultural Capabilities of Large Language Models across Scripts and Regions

Paper · 2026 · Quulgan Minggad, Xiao Zinan and Yuan Sun

Summary

LLMs can produce fluent Mongolian, but it is unclear whether they understand the cultural reasoning behind the language. MonCulture-Eval tests this. Its scope spans Mongolian's two writing systems (Cyrillic as well as the vertical Traditional script) along with subcultures from three regions, namely Horqin, Ordos and Alxa.

Native experts built the items from offline, non-digitized archives, which keeps the material out of web-crawled pre-training data. They then cross-checked the items double-blind and aligned the two script versions by hand on the Onon platform.

The questions follow Schein's model of cultural levels and form three layers: facts, situational appropriateness and underlying values. Separate tasks cover riddles, taboos, proverbs and benedictions.

The authors evaluated GPT-5.2, Gemini-3-Pro-Preview, Claude-Sonnet-4-5, DeepSeek-v3.2 and Qwen3-Max. Two recurring weaknesses emerged:

  • Most models score lower on Traditional script than on Cyrillic.
  • Models tend to explain sacred practices in secular, outsider terms (an "Etic Bias").

Gemini-3-Pro-Preview was the most stable across scripts. Overall, translation skill does not guarantee cultural alignment. The benchmark offers a contamination-resistant baseline for evaluating culture in low-resource languages.

Key points

  • Resource: MonCulture-Eval has parallel Traditional and Cyrillic versions of every item and covers Alxa, Ordos, Horqin and pan-Mongolian domains (history, rituals). Data and code are at github.com/Ayakades/MonCulture-Eval.
  • Structure: Layer 1 (Factual) uses single- and multiple-choice questions. Layer 2 (Situational) includes an open-ended Taboos task. Layer 3 (Values) asks models to pick the best explanation from outsider (Etic), shallow-insider and insider (Emic) options. Riddles, Proverbs and Benedictions run as additional tasks.
  • "Script Gap": performance usually drops on Traditional script. Qwen3-Max, for example, loses 14.19% compared with Cyrillic.
  • Judging: Gemini-2.5-Pro is the main LLM judge, and GPT-5.2 serves as a cross-judge for the Script Alignment Consistency (SAC) analysis. Model rankings stayed stable under both judges. Agreement between human annotators and the AI judge was 72.1% under strict matching and 85.4% under tolerance-based matching.
  • Limitations: results are sensitive to prompts (persona prompts were used). The benchmark reflects static, traditional norms rather than modern urban culture. Models also produced confident hallucinations on metaphor-heavy items.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Minzu University of ChinaBeijing · China5 works · 2 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Language models & pretraining · 3,320 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Quulgan Minggad, Xiao Zinan and Yuan Sun. 2026. MonCulture-Eval: A Hierarchical Benchmark for Evaluating Mongolian Cultural Capabilities of Large Language Models across Scripts and Regions. Findings of the Association for Computational Linguistics: ACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1449

@inproceedings{minggad2026monculture,
  title = {{MonCulture-Eval: A Hierarchical Benchmark for Evaluating Mongolian Cultural Capabilities of Large Language Models across Scripts and Regions}},
  author = {Quulgan Minggad, Xiao Zinan and Yuan Sun},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1449},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1449}
}

References

29 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing Claude Sonnet 4.5. find
  2. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, page 610–623, New York, NY, USA. Associat… find
  3. Damian Blasi, Antonios Anastasopoulos, and Graham Neubig. 2022. Systematic inequalities in language technology performance across the world’s languages. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5486–5505, Dublin, Irelan…
  4. Zhengcuo Dan and Yuan Sun. 2024. Tibetanqa2.0: Dataset with unanswerable questions for tibetan machine reading comprehension. DATA INTELLI-GENCE, 6(4):1158–1167. find
  5. DeepSeek-AI. 2024. Deepseek-v3 technical report. CoRR, abs/2412.19437. find
  6. Esin Durmus, Karina Nguyen, Thomas I. Liao, Nicholas Schiefer, Amanda Askell, Anton Bakhtin, Carol Chen, Zac Hatfield-Dodds, Danny Hernandez, Nicholas Joseph, Liane Lovitt, Sam McCandlish, Orowa Sikder, Alex Tamkin, Janel Thamkul, Jared Kaplan, Jack Clark, and Deep Ganguli. 2024. Towards measuring t…
  7. Maxwell Forbes, Jena D. Hwang, Vered Shwartz, Maarten Sap, and Yejin Choi. 2020. Social chemistry 101: Learning to reason about social and moral norms. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 653–670, Online. Association for Computatio… find
  8. Gemini Team. 2025. A new era of intelligence with gemini 3. find
  9. E.T. Hall. 1976. Beyond Culture. Knopf Doubleday Publishing Group. find
  10. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net.
  11. Daniel Hershcovich, Stella Frank, Heather Lent, Miryam de Lhoneux, Mostafa Abdou, Stephanie Brandl, Emanuele Bugliarello, Laura Cabello Piqueras, Ilias Chalkidis, Ruixiang Cui, Constanza Fierro, Katerina Margatina, Phillip Rust, and Anders Søgaard. 2022. Challenges and strategies in cross-cultural N…
  12. Yuzhen Huang, Yuzhuo Bai, Zhihao Zhu, Junlei Zhang, Jinghan Zhang, Tangjun Su, Junteng Liu, Chuancheng Lv, Yikai Zhang, Jiayi Lei, Yao Fu, Maosong Sun, and Junxian He. 2023. C-eval: A multi-level multi-discipline chinese evaluation suite for foundation models. In Advances in Neural Information Proce…
  13. Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Yejin Bang, Andrea Madotto, and Pascale Fung. 2023. Survey of hallucination in natural language generation. ACM Comput. Surv., 55(12):248:1–248:38.
  14. Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large language models are zero-shot reasoners. Preprint, arXiv:2205.11916.
  15. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in indonesia: A comprehensive test on indommlu. Preprint, arXiv:2310.04928. find
  16. Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022. JGLUE: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 2957–2966, Marseille, France. European Language Resources Association.
  17. Yan Liu, Renren Jin, Tianhao Shen, and Deyi Xiong. 2025. Cmgbench: Benchmarking chinese metaphor generation for large language models. DATA INTEL-LIGENCE, 7(4):1270–1290. find
  18. Kaiwen Lu, Yating Yang, Fengyi Yang, Rui Dong, Bo Ma, Aihetamujiang Aihemaiti, Abibilla Atawulla, Lei Wang, and Xi Zhou. 2025. Low-resource language expansion and translation capacity enhancement for LLM: A study on the Uyghur. In Proceedings of the 31st International Conference on Computational Lin… find
  19. Reem I. Masoud, Ziquan Liu, Martin Ferianc, Philip Treleaven, and Miguel Rodrigues. 2024. Cultural alignment in large language models: An explanatory analysis based on hofstede’s cultural dimensions. Preprint, arXiv:2309.12342.
  20. OpenAI. 2026. OpenAI GPT-5 system card. CoRR, abs/2601.03267.
  21. Aida Ramezani and Yang Xu. 2023. Knowledge of cultural moral norms in large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 428–446, Toronto, Canada. Association for Computational Linguistics.
  22. Phillip Rust, Jonas Pfeiffer, Ivan Vulić, Sebastian Ruder, and Iryna Gurevych. 2021. How good is your tokenizer? on the monolingual performance of multilingual language models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Join…
  23. E.H. Schein. 1985. Organizational Culture and Leadership. A Joint publication in the Jossey-Bass management series and the Jossey-Bass social and behavioral science series. Jossey-Bass Publishers. find
  24. Taylor Sorensen, Jared Moore, Jillian Fisher, Mitchell Gordon, Niloofar Mireshghallah, Christopher Michael Rytting, Andre Ye, Liwei Jiang, Ximing Lu, Nouha Dziri, Tim Althoff, and Yejin Choi. 2024. A roadmap to pluralistic alignment. Preprint, arXiv:2402.05070.
  25. Zeerak Talat, Aurélie Névéol, Stella Biderman, Miruna Clinciu, Manan Dey, Shayne Longpre, Sasha Luccioni, Maraim Masoud, Margaret Mitchell, Dragomir Radev, Shanya Sharma, Arjun Subramonian, Jaesung Tae, Samson Tan, Deepak Tunuguntla, and Oskar Van Der Wal. 2022. You reap what you sow: On the challen…
  26. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. 2023. Chain-of-thought prompting elicits reasoning in large language models. Preprint, arXiv:2201.11903.
  27. Shaoyang Xu, Yongqi Leng, Linhao Yu, and Deyi Xiong. 2025. Self-pluralising culture alignment for large language models. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 - Volume 1:… find
  28. Qwen Team: An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, and 40 others. 2025. Qwen3 technical report. CoRR, abs/2505.09388.
  29. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems 36: A…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/minggad-2026-monculture-eval-hierarchical-benchmark-evaluating/