Elamur Research⌕
☰
home / papers / 2025 / TLUE: A Tibetan Language Understanding Evaluation Benchmark

TLUE: A Tibetan Language Understanding Evaluation Benchmark

Paper · 2025 · Fan Gao, Cheng Huang, Yutong Liu et al.

Summary

Tibetan has more than seven million speakers, yet it is largely absent from LLM evaluation. The authors address this with TLUE, which they describe as the first large-scale benchmark of Tibetan language understanding. TLUE has two parts. Ti-MMLU tests general knowledge, and Ti-SafetyBench tests safety behaviour. Both were built by adapting the Chinese benchmarks CMMLU and SafetyBench in four steps:

  • machine translation with Claude-3.5-Sonnet
  • rule-based cleanup
  • revision by Tibetan linguists and annotators
  • expert checks of legal, medical and scientific items

The authors tested a mix of proprietary and open-source models in a zero-shot setting. These included GPT-4o, O1-Mini, Claude-3.5-Sonnet, Gemini-1.5-Flash, LlaMA-3.1, Qwen-2.5, DeepSeek-V3 and DeepSeek-R1. Most models scored at or even under the random-choice baseline. Larger models did not reliably do better. Reasoning-oriented models often failed to produce usable answers to Tibetan prompts. The work matters because it offers a reproducible yardstick for an under-served language. It also shows how far current multilingual training is from covering such languages.

Key points

  • TLUE contains 22,963 questions in total. Ti-MMLU has 11,528 multiple-choice items over 67 subjects in 5 domains, and Ti-SafetyBench has 11,435 items in 7 safety categories, such as ethics, bias, health and privacy.
  • Curation combined Claude-3.5-Sonnet translation with rule-based filtering and review by 2 Tibetan specialists plus 5 annotators. Domain experts then validated the specialised content. The data is released at github.com/Vicentvankor/TLUE.
  • All models were evaluated zero-shot with default temperature settings. Most results were near or under random chance.
  • Scaling effects were inconsistent. Qwen-2.5-7B matched or beat Qwen-2.5-72B in several knowledge domains. On safety, however, Qwen-2.5-72B clearly outperformed Qwen-2.5-32B.
  • The authors acknowledge that the benchmark includes little material on Tibetan culture and folklore. They plan to address this in a future TLUE+ extension.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of Electronic Science and Technology of ChinaChengdu · China38 works · 2,088 citations🎓University of ConnecticutStorrs · United States13 works · 357 citations

Also on the byline: Tibet University.

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 works

🧭 Topics 1Research subjects and languages the work is about.Evaluation & benchmarks · 1,341 works

Cite

Cite as Fan Gao, Cheng Huang, Yutong Liu, Nyima Tashi, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeng Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Xiao Feng Cd, Yongbin Yu and Hao Wang. 2025. TLUE: A Tibetan Language Understanding Evaluation Benchmark. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2025.emnlp-main.1777

@inproceedings{gao2025tlue,
  title = {{TLUE: A Tibetan Language Understanding Evaluation Benchmark}},
  author = {Fan Gao, Cheng Huang, Yutong Liu, Nyima Tashi, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeng Duojie, Gadeng Luosang, Rinchen Dongrub, Dorje Tashi, Xiao Feng Cd, Yongbin Yu and Hao Wang},
  booktitle = {Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
  doi = {10.18653/v1/2025.emnlp-main.1777},
  url = {https://doi.org/10.18653/v1/2025.emnlp-main.1777}
}

References

34 references, 16 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
  2. Bo An. 2023. Prompt-based for low-resource tibetan text classification. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(8):1–13. find
  3. Anthropic. 2024. Claude 3.5 sonnet model card. find
  4. Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, pages 610–623. find
  5. Google DeepMind. 2024. Gemini 1.5 technical report. find
  6. DeepSeek-AI et al. 2024. Deepseek-v3: Technical report. arXiv preprint arXiv:2412.19437. find
  7. DeepSeek-AI et al. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  8. John Doe and Jane Smith. 2024. Private benchmarking to prevent contamination and improve comparative evaluation of llms. arXiv preprint arXiv:2403.00393. — [arXiv:2403.00393](https://arxiv.org/abs/2403.00393) find
  9. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  10. Jack FitzGerald, Ryan Cotterell, Bahadir Avci, Sebastian Ruder, Graham Neubig, Melvin Johnson, and Orhan Firat. 2022. Massive: A 51-language multilingual dataset for task-oriented semantic parsing. arXiv preprint arXiv:2204.08582. find
  11. Fan Gao, Cheng Huang, Nyima Tashi, Yutong Liu, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeg Duojie, Gadeng Luosang, Rinchen Dongrub, et al. 2025. Tibstc-cot: A multi-domain instruction dataset for chain-of-thought reasoning in language models. arXiv preprint arXiv:2508.01977. — [arXiv:2508.01977](https://arxiv.org/abs/2508.01977) find
  12. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300. and Nyima Tashi. 2024. Tibetan syllable component recognization with memristor. In 2024 International Conf…
  13. Jiajun Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. 2020. Xtreme: A massively multilingual multi-task benchmark for evaluating cross-lingual generalization. arXiv preprint arXiv:2003.11080.
  14. Yutong Liu, Feng Xiao, Ziyue Zhang, Yongbin Yu, Cheng Huang, Fan Gao, Xiangxiang Wang, Ma-bao Ban, Manping Fan, Thupten Tsering, et al. 2025a. Tispell: A semi-masked methodology for tibetan spelling correction covering multi-level error with data augmentation. arXiv preprint arXiv:2505.08037. — [arXiv:2505.08037](https://arxiv.org/abs/2505.08037) find
  15. Cheng Huang, Fan Gao, Nyima Tashi, Yutong Liu, Xiangxiang Wang, Thupten Tsering, Ban Ma-bao, Renzeg Duojie, Gadeng Luosang, Rinchen Dongrub, et al. 2025. Sun-shine: A large language model for tibetan culture. arXiv preprint arXiv:2503.18288.
  16. Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual corpora, evaluation benchmarks and pre-trained multilingual language models for Indian languages. In Findings of the Association for Computation…
  17. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212.
  18. Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric… find
  19. Yaobo Liang, Nan Duan, Yeyun Gong, Ning Wu, Daxin Guo, Weizhu Qi, and Ming Zhou. 2020. Xglue: A new benchmark dataset for cross-lingual pre-training, understanding and generation. arXiv preprint arXiv:2004.01401.
  20. Sisi Liu, Junjie Deng, Yuan Sun, and Xiaobing Zhao. 2022. Tibert: Tibetan pre-trained language model. In 2022 IEEE International Conference on Systems, Man, and Cybernetics (SMC), pages 2956–2961. IEEE. find
  21. Yutong Liu, Jiarun Shen, Yongbin Yu, Ban Ma-Bao, Thupten Tsering, Xiangxiang Wang, Cheng Huang, Yutong Liu, Ziyue Zhang, Ban Ma-bao, Yuqing Cai, Yongbin Yu, Renzeng Duojie, Xiangxiang Wang, Fan Gao, Cheng Huang, and Nyima Tashi. 2025b. Fmsd-tts: Few-shot multi-speaker multi-dialect text-to-speech sy… — [arXiv:2505.14351](https://arxiv.org/abs/2505.14351) find
  22. Hui Lv, Chi Pu, La Duo, Yan Li, Qingguo Zhou, and Jun Shen. 2025. T-llama: a tibetan large language model based on llama2. Complex & Intelligent Systems, 11(1):72. find
  23. Manuel Mager, Arturo Oncevay, Abteen Ebrahimi, John Ortega, Annette Rios, Angela Fan, Ximena Gutierrez-Vasques, Luis Chiruzzo, Gustavo Giménez-Lugo, Ricardo Ramos, Ivan Vladimir Meza Ruiz, Rolando Coto-Solano, Alexis Palmer, Elisabeth Mager-Hois, Vishrav Chaudhary, Graham Neubig, Ngoc Thang Vu, and… — Findings of the AmericasNLP 2021 Shared Task on Open Machine Translation for Indigenous Languages of the Americas
  24. National Bureau of Statistics of China. 2021. China statistical yearbook 2021. Accessed: 2023-02-07. find
  25. Wilhelmina Nekoto, Vukosi Marivate, Tshinondiwa Matsila, Timi Fasubaa, Tajudeen Kolawole, Taiwo Fagbohungbe, Solomon Oluwole Akinola, Sham-suddee Hassan Muhammad, Salomon Kabongo, Salomey Osei, et al. 2020. Participatory research for low-resourced machine translation: A case study in african languag…
  26. OpenAI. 2024. O1 system card. find
  27. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL ’02, page 311–318, USA. Association for Computational Linguistics.
  28. Chanjun Park, Hyeonwoo Kim, Dahyun Kim, SeongHwan Cho, Sanghoon Kim, Sukyung Lee, Yungi Kim, and Hwalsuk Lee. 2024. Open Ko-LLM leaderboard: Evaluating large language models in Korean with Koh5 benchmark. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Vol… find
  29. Qwen Team. 2024. Qwen2.5: A party of foundation models. find
  30. Harman Singh, Nitish Gupta, Shikhar Bharadwaj, Dinesh Tewari, and Partha Talukdar. 2024. IndicGen-Bench: A multilingual benchmark to evaluate generation capabilities of LLMs on Indic languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long… find
  31. Aarohi Srivastava et al. 2022. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615.
  32. Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.
  33. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–3…
  34. Zhexin Zhang, Leqi Lei, Lindong Wu, Rui Sun, Yongkang Huang, Chong Long, Xiao Liu, Xuanyu Lei, Jie Tang, and Minlie Huang. 2023. Safetybench: Evaluating the safety of large language models with multiple choice questions. arXiv preprint arXiv:2309.07045.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/gao-2025-tlue-tibetan-language-understanding-evaluation/