Elamur Research⌕
☰
home / papers / 2026 / mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?

Paper · 2026 · Yerzhan Sapenov and Jaromir Savelka

Summary

The paper asks whether frontier LLMs reason equally well in many languages. It also asks whether machine-translated test items can stand in for professional translations. The authors build mmPISA-bench from publicly released OECD PISA items, which are carefully localized and validated so that scores can be compared across countries. They hand-picked 25 text-only multiple-choice questions and collected the official versions in 43 languages. They then created matching Google Translate versions, giving 2,150 data points in total. GPT-5.1 and Claude models were tested at several reasoning-effort settings, with five runs per configuration.

Both systems answered well in every language, roughly matching the accuracy of 15-year-old students, though some languages scored lower than others. Machine-translated items did not lower accuracy compared with the official translations, so synthetic translation looks usable for scaling up multilingual evaluation. The authors also found that some languages are both costlier and less accurate. This happens because of tokenization overhead and longer reasoning traces. They therefore argue that multilingual evaluations should report cost alongside accuracy.

Key points

  • Dataset composition: 11 mathematics items come from PISA 2022 and 14 reading items from PISA 2018. Each item carries an official PISA difficulty level: 10 at levels 1–2, 6 at levels 3–4 and 9 at levels 5–6. The data is released on GitHub (ysapenov/mmPISA-bench).
  • Scale of the experiments: the authors made 107,500 API calls. GPT-5.1-2025-11-13 ran at five effort configurations, including a double-prompt variant without reasoning. Claude Opus-4-5-20251101 covered the higher effort levels, and Haiku-4-5-20251001 approximated the no-reasoning setting.
  • Claude's costliest language: Thai had the largest token premium for Claude, 2.71× for input and 2.01× for output relative to English. It cost about 2.22× more than English and scored 5.1 percentage points lower.
  • GPT's token premiums: for GPT, Greek had the largest input premium (1.84) and Icelandic the largest output premium (1.51). Greek cost about 1.69× as much as German, with accuracy 5.3 percentage points lower.
  • Language switching in reasoning: Claude sometimes reasoned in a third language, for example Russian when the prompt was in Kazakh. This suggests a latent pivot language and shows the value of native-speaker inspection. Note that Ukrainian is not among the 43 covered languages.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States539 works · 179,533 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputer Vision and Pattern Recognition · 2,869 works

🧭 Topics 1Research subjects and languages the work is about.Language models & pretraining · 3,320 works

Cite

Cite as Yerzhan Sapenov and Jaromir Savelka. 2026. mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?. Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mellm-1.16

@inproceedings{sapenov2026mmpisa,
  title = {{mmPISA-bench: Do LLMs Reason Equally Well Across 43 Languages?}},
  author = {Yerzhan Sapenov and Jaromir Savelka},
  booktitle = {Proceedings of the 1st Workshop on Multilinguality in the Era of Large Language Models (MeLLM 2026)},
  year = {2026},
  doi = {10.18653/v1/2026.mellm-1.16},
  url = {https://doi.org/10.18653/v1/2026.mellm-1.16}
}

References

29 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David R. Mortensen, Noah A. Smith, and Yulia Tsvetkov. 2023. Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, page…
  2. Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2024. BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer. In Proceedings of the 2024 Conference of the North American Chapter…
  3. Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2024. The Belebele Benchmark: a Parallel Reading Comprehension Dataset in 122 Language Variants. In Proceedings of the 62nd Annual M…
  4. Mehmet Başaran, Ömer Faruk Vural, and Cennet Tandırcı. 2025. Assessing AI in Educational Evaluation: A Comprehensive Analysis of ChatGPT’s Performance on PISA Reading Skills. Technology, Knowledge and Learning. find
  5. Luel Hagos Beyene, Vivek Verma, Min Ma, Jesujoba O. Alabi, Fabian David Schmidt, Joyce Nakatumba-Nabende, and David Ifeoluwa Adelani. 2025. mSTEB: Massively Multilingual Evaluation of LLMs on Speech and Text Tasks. In Proceedings of the 2025 IEEE Automatic Speech Recognition and Understanding Worksh… find
  6. Alexis Conneau, Min Ma, Simran Khanuja, Yu Zhang, Vera Axelrod, Siddharth Dalmia, Jason Riesa, Clara Rivera, and Ankur Bapna. 2022. FLEURS: Few-shot Learning Evaluation of Universal Representations of Speech. In Proceedings of the 2022 IEEE Spoken Language Technology Workshop (SLT), pages 798–805. I…
  7. Marta Ruiz Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, and… find
  8. Akash Ghosh, Debayan Datta, Sriparna Saha, and Chirag Agarwal. 2025. A Survey of Multilingual Reasoning in Language Models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP). find
  9. Gracjan Góral, Emilia Wiśnios, Piotr Sankowski, and Paweł Budzianowski. 2025. Wait, that’s not an option: LLMs Robustness with Incorrect Multiple-Choice Options. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1495–1515. find
  10. Patrick Haller, Fabio Barth, Jonas Golde, Georg Rehm, and Alan Akbik. 2025. PISA-Bench: The PISA Index as a Multilingual and Multimodal Metric for the Evaluation of Vision-Language Models. arXiv preprint arXiv:2510.24792. — [arXiv:2510.24792](https://arxiv.org/abs/2510.24792) find
  11. Zixian Huang, Wenhao Zhu, Gong Cheng, Lei Li, and Fei Yuan. 2024. MindMerger: Efficiently Boosting LLM Reasoning in non-English Languages. In Advances in Neural Information Processing Systems (NeurIPS). find
  12. Amir Hossein Kargaran, Ayyoob Imani, François Yvon, and Hinrich Schütze. 2023. GlotLID: Language Identification for Low-Resource Languages. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP).
  13. Yaniv Leviathan, Matan Kalman, and Yossi Matias. 2025. Prompt Repetition Improves Non-Reasoning LLMs. arXiv preprint arXiv:2512.14982. find
  14. Hengyu Luo, Zihao Li, Joseph Attieh, Sawal Devkota, Ona de Gibert, Xu Huang, Shaoxiong Ji, Peiqin Lin, Bhavani Sai Praneeth Varma Mantina, Ananda Sreenidhi, Raúl Vázquez, Mengjie Wang, Samea Yusofi, Fei Yuan, and Jörg Tiedemann. 2025. GlotEval: A Test Suite for Massively Multilingual Evaluation of L… find
  15. OECD. 2023a. PISA 2022 Assessment and Analytical Framework. PISA. OECD Publishing, Paris. find
  16. OECD. 2023b. PISA 2022 Results (Volume I): The State of Learning and Equity in Education. PISA. OECD Publishing, Paris. find
  17. OECD. 2024. PISA 2022 Technical Report. PISA. OECD Publishing, Paris. find
  18. OECD. 2026. PISA Test Examples: Mathematics and Reading. https://www.oecd.org/en/ about/programmes/pisa/pisa-test. html. Accessed: December 5, 2025. ↗
  19. Aleksandr Petrov, Emanuele La Malfa, Philip H. S. Torr, and Adel Bibi. 2023. Language Model Tokenizers Introduce Unfairness Between Languages. In Proceedings of the 37th International Conference on Neural Information Processing Systems (NeurIPS 2023), pages 36963–36990. Curran Associates Inc.
  20. David Pomerenke, Jonas Nothnagel, and Simon Ostermann. 2025. The AI Language Proficiency Monitor – Tracking the Progress of LLMs on Multilingual Benchmarks. arXiv preprint arXiv:2507.08538. — [arXiv:2507.08538](https://arxiv.org/abs/2507.08538) find
  21. M3Exam: A Multilingual, Multimodal, Multilevel Benchmark for Examining Large Language Models. In Advances in Neural Information Processing Systems 36 (NeurIPS). find
  22. Jirui Qi, Shan Chen, Zidi Xiong, Raquel Fernández, Danielle S. Bitterman, and Arianna Bisazza. 2025. When Models Reason in Your Language: Controlling Thinking Language Comes at the Cost of Accuracy. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 20279– 20296. find
  23. Weixiang Zhao, Jiahe Guo, Yang Deng, Tongtong Wu, Wenxuan Zhang, Yulin Hu, Xingyu Sui, Yanyan Zhao, Wanxiang Che, Bing Qin, Tat-Seng Chua, and Ting Liu. 2025. When Less Language is More: Language-Reasoning Disentanglement Makes LLMs Better Multilingual Reasoners. In Advances in Neural Information Pr… find
  24. Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S. Yu. 2025. A survey of multilingual large language models. Patterns, 6(1):101118. find
  25. A Limitations Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language Models are Multilingual Chain-of-Thought Reasoners. In Proceedings of the 11th International Conferen…
  26. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David Ifeoluwa Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Raymond Ng, Shayne Longpre, Alice Oh, Andre F. T. Martins, Leshem Choshen, Daphne Ippolito, Enzo Ferrante, Marzieh Fadaee, Beyza Er… — Global MMLU: Understanding and Addressing Cultural and Linguistic Biases in Multilingual Evaluation
  27. Kyosuke Takami. 2023. Exploring ChatGPT Performance on PISA Multiple Choice Sample Questions Comparing English and Japanese Expression. In Proceedings of the Workshop on The Applications of Generative Artificial Intelligence (GAI) in Education, 31st International Conference on Computers in Education… find
  28. Minghao Wu, Weixuan Wang, Sinuo Liu, Huifeng Yin, Xintong Wang, Yu Zhao, Chenyang Lyu, Longyue Wang, Weihua Luo, and Kaifu Zhang. 2025. The Bitter Lesson Learned from 2,000+ Multilingual Benchmarks. arXiv preprint arXiv:2504.15521. — [arXiv:2504.15521](https://arxiv.org/abs/2504.15521) find
  29. Weihao Xuan, Rui Yang, Heli Qi, Qingcheng Zeng, Yunze Xiao, Aosong Feng, Dairui Liu, Yun Xing, Junjue Wang, Fan Gao, Jinghui Lu, Yuang Jiang, Huitao Li, Xin Li, Kunyu Yu, Ruihai Dong, Shangding Gu, Yuekang Li, Xiaofei Xie, and 13 others. 2025. MMLU-ProX: A Multilingual Benchmark for Advanced Large L…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sapenov-2026-mmpisa-bench-do-llms-reason/