Elamur Research⌕
☰
home / papers / 2026 / MATH-IDN: A Multilingual Mathematical Problem Solving Dataset Featuring Local Languages in Indonesia

MATH-IDN: A Multilingual Mathematical Problem Solving Dataset Featuring Local Languages in Indonesia

Paper · 2026 · Xiao Xiao, Iftitahu Ni'mah, Yuyun Wabula et al.

Summary

LLMs perform well on English math benchmarks, but little is known about how they handle Indonesia's local languages, and existing multilingual math sets leave them out. The authors present MATH-IDN, a five-way parallel benchmark built from the 500 problems of MATH500 (from the MATH dataset). It covers English, Indonesian, Javanese, Sundanese, and Buginese. gpt-4.1 produced draft translations, and native-speaker annotators then scored each draft for fluency and meaning preservation on 5-point rubrics before correcting it. Seven open models under 10B parameters were tested with zero-shot chain-of-thought and system prompts in each language. The models fall into three groups: general-purpose, math-specialized, and Southeast-Asia-adapted. Beyond accuracy, the authors measure cross-lingual consistency, language-unique successes, and pivot accuracy, in which local-language inputs are first machine-translated into English or Indonesian. Every model loses accuracy in at least one local language, and Buginese is usually the hardest. Pivoting raises scores, which suggests that the main bottleneck is understanding the language rather than the mathematical reasoning itself. The resource helps make AI reasoning more equitable for Indonesian speakers.

Key points

  • Dataset construction: MATH-IDN has 500 problems in 5 parallel languages (EN, ID, JV, SU, BG). Drafts came from gpt-4.1-2025-04-14 and were refined by human translators. Data and code are at github.com/aialt/MATH-IND.
  • Models tested: LLaMA-3.1-8B-Instruct, Gemma-2-9B-IT, Qwen2.5-7B-Instruct, Qwen2.5-Math-7B-Instruct, Bakpia-V1-1.5B-Javanese, Komodo-7B-Base, and Llama-SEA-LION-v3.5-8B-R.
  • Best model: Qwen2.5-Math-7B-Instruct scored highest in English (64.6%) but fell to 50.2% in Buginese. It also had the highest Consistency Correct rate, 40.8%.
  • Unique successes: Problems solved in only one language are mostly English cases. Buginese rarely produces such unique wins, at 0.2% for all three SEA-specific models.
  • Pivot translation: Translating inputs into English helps substantially. Llama-SEA-LION-v3.5-8B-R on Buginese rises from 42.0% to 58.2%, and Qwen2.5-Math-7B-Instruct rises from 50.2% to 58.6%.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓University of LiverpoolLiverpool · United Kingdom16 works · 745 citations🎓Eindhoven University of TechnologyEindhoven · Netherlands9 works · 602 citations

Also on the byline: National Research and Innovation Agency.

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 18,614 worksArtificial Intelligence · 17,965 worksArts and Humanities · 17,450 worksLanguage and Linguistics · 17,296 worksComputational Theory and Mathematics · 321 works

🧭 Topics 2Research subjects and languages the work is about.Reasoning · 1,685 worksMultilingual & cross-lingual NLP · 882 works

Cite

Cite as Xiao Xiao, Iftitahu Ni'mah, Yuyun Wabula, Mykola Pechenizkiy and Meng Fang. 2026. MATH-IDN: A Multilingual Mathematical Problem Solving Dataset Featuring Local Languages in Indonesia. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.231

@inproceedings{xiao2026math,
  title = {{MATH-IDN: A Multilingual Mathematical Problem Solving Dataset Featuring Local Languages in Indonesia}},
  author = {Xiao Xiao, Iftitahu Ni'mah, Yuyun Wabula, Mykola Pechenizkiy and Meng Fang},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.231},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.231}
}

References

23 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
  2. Samuel Cahyawijaya, Holy Lovenia, Alham Fikri Aji, Genta Winata, Bryan Wilie, Fajri Koto, Rahmad Mahendra, Christian Wibisono, Ade Romadhony, Karissa Vincentio, Jennifer Santoso, David Moeljadi, Cahya Wirawan, Frederikus Hudi, Muhammad Satrio Wicaksono, Ivan Parmonangan, Ika Alfina, Ilham Firdausi P… find
  3. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  4. Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.
  5. Meng Fang, Xiangpeng Wan, Fei Lu, Fei Xing, and Kai Zou. 2025. Mathodyssey: Benchmarking mathematical problem-solving skills in large language models using odyssey math data. Scientific Data, 12(1):1392. find
  6. Jia Guo, Longxu Dou, Guangtao Zeng, Stanley Kok, Wei Lu, and Qian Liu. 2024. Sailcompass: Towards reproducible and robust evaluation for southeast asian languages. arXiv preprint arXiv:2412.01186. — [arXiv:2412.01186](https://arxiv.org/abs/2412.01186) find
  7. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NeurIPS Datasets an…
  8. Fajri Koto, Nurul Aisyah, Haonan Li, and Timothy Baldwin. 2023. Large language models only pass primary school exams in Indonesia: A comprehensive test on IndoMMLU. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12359–12374, Singapore. Association fo… find
  9. Huiyuan Lai and Malvina Nissim. 2024. mCoT: Multilingual instruction tuning for reasoning consistency in language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12012– 12026, Bangkok, Thailand. Association for Computa…
  10. Chaoqun Liu, Wenxuan Zhang, Jiahao Ying, Mahani Aljunied, Anh Tuan Luu, and Lidong Bing. 2025. SeaExam and SeaBench: Benchmarking LLMs with local multilingual questions in Southeast Asia. In Findings of the Association for Computational Linguistics: NAACL 2025, pages 6119–6136, Albuquerque, New Mexi… find
  11. Holy Lovenia, Rahmad Mahendra, Salsabil Maulana Akbar, Lester James V. Miranda, Jennifer Santoso, Elyanah Aco, Akhdan Fadhilah, Jonibek Mansurov, Joseph Marvin Imperial, Onno P. Kampman, Joel Ruben Antony Moniz, Muhammad Ravi Shulthan Habibi, Frederikus Hudi, Railey Montalan, Ryan Ignatius, Joanito… — SEACrowd: A Multilingual Multimodal Data Hub and Benchmark Suite for Southeast Asian Languages
  12. Wenyang Luo, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2025. Mmath: A multilingual benchmark for mathematical reasoning. arXiv preprint arXiv:2505.19126. find
  13. Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. SeaLLMs - large language models for Southeast Asia. In Proceedings…
  14. David Ong and Peerat Limkonchotiwat. 2023. SEA-LION (Southeast Asian languages in one network): A family of Southeast Asian language models. In Proceedings of the 3rd Workshop for Natural Language Processing Open Source Software (NLP-OSS 2023), pages 245–245, Singapore. Association for Computational… find
  15. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and 1 others. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
  16. An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, and 1 others. 2024. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409.12122. — Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement
  17. Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, and Songfang Huang. 2023. How well do large language models perform in arithmetic tasks? arXiv preprint arXiv:2304.02015.
  18. Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, and 1 others. 2022. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744.
  19. Louis Owen, Vishesh Tripathi, Abhay Kumar, and Biddwan Ahmed. 2024. Komodo: A linguistic expedition into indonesia’s regional languages. arXiv preprint arXiv:2403.09362. — [arXiv:2403.09362](https://arxiv.org/abs/2403.09362) find
  20. Qwen. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671. find
  21. Morgane Rivière, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan, Sammy Jerome, Anton Tsitsulin, an… find
  22. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In The Eleventh International Conference on Learning Representa…
  23. Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. 2025. Linguistic generalizability of test-time scaling in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2025, Vienna, Austria, July 27 - August 1, 2…

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/xiao-2026-math-idn-multilingual-mathematical-problem/