Elamur Research⌕
☰
home / papers / 2026 / MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning

MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning

Paper · 2026 · Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin et al.

Summary

Most benchmarks for LLM mathematical reasoning are English-centric or cover only a few high-resource languages. They also seldom look beyond final answers to intermediate reasoning, code-switching, or the ability to recognise flawed solutions. MathMist addresses this with a parallel multilingual benchmark built from Bangladeshi NCTB secondary-school mathematics textbooks (2018–2019 editions). The authors digitised problems with Gemini OCR, kept the notation in LaTeX, and checked the results by hand. They then expanded the set using LLMs: translation into thirteen languages, multiple-choice variants, and solutions with deliberately injected errors, all verified by subject experts. Open small and medium models, proprietary systems, and reasoning-oriented models were tested in zero-shot, chain-of-thought, perturbed-reasoning, and code-switched CoT settings. Scoring used accuracy, LLM-as-a-Judge, and human review. Models were inconsistent across languages and weakest in low-resource ones. They flagged incorrect solutions much more reliably than they located the actual mistake, and Bangla code-switched CoT often produced garbled mixed-token output. The pipeline offers a template for building parallel math evaluations in under-resourced languages.

Key points

  • Core resource: 2,890 gold Bangla–English parallel problems (1,445 per language) expanded to ≈30K aligned QA pairs. These comprise 18.9K parallel problems, 2.2K MCQs, and 8.6K perturbed solutions.
  • Languages: the thirteen covered are Bangla, English, French, Finnish, Lithuanian, Turkish, Kazakh, Persian, Arabic, Gujarati, Amharic, Swahili and Hausa. The conclusion mentions "seven", which is an inconsistency in the paper.
  • Chapters needing diagrams were excluded. Perturbations follow three strategies (σ1–σ3), and the multi-fallacy σ3 variant was easiest for models to detect.
  • Error detection vs. diagnosis: for example, Qwen-8B scores ~87% on binary detection but far lower on identifying the error. GPT-OSS-20B was strongest at Bangla error identification.
  • Scaling Qwen-3 from 0.6B to 14B raised English accuracy from 26.85% to 45.95% and shrank the English–Bangla gap from ~7% to ~1%. The data and code are on GitHub and HuggingFace (MathMist).

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓BRAC UniversityDhaka · Bangladesh6 works · 10 citations🎓United International UniversityDhaka · Bangladesh4 works · 4 citations

 Fields, topics and concepts

📚 Shelves 5The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 worksComputational Theory and Mathematics · 252 works

🧭 Topics 3Research subjects and languages the work is about.Reasoning · 1,426 worksEvaluation & benchmarks · 1,220 worksMultilingual & cross-lingual NLP · 828 works

Cite

Cite as Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam and Swakkhar Shatabda. 2026. MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.131

@inproceedings{sobhani2026mathmist,
  title = {{MathMist: A Parallel Multilingual Benchmark Dataset for Mathematical Problem Solving and Reasoning}},
  author = {Mahbub E Sobhani, Md. Faiyaz Abdullah Sayeedi, Tasnim Mohiuddin, Md Mofijul Islam and Swakkhar Shatabda},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.131},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.131}
}

References

39 references, 17 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, and 1 others. 2025. gpt-oss-120b & gpt-oss-20b model card. arXiv preprint arXiv:2508.10925.
  2. Gustavo Aguilar, Sudipta Kar, and Thamar Solorio. 2020. Lince: A centralized benchmark for linguistic code-switching evaluation. arXiv preprint arXiv:2005.04322.
  3. Kawsar Ahmed, Md Osama, Omar Sharif, Eftekhar Hossain, and Mohammed Moshiul Hoque. 2025. Ben-NumEval: A benchmark to assess LLMs’ numerical reasoning capabilities in Bengali. In Findings of the Association for Computational Linguistics: ACL 2025, pages 17782–17799, Vienna, Austria. Association for C… find
  4. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages…
  5. Reem Alghamdi, Zhenwen Liang, and Xiangliang Zhang. 2022. ArMATH: a dataset for solving Arabic math word problems. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 351–362, Marseille, France. European Language Resources Association.
  6. Aida Amini, Saadia Gabriel, Shanchuan Lin, Rik Koncel-Kedziorski, Yejin Choi, and Hannaneh Hajishirzi. 2019. MathQA: Towards interpretable math word problem solving with operation-based formalisms. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computation…
  7. Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xinnian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and 1 others. 2025. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence, vol… find
  8. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2024. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7001–7016, Miami, Florida, USA. Association for Co… find
  9. Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W Cohen. 2022. Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588. find
  10. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168.
  11. DeepL SE. 2025. Deepl translator. https://www.deepl.com/translator. Accessed: 2025-10. ↗
  12. DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, and 181 others. 2025. Deepseek-r1: Incentivizing reasoning capability in ll…
  13. Jalisha Jashim Era, Bidyarthi Paul, Tahmid Sattar Aothoi, Mirazur Rahman Zim, and Faisal Muhammad Shah. 2024. Empowering bengali education with ai: Solving bengali math word problems through transformer models. In 2024 27th International Conference on Computer and Information Technology (ICCIT), pag…
  14. Esin Gedik and Tunga Güngör. 2023. Solving turkish math word problems by sequence-to-sequence encoder-decoder models. Turkish Journal of Electrical Engineering and Computer Sciences, 31(2):431– 447. find
  15. Google LLC. 2025. Google translate. https://translate.google.com. Accessed: 2025-10. ↗
  16. Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.
  17. Injy Hamed, Caroline Sabty, Slim Abdennadher, Ngoc Thang Vu, Thamar Solorio, and Nizar Habash. 2025. A survey of code-switched arabic nlp: Progress, challenges, and future directions. arXiv preprint arXiv:2501.13419.
  18. Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021. Measuring mathematical problem solving with the MATH dataset. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
  19. Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin’e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, and Hongkai Xiong. 2023. From clip to dino: Visual encoders shout in multi-modal large language models. arXiv preprint arXiv:2310.08825. — [arXiv:2310.08825](https://arxiv.org/abs/2310.08825) find
  20. Kyung Seo Ki, Dong Geon Lee, and Gahgene Gweon. 2020. Kotab: Korean template-based arithmetic solver with bert. In 2020 IEEE International Conference on Big Data and Smart Computing (BigComp), pages 279–282. find
  21. Hui-Chi Kuo and Yun-Nung Chen. 2023. Zero-shot prompting for implicit intent prediction and recommendation with commonsense reasoning. In Findings of the Association for Computational Linguistics: ACL 2023, pages 249–258, Toronto, Canada. Association for Computational Linguistics. find
  22. Wenyang Luo, Wayne Xin Zhao, Jing Sha, Shijin Wang, and Ji-Rong Wen. 2025. MMATH: A multilingual benchmark for mathematical reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2025, pages 11187–11202, Suzhou, China. Association for Computational Linguistics. find
  23. Abeer Mahgoub, Ghada Khoriba, and ElHassan Anas ElSabry. 2024. Mathematical problem solving in arabic: Assessing large language models. Procedia Computer Science, 244:86–95. find
  24. Mariam Mahran and Katharina Simbeck. 2025. Investigating bias: A multilingual pipeline for generating, solving, and evaluating math problems with llms. arXiv preprint arXiv:2509.17701. find
  25. Sanchita Mondal, Debnarayan Khatua, Sourav Mandal, Dilip K Prasad, and Arif Ahmed Sekh. 2025. Bmwp: the first bengali math word problems dataset for operation prediction and solving. Discover Artificial Intelligence, 5(1):1–15. find
  26. Bidyarthi Paul, Jalisha Jashim Era, Mirazur Rahman Zim, Tahmid Sattar Aothoi, and Faisal Muhammad Shah. 2025. Leveraging large language models for bengali math word problem solving with chain of thought reasoning. arXiv preprint arXiv:2505.21354. find
  27. Harshita Sharma, Pruthwik Mishra, and Dipti Sharma. 2022. HAWP: a dataset for Hindi arithmetic word problem solving. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 3479–3490, Marseille, France. European Language Resources Association. find
  28. Guijin Son, Jiwoo Hong, Hyunwoo Ko, and James Thorne. 2025a. Linguistic generalizability of test-time scaling in mathematical reasoning. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14333–14368, Vienna, Austria. Association…
  29. Guijin Son, Donghun Yang, Hitesh Laxmichand Patel, Amit Agarwal, Hyunwoo Ko, Chanuk Lim, Srikant Panda, Minhyuk Kim, Nikunj Drolia, Dasol Choi, and 1 others. 2025b. Pushing on multilingual reasoning models with language-mixed chain-of-thought. arXiv preprint arXiv:2510.04230.
  30. Fengwei Teng, Zhaoyang Yu, Quan Shi, Jiayi Zhang, Chenglin Wu, and Yuyu Luo. 2025. Atom of thoughts for markov llm test-time scaling. arXiv preprint arXiv:2502.12018. — [arXiv:2502.12018](https://arxiv.org/abs/2502.12018) find
  31. Junior Cedric Tonga, KV Srivatsa, Kaushal Kumar Maurya, Fajri Koto, and Ekaterina Kochmar. 2025. Simulating llm-to-llm tutoring for multilingual math feedback. arXiv preprint arXiv:2506.04920. find
  32. Yiming Wang, Pei Zhang, Jialong Tang, Hao-Ran Wei, Baosong Yang, Rui Wang, Chenshu Sun, Feitong Sun, Jiran Zhang, Junxuan Wu, Qiqian Cang, Yichang Zhang, Fei Huang, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. Polymath: Evaluating mathematical reasoning in multilingual contexts. In The Thirty-nin… find
  33. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824– 24837.
  34. Yanan Wu, Jie Liu, Xingyuan Bu, Jiaheng Liu, Zhanhui Zhou, Yuanxing Zhang, Chenchen Zhang, Zhiqi Bai, Haibin Chen, Tiezheng Ge, Wanli Ouyang, Wenbo Su, and Bo Zheng. 2024. ConceptMath: A bilingual concept-wise benchmark for measuring mathematical reasoning of large language models. In Findings of th… find
  35. Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mt5: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 2021 conference of the North American chapter of the association for computational ling…
  36. Brian Yan, Injy Hamed, Shuichiro Shimizu, Vasista Lodagala, William Chen, Olga Iakovenko, Bashar Talafha, Amir Hussein, Alexander Polok, Kalvin Chang, and 1 others. 2025. Cs-fleurs: A massively multilingual and code-switched speech dataset. arXiv preprint arXiv:2509.14161. — [arXiv:2509.14161](https://arxiv.org/abs/2509.14161) find
  37. An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, and 1 others. 2025. Qwen3 technical report. arXiv preprint arXiv:2505.09388.
  38. Dongxiang Zhang, Lei Wang, Luming Zhang, Bing Tian Dai, and Heng Tao Shen. 2020. The gap of semantic parsing: A survey on automatic math word problem solvers. IEEE Trans. Pattern Anal. Mach. Intell., 42(9):2287–2305. find
  39. Yuanchi Zhang, Yile Wang, Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. 2024. Enhancing multilingual capabilities of large language models through self-distillation from resource-rich languages. arXiv preprint arXiv:2402.12204.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sobhani-2026-mathmist-parallel-multilingual-benchmark-dataset/