Elamur Research⌕
☰
home / papers / 2026 / Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages

Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages

Summary

The authors test whether large language models solve math word problems (MWP) just as well when the problems contain local names, organizations and currencies. Benchmarks for low-resource languages are usually built by translating from English. As a result, they keep English names and dollars, and the evaluation actually measures how a model performs in an English-centric context. The authors propose an automatic LLM-based localization pipeline. First, a model identifies key entities in the English original. These are then replaced with equivalents from a database of local names, and an LLM rewrites the translated text with the new entities. Finally, automatic checks filter out failed variants. In this way, the authors created localized versions of the problems for 18 African languages from AfriMGSM. The experiments showed that translated benchmarks inflate scores: after localization, the accuracy of most models drops, by as much as 9% for GPT-4o-mini. Fine-tuning on localized and mixed data makes models more robust to such cultural shifts. The value of the work is that it offers a cheap way to bring multilingual evaluation closer to how native speakers actually use a model.

Key points

  • The pipeline consists of several stages: translation, LLM-based extraction of entities (personal names, organization names, currencies), a replacement dictionary, entity substitution, LLM rewriting of the text, and quality control. The quality control checks that no original entities remain in the text and that the new ones have appeared, and that similarity to the translation exceeds 0.8. If the check fails, the plain translation is kept.
  • The test split was fully verified by humans, and the training split was checked by sampling. The data is based on GSM8K and AfriMGSM (18 African languages). The code is published in the Auto-Localizer repository on GitHub.
  • On the localized data, GPT-4o-mini drops the most — by up to 9% in numeric answer match. Gemma-2-27b-it is the most stable. For French and Swahili, the effect is especially noticeable in Deepseek-r1-distill-llama-70b and Mistral-nemo-instruct-2407.
  • On English problems with local entities, LLaMA-3-70B-Instruct and GPT-4o-mini even achieve higher accuracy than on the original, while Aya-expanse-32b lags behind.
  • LLaMA-3-8B-Instruct and Gemma-2-9b-it were fine-tuned on subsets of 1k, 5k, 10k and 25k examples drawn from translated, partially replaced, localized and combined data. The models trained on a combination of all data types turned out to be the most robust, and Gemma-2-9b-it benefits more from this.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.42 · arXiv 2508.14913

@inproceedings{azime2026bridging,
  title = {{Bridging the Culture Gap: A Framework for LLM-Driven Socio-Cultural Localization of Math Word Problems in Low-Resource Languages}},
  author = {Azime, Israel Abebe and Belay, Tadesse Destaw and Klakow, Dietrich and Slusallek, Philipp and Chhabra, Anshuman},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.42},
  eprint = {2508.14913},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.42}
}

References

31 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Zain Ul Abedin, Shahzeb Qamar, Lucie Flek, and Akbar Karimi. 2025. Arithmattack: Evaluating robustness of llms to noisy context in math problem solving. arXiv preprint arXiv:2501.08203. find
  2. Akshit Achara and Anshuman Chhabra. 2025. Watching the AI watchdogs: A fairness and robustness analysis of AI safety moderation classifiers. In NAACL, pages 253–264. find
  3. David Ifeoluwa Adelani, Jessica Ojo, Israel Abebe Azime, Jian Yun Zhuang, Jesujoba O Alabi, Xuanli He, Millicent Ochieng, Sara Hooker, Andiswa Bukula, En-Shiun Annie Lee, and 1 others. 2024. Irokobench: A new benchmark for african languages in the age of large language models. arXiv preprint arXiv:2… find
  4. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024a. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157. find our card
  5. Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. 2024b. Large language models for mathematical reasoning: Progresses and challenges. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop, pages… find our card
  6. Jesujoba O Alabi, Michael A Hedderich, David Ifeoluwa Adelani, and Dietrich Klakow. 2025. Charting the landscape of african nlp: Mapping progress and shaping the road ahead. arXiv preprint arXiv:2505.21315. find
  7. Hadi Askari, Anshuman Chhabra, Muhao Chen, and Prasant Mohapatra. 2025. Assessing LLMs for zero-shot abstractive summarization through the lens of relevance paraphrasing. In NAACL (Findings), pages 2187–2201. find
  8. Anshuman Chhabra, Hadi Askari, and Prasant Mohapatra. 2024. Revisiting zero-shot abstractive summarization in the era of large language models from the perspective of position bias. In NAACL, pages 1–11. find
  9. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, and 1 others. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168. find our card
  10. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arX… find our card
  11. Aabid Karim, Abdul Karim, Bhoomika Lohana, Matt Keon, Jaswinder Singh, and Abdul Sattar. 2025. Lost in cultural translation: Do llms struggle with math across cultural contexts? arXiv preprint arXiv:2503.18018. find
  12. "Fajri Koto, Haonan Li, Sara Shatanawi, Jad Dough-man, Abdelrahman Boda Sadallah, Aisha Alraeesi, Khalid Almubarak, Zaid Alyafeai, Neha Sengupta, Shady Shehata, Nizar Habash, Preslav Nakov, and Timothy Baldwin". 2024. Arabicmmlu: Assessing massive multitask language understanding in arabic. In Findi… find our card
  13. Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212. find our card
  14. Senyu Li, Jiayi Wang, Felermino DMA Ali, Colin Cherry, Daniel Deutsch, Eleftheria Briakou, Rui Sousa-Silva, Henrique Lopes Cardoso, Pontus Stenetorp, and David Ifeoluwa Adelani. 2025. Ssa-comet: Do llms outperform learned metrics in evaluating mt for under-resourced african languages? arXiv preprint… find our card
  15. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia-Gonzalez, Prangthip Hansanti, and 20 o… find
  16. John Dang, Shivalika Singh, Daniel D’souza, Arash Ahmadian, Alejandro Salamanca, Madeline Smith, Aidan Peppin, Sungjin Hong, Manoj Govindassamy, Terrence Zhao, Sandra Kublik, Meor Amer, Viraat Aryabumi, Jon Ander Campos, Yi-Chern Tan, Tom Kocmi, Florian Strub, Nathan Grinsztajn, Yannis Flet-Berliac,… find our card
  17. Jessica Ojo, Kelechi Ogueji, Pontus Stenetorp, and David Ifeoluwa Adelani. 2023. How good are large language models on african languages? arXiv preprint arXiv:2311.07978. — [arXiv:2311.07978](https://arxiv.org/abs/2311.07978) find
  18. Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, and 5 others.… find
  19. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, and 1 others. 2022b. Language models are multilingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057. find our card
  20. Mahammed Kamruzzaman, Abdullah Al Monsur, Gene Louis Kim, and Anshuman Chhabra. 2025. From anger to joy: How nationality personas shape emotion attribution in large language models. In Proceedings of the 14th International Joint Conference on Natural Language Processing and the 4th Conference of the… find
  21. Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022a. Language models are multilingual chain-of-thought reasoners. find our card
  22. Guijin Son, Hanwool Lee, Sungdong Kim, Seungone Kim, Niklas Muennighoff, Taekyoon Choi, Cheon-bok Park, Kang Min Yoo, and Stella Biderman. 2024. Kmmlu: Measuring massive multitask language understanding in korean. arXiv preprint arXiv:2402.11548. find
  23. Kv Aditya Srivatsa and Ekaterina Kochmar. 2024. What makes math word problems challenging for LLMs? In Findings of the Association for Computational Linguistics: NAACL 2024, pages 1138–1148, Mexico City, Mexico. Association for Computational Linguistics. find
  24. Erik F. Tjong Kim Sang and Fien De Meulder. 2003. Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition. In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003, pages 142– 147. find
  25. Gemma is more sensitive to entity grounding and gains robustness when trained with culturally and linguistically aligned examples. find
  26. Aditya Tomar, Nihar Ranjan Sahoo, Ashish Mittal, Rudra Murthy, and Pushpak Bhattacharyya. 2025a. Mathematics isn’t culture-free: Probing cultural gaps via entity and scenario perturbations. Preprint, arXiv:2507.00883. — [arXiv:2507.00883](https://arxiv.org/abs/2507.00883) find
  27. Aditya Tomar, Nihar Ranjan Sahoo, Ashish Mittal, Rudra Murthy, and Pushpak Bhattacharyya. 2025b. Mathematics isn’t culture-free: Probing cultural gaps via entity and scenario perturbations. arXiv preprint arXiv:2507.00883. — [arXiv:2507.00883](https://arxiv.org/abs/2507.00883) find
  28. Yibo Yan, Jiamin Su, Jianxiang He, Fangteng Fu, Xu Zheng, Yuanhuiyi Lyu, Kun Wang, Shen Wang, Qingsong Wen, and Xuming Hu. 2024. A survey of mathematical reasoning in the era of multimodal large language model: Benchmark, method & challenges. arXiv preprint arXiv:2412.11936. find
  29. Junjie Ye, Nuo Xu, Yikun Wang, Jie Zhou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024. Llm-da: Data augmentation via large language models for few-shot named entity recognition. arXiv preprint arXiv:2402.14568. — [arXiv:2402.14568](https://arxiv.org/abs/2402.14568) find
  30. Hao Yu, Jesujoba O Alabi, Andiswa Bukula, Jian Yun Zhuang, En-Shiun Annie Lee, Tadesse Kebede Guge, Israel Abebe Azime, Happy Buzaaba, Blessing Kudzaishe Sibanda, Godson K Kalipe, and 1 others. 2025. Injongo: A multicultural intent detection and slot-filling dataset for 16 african languages. arXiv p… find
  31. Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. 2023. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309.12284. find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/azime-2026-bridging-culture-gap-framework-llm/