Elamur Research⌕
☰
home / papers / 2026 / Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results

Summary

The paper asks whether the language gap that LLMs show on multilingual math benchmarks is real or a byproduct of flawed evaluation. The authors tested open models (Gemma 3, DeepSeek V3) and proprietary ones (Gemini, Claude, GPT) on MGSM, using the standard Kaggle prompt and its answer-extraction method. English accuracy was in the high 90s, but other languages fell behind, including high-resource French. This would normally suggest weak cross-lingual transfer.

Closer inspection showed two problems. First, the human translations contained subtle errors that changed the meaning of some problems. Second, extraction ignored local number formats, such as dot or space thousand separators and Bengali digits. The authors propose a semi-automatic quality-assurance procedure that uses strong models to flag suspicious items, and they recommend language-aware answer parsing. After both fixes, the gap mostly vanishes for the strongest models, which reverses the original conclusion. The work warns that benchmark quality and evaluation details can distort multilingual findings, especially when benchmarks approach saturation.

Key points

  • MGSM consists of 250 grade-school problems drawn from GSM8K, which professional translators rendered into 10 typologically diverse languages: Bengali (bn), German (de), Spanish (es), French (fr), Japanese (ja), Russian (ru), Swahili (sw), Telugu (te), Thai (th) and Chinese (zh).
  • With the original data, French showed large drops: 15.2% for Gemini 2.5 Pro and 17.6% for GPT5. These losses sometimes exceeded those for Swahili or Telugu, and the pattern was consistent across models.
  • In one example error, a German translation turned "every second glass" into "each additional glass." This changes the correct numerical answer.
  • The model setups were identical before and after the fixes; only the data and the answer extraction changed. The corrected dataset is released as MGSM-Rev2 on GitHub (google-research-datasets).
  • The authors note several caveats:
    • Gemini was used to translate and verify the corrections.
    • The English source wording was slightly refined, so scores are not directly comparable with older MGSM results.
    • Contamination of GSM8K remains possible.
    • MGSM now looks saturated, so benchmarks with more headroom may still reveal real gaps.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.gem-main.22

@inproceedings{peter2026mind,
  title = {{Mind the Gap... or Not? How Translation Errors and Evaluation Details Skew Multilingual Results}},
  author = {Peter, Jan-Thorsten and Vilar, David and Domhan, Tobias and Malkin, Dan and Freitag, Markus},
  booktitle = {Proceedings of the Fifth Workshop on Generation, Evaluation and Metrics (GEM)},
  year = {2026},
  doi = {10.18653/v1/2026.gem-main.22},
  url = {https://doi.org/10.18653/v1/2026.gem-main.22}
}

References

14 references, 4 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anthropic. 2025. Introducing Claude Sonnet 4.5 (blog post). https://www.anthropic.com/news/ claude-sonnet-4-5. ↗
  2. Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901. find
  3. Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2023. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. arXiv preprint arXiv:2310.20246. ↗
  4. Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1–113. find
  5. Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. Preprint, arXiv:2110.14168. ↗ our card
  6. Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023. Do multilingual language models think better in english? arXiv preprint arXiv:2308.01223. ↗
  7. Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805. ↗ our card
  8. Gemini Team, Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, and 1 others. 2025. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. ↗ our card
  9. Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, and 1 others. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786. ↗
  10. Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023. Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting. arXiv preprint arXiv:2305.07004. ↗ our card
  11. Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, and 1 others. 2024a. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437. ↗
  12. Chaoqun Liu, Wenxuan Zhang, Yiran Zhao, Anh Tuan Luu, and Lidong Bing. 2024b. Is translation all you need? a study on solving multilingual tasks with large language models. arXiv preprint arXiv:2403.10258. ↗
  13. OpenAI. 2025. Introducing GPT-5 (blog post). https://openai.com/index/ introducing-gpt-5/. ↗
  14. Irene Plaza, Nina Melero, Cristina del Pozo, Javier Conde, Pedro Reviriego, Marina Mayor-Rocher, and María Grandury. 2024. Spanish and llm benchmarks: is mmlu lost in translation? Preprint, arXiv:2406.17789. ↗

This page in Markdown → https://md.research.elamur.ai/free/papers/peter-2026-mind-gap-or-not-how/