Elamur Research⌕
☰
home / papers / 2026 / Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets

Summary

Translated versions of English benchmarks are widely used to evaluate LLMs in other languages. Many were made with classical MT tools or older models, often translating questions and answer options separately. This breaks context and grammatical agreement. In gendered languages it can also hint at the correct option, which distorts scores.

The authors built an automated, configurable pipeline for translating datasets and benchmarks. It offers four strategies at increasing cost:

  • single-prompt translation with optional self-correction;
  • Best-of-N with an LLM judge;
  • Universal Self-Improvement (USI), which merges sampled candidates;
  • their new T-RANK, which ranks multi-prompt candidates over several competitive rounds.

They applied it to MMLU, Hellaswag, ARC and Winogrande in eight Eastern and Southern European languages, including Ukrainian. T-RANK and USI achieved the best COMET scores on WMT24++ and FLORES. The new translations also beat existing ones such as Global-MMLU in LLM-as-a-judge comparisons. Models scored consistently higher on the new versions, suggesting that poor translations had been understating model ability.

Key points

  • Languages covered: Ukrainian, Bulgarian, Slovak, Romanian, Lithuanian, Estonian, Turkish and Greek. Not every benchmark covers all eight; for example, ARC-Challenge and Hellaswag were done only for UK, RO, SK and BG.
  • In pairwise comparisons judged by Gemini-2.5-Flash, the T-RANK/USI translations won clearly more often than Global-MMLU in every evaluated language.
  • Average score gains on the new translations (Gemma 3, Qwen 3 and Llama 3.1 evaluated) were:
    • Winogrande: +3.42%
    • ARC-Challenge: +2.35%
    • Hellaswag: +1.63%
    • MMLU: +0.94%
  • Winogrande gained most because the pipeline keeps question–answer context intact and removes answer leakage through gender endings. By language, the largest gains were Greek (+3.89%) and Ukrainian (+2.7%).
  • The code is released as insait-institute/ritranslation and the benchmarks as insait-institute/multilingual-benchmarks. Main limitations:
    • translation relied mostly on closed-source models;
    • candidate selection used an LLM judge rather than COMET.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.2067

@inproceedings{yukhymenko2026recovered,
  title = {{Recovered in Translation: Efficient Pipeline for Automated Translation of Benchmarks and Datasets}},
  author = {Yukhymenko, Hanna and Alexandrov, Anton and Vechev, Martin},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.2067},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.2067}
}

References

20 references, 7 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023. Universal self-consistency for large language model generation. ↗
  2. Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. Preprint, arXiv:1803.05457. ↗
  3. Daniel Deutsch, Eleftheria Briakou, Isaac Caswell, Mara Finkelstein, Rebecca Galor, Juraj Juraska, Geza Kovacs, Alison Lui, Ricardo Rei, Jason Riesa, Shruti Rijhwani, Parker Riley, Elizabeth Salesky, Firas Trabelsi, Stephanie Winkler, Biao Zhang, and Markus Freitag. 2025. Wmt24++: Expanding the language coverage of wmt24 to 55 languages & dialects. Preprint, arXiv:2502.12404. ↗
  4. Zhaopeng Feng, Yan Zhang, Hao Li, Bei Wu, Jiayu Liao, Wenqiang Liu, Jun Lang, Yang Feng, Jian Wu, and Zuozhu Liu. 2024. Tear: Improving llm-based machine translation with systematic self-refinement. find
  5. Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc’Aurelio Ranzato, Francisco Guzmán, and Angela Fan. 2022. The flores-101 evaluation benchmark for low-resource and multilingual machine translation. Transactions of the Association for Computational Linguistics, 10:522–538. ↗
  6. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2023. xcomet: Transparent machine translation evaluation through fine-grained error detection. Preprint, arXiv:2310.10482. ↗ our card
  7. Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc’Aurelio Ranzato. 2019. The flores evaluation datasets for low-resource machine translation: Nepali-english and sinhala-english. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 6098– 6111. Association for Computational Linguistics. find our card
  8. Wenhan Han, Yifan Zhang, Zhixun Chen, Binbin Liu, Haobin Lin, Bingni Zhang, Taifeng Wang, Mykola Pechenizkiy, Meng Fang, and Yin Zheng. 2025. Mubench: Assessment of multilingual capabilities of large language models across 61 languages. Preprint, arXiv:2506.19468. ↗
  9. Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. Preprint, arXiv:2009.03300. ↗ our card
  10. Ammar Khairi, Daniel D’souza, Marzieh Fadaee, and Julia Kreutzer. 2025a. Making, not taking, the best of n. Preprint, arXiv:2510.00931. ↗
  11. Ammar Khairi, Daniel D’souza, Ye Shen, Julia Kreutzer, and Sara Hooker. 2025b. When life gives you samples: The benefits of scaling up inference compute for multilingual llms. Preprint, arXiv:2506.20544. ↗
  12. Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan A. Rossi, and Thien Huu Nguyen. 2023. Okapi: Instruction-tuned large language models in multiple languages with reinforcement learning from human feedback. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 318–327, Singapore. Association for Computational Linguistics. ↗ our card
  13. Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André F. T. Martins. 2023. Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task. Preprint, arXiv:2309.11925. ↗ our card
  14. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  15. Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Winogrande: An adversarial winograd schema challenge at scale. Preprint, arXiv:1907.10641. ↗
  16. Shivalika Singh, Angelika Romanou, Clémentine Fourrier, David I Adelani, Jian Gang Ngui, Daniel Vila-Suero, Peerat Limkonchotiwat, Kelly Marchisio, Wei Qi Leong, Yosephine Susanto, and 1 others. 2024. Global mmlu: Understanding and addressing cultural and linguistic biases in multilingual evaluation. find our card
  17. Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008– 3021. find
  18. Lei Tang, Jinghui Qin, Wenxuan Ye, Hao Tan, and Zhijing Yang. 2025. Adaptive few-shot prompting for machine translation with pre-trained language models. find
  19. Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? Preprint, arXiv:1905.07830. ↗
  20. Vilém Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. Pitfalls and outlooks in using COMET. In Proceedings of the Ninth Conference on Machine Translation, pages 1272– 1288, Miami, Florida, USA. Association for Computational Linguistics. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/yukhymenko-2026-recovered-translation-efficient-pipeline-automated/