Elamur Research⌕
☰
home / papers / 2026 / Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis

Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis

Summary

Reviewing translated game text by hand is slow and costly. It has also been unclear how well LLMs can automate this localization quality assurance (LQA) across different models, target languages and game genres. The authors build a bug-seeded benchmark from professionally localized strings of three commercial games released before 2023: one RPG and two strategy titles. They inject errors using 16 rule-based patterns at three difficulty levels, covering linguistic, technical and cultural problems. This yields a 143,665-sample corpus. From it they draw 48,000 items spanning eight LLMs (closed-source and open-weight), six target languages and two genres. Each item is a binary bug/no-bug decision scored with F1. Detection runs in a pipeline that pairs cheap rule-based flagging with LLM re-validation. Claude Sonnet 4 performs best, ahead of Qwen-2.5-72B and Gemini 2.0 Flash. Neither target language nor genre significantly changes accuracy, although Japanese is the hardest language and French the most stable. The study offers studios evidence-based guidance for choosing LQA models under quality and budget constraints.

Key points

  • Benchmark setup: 96 settings (8 models × 6 languages: zh, ja, ko, fr, de, es × 2 genres) with 500 samples each.
    • The corpus is roughly 80% clean and 20% bug-seeded.
    • Positive rates per setting range from 6% to 32%, about 19% overall.
    • Bugs are split by difficulty into 35% easy, 46% medium and 19% hard.
  • Models tested: Claude-Sonnet-4, GPT-4o, Gemini-2.0-Flash, GPT-4o-mini, DeepSeek-v3, Llama-3.3-70B, Qwen-2.5-72B and Mixtral-8x22B. All were queried with January 2026 API versions at temperature 0.1.
  • Top results: Claude Sonnet 4 reached F1 = 0.766, Qwen-2.5-72B 0.711 and Gemini 2.0 Flash 0.691. The effect of target language was not significant (p = 0.285).
  • Cost: Qwen-2.5-72B is presented as a much cheaper open-weight alternative with competitive quality. The conclusion names Gemini-2.0-Flash as the best overall cost–performance tradeoff.
  • Real-world check: the top three models were tested on 500 real bug reports from two studios.
    • Their rankings matched the benchmark closely (Spearman ρ = 0.89).
    • Absolute F1 was 4–6 points lower than on the benchmark.
    • The authors therefore advise expecting a 5–10% performance drop in production.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{tian2026benchmarking,
  title = {{Benchmarking Large Language Models for Game Localization Quality Assurance: A Cross-Model, Cross-Lingual Analysis}},
  author = {Tian, Mao and Wu, Na},
  booktitle = {Proceedings of the 17th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track)},
  year = {2026},
  url = {https://aclanthology.org/2026.amta-research.12/}
}

References

16 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Banerjee, S. and Lavie, A. (2005). METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Proceedings of the 17th Conference of the Association for Machine Translation in the Americas, Québec City, Canada, August 31 - September 2, 2026… find our card
  2. Bernal-Merino, M. Á. (2015). Translation and localisation in video games: Making entertainment software global. Routledge. find
  3. Fernandes, P., Deutsch, D., Finkelstein, M., Riley, P., Martins, A. F., Neubig, G., Garg, A., Clark, J. H., Freitag, M., and Firat, O. (2023). The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Mach… find our card
  4. Freitag, M., Mathur, N., Lo, C.-k., Avramidis, E., Rei, R., Thompson, B., Kocmi, T., Blain, F., Deutsch, D., Stewart, C., Zerva, C., Castilho, S., Lavie, A., and Foster, G. (2023). Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Ei… find our card
  5. Guerreiro, N. M., Rei, R., van Stigt, D., Coheur, L., Colombo, P., and Martins, A. F. (2024). xCOMET: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. find our card
  6. Jung, D., Eo, S., Park, C., and Lim, H. (2024). Explainable CED: A dataset for explainable critical error detection in machine translation. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 4: Stu… find
  7. O’Hagan, M. and Mangiron, C. (2013). Game localization: Translating for the global digital entertainment industry. John Benjamins Publishing. find
  8. Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318. find our card
  9. Rei, R., Stewart, C., Farinha, A. C., and Lavie, A. (2020). COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702. find our card
  10. Rei, R., Treviso, M., Guerreiro, N. M., Zerva, C., Farinha, A. C., Maroti, C., de Souza, J. G., Glushkova, T., Alves, D., Coheur, L., Lavie, A., and Martins, A. F. (2022). find
  11. COMETKiwi: IST-Unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 634–645. find our card
  12. Sellam, T., Das, D., and Parikh, A. P. (2020). BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892. find
  13. Valli, P. (2015). The TAUS quality dashboard. In Proceedings of Translating and the Computer 37, London, UK. AsLing. find
  14. Kocmi, T. and Federmann, C. (2023). Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203. find
  15. Liu, N. F., Lin, K., Hewitt, J., Paranjape, A., Bevilacqua, M., Petroni, F., and Liang, P. (2023). Lost in the middle: How language models use long contexts. arXiv preprint arXiv:2307.03172. find our card
  16. Lommel, A., Uszkoreit, H., and Burchardt, A. (2014). Multidimensional quality metrics (MQM): A framework for declaring and describing translation quality Proceedings of the 17th Conference of the Association for Machine Translation in the Americas, Québec City, Canada, August 31 - September 2, 2026.… find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/tian-2026-benchmarking-large-language-models-game/