Elamur Research⌕
☰
home / papers / 2026 / How Important is ‘Perfect' English for Machine Translation Prompts?

How Important is ‘Perfect' English for Machine Translation Prompts?

Paper · 2026 · Patrícia Schmidtová, Niyati Bafna, Seth Aycock et al.

Summary

LLMs now lead machine translation benchmarks, but they are tuned on carefully written standard English prompts. Real users, including L2 English speakers and casual lay users, often write prompts with mistakes. The authors ask how much such imperfections actually hurt translation and translation evaluation. They built error augmenters grounded in L2 English research, covering spelling, phonetic, phrasal, and register deviations, plus combinations simulating low-proficiency and "lazy" users. Each was applied at controlled intensities to several base MT prompts. Six closed and open LLMs were then tested on MT, scored with COMET, and on GEMBA-DA quality estimation.

Spelling errors caused the clearest drop. Phrase-level simplifications had little effect and sometimes helped. Still, the spread caused by picking a different starting prompt was larger than most realistic error effects. Errors mainly weakened instruction following, such as extra text or wrong-language output, rather than translation ability itself. The work shows that clean-prompt benchmarks may overstate real-world quality. It also raises fairness concerns for non-native users.

Key points

  • Setup: six models (GPT-4o-mini, Gemini-2.0-flash, Llama 3.1, EuroLLM, Qwen2.5, Tower) were tested with seven error augmenters across multiple language pairs. The abstract and dataset description say three pairs, while the conclusion mentions four. Five base prompts were used, including a minimal one.
  • Orthographic noise: character-level perturbations were applied with probability p in [0, 0.4], sampled at 10 evenly spaced levels. Phrasal and register variants were generated with an LLM at two intensity levels and then manually post-edited.
  • Robustness under stress: in stress tests with heavy uniform noise (p ≥ 0.7), prompts were unreadable to humans. Models still frequently returned valid translations, apparently relying on the language names and source text.
  • Quality estimation: degraded prompts had only a weak negative effect on GEMBA-DA, measured by system-level correlation with human judgments.
  • Released resources: the authors release a dataset of translation outputs and their code. Total compute cost was estimated at under USD 200.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Johns Hopkins UniversityBaltimore · United States240 works · 96,487 citations🎓University of AmsterdamAmsterdam · Netherlands144 works · 173,906 citations🎓Charles UniversityPrague · Czechia133 works · 20,686 citations🎓ETH ZurichZurich · Switzerland72 works · 5,176 citations🎓Technical University of MunichMunich · Germany55 works · 48,474 citations🎓Munich Center for Machine LearningMunich · Germany57 works · 1,094 citations🎓Poznań University of TechnologyPoznan · Poland3 works · 1 citation

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksEnglish · 2,263 works

Cite

Cite as Patrícia Schmidtová, Niyati Bafna, Seth Aycock, Gianluca Vico, Wiktor Kamzela, Kathy Hämmerl and Vilém Zouhar. 2026. How Important is ‘Perfect' English for Machine Translation Prompts?. Findings of the Association for Computational Linguistics: EACL 2026.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-eacl.38

@inproceedings{schmidtova2026how,
  title = {{How Important is ‘Perfect' English for Machine Translation Prompts?}},
  author = {Patrícia Schmidtová, Niyati Bafna, Seth Aycock, Gianluca Vico, Wiktor Kamzela, Kathy Hämmerl and Vilém Zouhar},
  booktitle = {Findings of the Association for Computational Linguistics: EACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-eacl.38},
  url = {https://doi.org/10.18653/v1/2026.findings-eacl.38}
}

References

44 references, 20 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Duarte Alves, Nuno Guerreiro, João Alves, José Pombal, Ricardo Rei, José de Souza, Pierre Colombo, and Andre Martins. 2023. Steering large language models for machine translation with finetuning and in-context learning. In Findings of the Association for Computational Linguistics: EMNLP 2023, 11127–… find
  2. Niyati Bafna, Kenton Murray, and David Yarowsky. 2024. Evaluating large language models along dimensions of language variation: A systematik invesdigatiom uv cross-lingual generalization. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 18742–18762. Associat… find
  3. Rachel Bawden and François Yvon. 2023. Investigating the translation performance of a large multilingual language model: the case of BLOOM. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 157–170. European Association for Machine Translation.
  4. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In International Conference on Learning Representations. find
  5. Eleftheria Briakou, Colin Cherry, and George Foster. 2023. Searching for needles in a haystack: On the role of incidental bilingualism in PaLM‘s translation capability. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 9432–9452. Asso… find
  6. Eleftheria Briakou, Zhongtao Liu, Colin Cherry, and Markus Freitag. 2024. On the implications of verbose LLM outputs: A case study in translation evaluation. find
  7. Vivian J Cook. 1997. L2 users and English spelling. Journal of Multilingual and Multicultural Development, 18(6):474–488. find
  8. Krishno Dey, Prerona Tarannum, Md. Arid Hasan, Imran Razzak, and Usman Naseem. 2024. Better to ask in english: Evaluation of large language models on english, low-resource and cross-lingual settings.
  9. Abhimanyu Dubey et al. 2024. The Llama 3 herd of models.
  10. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eig…
  11. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared…
  12. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references…
  13. Hila Gonen, Srini Iyer, Terra Blevins, Noah A. Smith, and Luke Zettlemoyer. 2024. Demystifying prompts in language models via perplexity estimation. find
  14. Google. 2024. Introducing gemini 2.0: Our new ai model for the agentic era. find
  15. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995.
  16. Amr Hendy, Mohamed Abdelrehim, Amr Sharaf, Vikas Raunak, Mohamed Gabr, Hitokazu Matsushita, Young Jin Kim, Mohamed Afify, and Hany Hassan Awadalla. 2023. How good are GPT models at machine translation? A comprehensive evaluation. find
  17. Xu Huang, Zhirui Zhang, Xiang Geng, Yichao Du, Jiajun Chen, and Shujian Huang. 2024. Lost in the source language: How large language models evaluate the quality of machine translation. In Findings of the Association for Computational Linguistics: ACL 2024, 3546–3562, Bangkok, Thailand. Association f… find
  18. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popov… — Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet
  19. Tom Kocmi and Christian Federmann. 2023a. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, 768–775. Association for Computational Linguistics.
  20. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, 193–203, Tampere, Finland. European Association for Machine Translation. find
  21. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024b. Error span annotation: A balanced approach for human evaluation of machine translation. In Proceedings of the Ninth Conference on Machine Translation, 14…
  22. Qingyu Lu, Baopu Qiu, Liang Ding, Kanjian Zhang, Tom Kocmi, and Dacheng Tao. 2024. Error analysis prompting enables human-like translation evaluation in large language models. In Findings of the Association for Computational Linguistics: ACL 2024, 8801– 8816, Bangkok, Thailand. Association for Compu…
  23. Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. 2024. Eurollm: Multilingual language m…
  24. OpenAI. 2024. Gpt-4o mini: Advancing cost-efficient intelligence. find
  25. Ben Peters and Andre Martins. 2025. Did translation models get more robust without anyone Even noticing? In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2445–2458, Vienna, Austria. Association for Computational Linguistics. find
  26. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, 392–395. Association for Computational Linguistics.
  27. Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, 186–191. Association for Computational Linguistics.
  28. Shenbin Qian, Archchana Sindhujan, Minnie Kabra, Diptesh Kanojia, Constantin Orasan, Tharindu Ranasinghe, and Fred Blain. 2024. What do large language models need for machine translation evaluation? In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, 3660–3674.… find
  29. Yao Qiang, Subhrangshu Nandi, Ninareh Mehrabi, Greg Ver Steeg, Anoop Kumar, Anna Rumshisky, and Aram Galstyan. 2024. Prompt perturbation consistency learning for robust language models. In Findings of the Association for Computational Linguistics: EACL 2024, 1357–1370. Association for Computational… find
  30. Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022a. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), 578–5…
  31. Ricardo Rei, Jose Pombal, Nuno M. Guerreiro, Jo textasciitilde ao Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, Sweta Agrawal, Antonio Farinhas, José G. C. De Souza, and André Martins. 2024. Tower v2: Unbabel-IST 2024 submission for the general… find
  32. Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. 2022b. find
  33. Cometkiwi: Ist-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), 634–645, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
  34. Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  35. Ayako Sato, Kyotaro Nakajima, Hwichan Kim, Zhousi Chen, and Mamoru Komachi. 2024. TMU-HIT‘s submission for the WMT24 quality estimation shared task: Is GPT-4 a good evaluator for machine translation? In Proceedings of the Ninth Conference on Machine Translation, 529–534. Association for Computationa… find
  36. Aarohi Srivastava and David Chiang. 2025. We’re calling an intervention: Exploring fundamental hurdles in adapting language models to nonstandard text. In Proceedings of the Tenth Workshop on Noisy and User-generated Text, 45–56, Albuquerque, New Mexico, USA. Association for Computational Linguistic… find
  37. Anton Voronov, Lena Wolf, and Max Ryabinin. 2024. Mind your format: Towards consistent evaluation of in-context learning improvements. In Findings of the Association for Computational Linguistics: ACL 2024, 6287–6310. Association for Computational Linguistics. find
  38. Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla. 2024. A paradigm shift in machine translation: Boosting translation performance of large language models. In The Twelfth International Conference on Learning Representations. find
  39. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei… find
  40. J.D. Zamfirescu-Pereira, Richmond Y. Wong, Bjoern Hartmann, and Qian Yang. 2023. Why johnny can’t prompt: How non-ai experts try (and fail) to design llm prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, CHI ’23, New York, NY, USA. Association for Computing Ma… find
  41. Biao Zhang, Barry Haddow, and Alexandra Birch. 2023. Prompting large language model for machine translation: A case study. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org.
  42. Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595–4…
  43. Kaijie Zhu, Jindong Wang, Jiaheng Zhou, Zichen Wang, Hao Chen, Yidong Wang, Linyi Yang, Wei Ye, Yue Zhang, Neil Gong, and Xing Xie. 2024. Promptrobust: Towards evaluating the robustness of large language models on adversarial prompts. In Proceedings of the 1st ACM Workshop on Large AI Systems and Mo…
  44. Vilém Zouhar, Pinzhen Chen, Tsz Kin Lam, Nikita Moghe, and Barry Haddow. 2024. Pitfalls and outlooks in using COMET. In Proceedings of the Ninth Conference on Machine Translation, 1272–1288. Association for Computational Linguistics. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmidtova-2026-how-important-is-perfect-english/