Elamur Research⌕
☰
home / papers / 2026 / Generating Difficult-to-Translate Texts

Generating Difficult-to-Translate Texts

Paper · 2026 · Vilém Zouhar, Wenda Xu, Parker Riley et al.

Summary

Machine translation benchmarks built from real-world text become outdated quickly. Current systems translate most of their segments almost perfectly, so the test sets can no longer rank strong models or reveal their weak spots. Existing fixes have drawbacks. Selecting a hard subset of existing data finds too few naturally difficult cases. Having an LLM write hard examples from scratch gives texts that are too easy or too uniform. The authors propose MT-breaker, which copies how human experts probe a system. An LLM (Gemini 2.5-pro) starts from a seed text, reads the target MT model's translation and rewrites the source to be harder, over about ten rounds. At the end, the version whose translation gets the worst quality-estimation score is kept. Experiments use 100 English seeds from WMT 2024, five target languages and four MT systems. MT-breaker produces much harder inputs than the seeds and zero-shot generation while keeping the diversity of natural text. Expert MQM annotation confirms this. The difficulty also partly carries over to other systems and languages, which makes the method useful for targeted stress-testing during model development.

Key points

  • Setup: English→{Czech, German, Chinese, Vietnamese, Polish}, with 100 randomly drawn WMT 2024 sources as seeds.
  • Target MT systems: Google Translate, Gemini 2.5-flash, Gemini 2.0-flash and Gemma 3-27b-it.
  • Difficulty signal: the mean of MetricX-24-QE and a prompted LLM-as-QE, on a 0–100 scale where difficulty equals 100 minus QE.
  • Compared methods: Seeds, Zeroshot (plain, history, min) and MT-breaker (seeded, seedless, each optionally with QE feedback).
    • Translation quality falls with each iteration and levels off near step 10.
    • Showing QE scores to the LLM does not help the seeded variant and hurts the seedless one. The authors attribute this to lower output variance (σ²=297 vs. 325 without QE).
  • Recommended use: the generated data is deliberately biased toward the targeted model's weaknesses, so it should not serve as a fair cross-model benchmark. Instead, use it to find and hill-climb on one model's failures, or to compare models' worst-case behaviour by breaking each one separately.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓ETH ZurichZurich · Switzerland72 works · 5,176 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

Cite

Cite as Vilém Zouhar, Wenda Xu, Parker Riley, Juraj Juraska, Mara Finkelstein, Markus Freitag and Daniel Deutsch. 2026. Generating Difficult-to-Translate Texts. Proceedings of the First Workshop on Multilingual Multicultural Evaluation.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.mme-main.14

@inproceedings{zouhar2026generating,
  title = {{Generating Difficult-to-Translate Texts}},
  author = {Vilém Zouhar, Wenda Xu, Parker Riley, Juraj Juraska, Mara Finkelstein, Markus Freitag and Daniel Deutsch},
  booktitle = {Proceedings of the First Workshop on Multilingual Multicultural Evaluation},
  year = {2026},
  doi = {10.18653/v1/2026.mme-main.14},
  url = {https://doi.org/10.18653/v1/2026.mme-main.14}
}

References

23 references, 12 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Chantal Amrhein, Nikita Moghe, and Liane Guillou. 2022. ACES: Translation accuracy challenge sets for evaluating machine translation metrics. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 479–513. Association for Computational Linguistics.
  2. Bjarki Ármannsson, Hinrik Hafsteinsson, Atli Jasonarson, and Steinthor Steingrimsson. 2024. Killing two flies with one stone: An attempt to break LLMs using English-Icelandic idioms and proper names. In Proceedings of the Ninth Conference on Machine Translation, pages 451–458. Association for Comput… find
  3. Xiaoyu Chen, Daimeng Wei, Zhanglin Wu, Ting Zhu, Hengchao Shang, Zongyao Li, Jiaxin Guo, Ning Xie, Lizhi Lei, Hao Yang, and Yanfei Jiang. 2023. Multifaceted challenge set for evaluating machine translation performance. In Proceedings of the Eighth Conference on Machine Translation, pages 217–223. As… find
  4. Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, Luke Marris, Sam Petulla, Colin Gaffney, Asaf Aharoni, Nathan Lintz, Tiago Cardal Pais, Henrik Jacobsson, Idan Szpektor, Nan-Jiang Jiang, and 3290 other…
  5. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474.
  6. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared…
  7. Pierre Isabelle, Colin Cherry, and George Foster. 2017. A challenge set approach to evaluating machine translation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2486–2496. Association for Computational Linguistics.
  8. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504. Association for Computational Linguistics.
  9. Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 756–767. Association for Computational Linguistic…
  10. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popov… — Findings of the WMT24 General Machine Translation Shared Task: The LLM Era Is Here but MT Is Not Solved Yet
  11. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024b. Error span annotation: A balanced approach for human evaluation of machine translation. In Proceedings of the Ninth Conference on Machine Translation, pa…
  12. Clayton Lewis and John Rieman. 1993. Task-Centered User Interface Design: A Practical Introduction. University of Colorado, Boulder, CO. find
  13. Cong Lu, Shengran Hu, and Jeff Clune. 2025. Automated capability discovery via foundation model self-exploration. Preprint, arXiv:2502.07577. — [arXiv:2502.07577](https://arxiv.org/abs/2502.07577) find
  14. Shushen Manakhimova, Eleftherios Avramidis, Vivien Macketanz, Ekaterina Lapshinova-Koltunski, Sergei Bagdasarov, and Sebastian Möller. 2023. Linguistically motivated evaluation of the 2023 state-of-the-art machine translation: Can ChatGPT outperform NMT? In Proceedings of the Eighth Conference on Ma… find
  15. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Edoardo Barba, and Roberto Navigli. 2024. Guardians of the machine translation meta-evaluation: Sentinel metrics fall in! In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 16…
  16. José Pombal, Nuno M. Guerreiro, Ricardo Rei, and André F. T. Martins. 2025. Zero-shot benchmarking: A framework for flexible and scalable automatic evaluation of language models. Preprint, arXiv:2504.01001. — [arXiv:2504.01001](https://arxiv.org/abs/2504.01001) find
  17. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395. Association for Computational Linguistics.
  18. Lorenzo Proietti, Stefano Perrella, Vilém Zouhar, Roberto Navigli, and Tom Kocmi. 2025. Estimating machine translation difficulty. Preprint, arXiv:2508.10175. find
  19. Lindia Tjuatja and Graham Neubig. 2025. BehaviorBox: Automated discovery of fine-grained performance differences between language models. Preprint, arXiv:2506.02204. find
  20. Xinze Zhang, Junzhe Zhang, Zhenhua Chen, and Kun He. 2021. Crafting adversarial examples for neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Lo… find
  21. Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, and Jiajun Chen. 2020. A reinforced generation of adversarial examples for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3486–3497. Association for Computational Linguistics. find
  22. Vilém Zouhar, Peng Cui, and Mrinmaya Sachan. 2025. How to select datapoints for efficient human evaluation of nlg models? Preprint, arXiv:2501.18251. - syntax complexity: on a scale from 0 (simplest possible sentence) to 100 (most complex and hard to understand). - topics: list of 1 to 5 topics that…
  23. Provide only the output in JSON and nothing else. The output should look like this (no extra backticks or newlines): *Figure 4: Results for difficulty for translation into English, compare to diagonal in Table 5.* find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zouhar-2026-generating-difficult-translate-texts/