Elamur Research⌕
☰
home / papers / 2026 / Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains

Summary

Most quality estimation (QE) metrics for machine translation are built and validated on WMT data, so it is unclear whether they remain reliable on new domains. Earlier out-of-domain studies changed annotators, MT systems or setups between domains, which mixes domain effects with human labeling noise. The authors build CD-ESA, a multi-annotator Error Span Annotation dataset. In it, the same MT systems (GPT-4o, DeepL, Azure MS, X-ALMA, Tower-Instruct-v2 and a Tower model fine-tuned on PharmaChem) translate three kinds of text: WMT23 news, Emea medical documents filtered with a SciBERT domain classifier, and proprietary PharmaChem shift-worker logs. Annotators stay fixed within each language pair, so metric–human agreement can be compared directly with inter-annotator agreement.

Raw segment-level scores make the metrics look robust. Once annotations are averaged or segments are grouped, clear gaps appear on unseen domains. There, metrics underrate translation quality and favour weaker open-source systems. Fine-tuned metrics such as xCOMET_QE show this most strongly, while LLM-as-a-judge methods hold up better. The main recommendation is to judge metrics against human–human agreement measured under identical conditions.

Key points

  • CD-ESA contains 18.8k error span annotations collected with Label Studio. It covers en-de, en-zh and en-ko, with 2–4 annotators per pair, and includes 1354 translations. The WMT23 and Emea portions are public on Hugging Face (FinnSchmidt/CD-ESA), but PharmaChem cannot be released.
  • Error density differs by domain, with 2.8 errors per segment on PharmaChem, 1.4 on WMT23 and 0.8 on Emea. Emea segments are also shorter, averaging 22.4 tokens versus about 38 for the other two domains.
  • At the segment level, metric–human agreement reaches up to 0.69. Averaging human annotations raises inter-annotator agreement by up to +0.11.
  • On PharmaChem, inter-annotator agreement reaches 0.96, while metrics reach about 0.78–0.83 (the introduction reports up to 0.87). Metrics underestimate quality by as much as 0.37 normalized points.
  • xCOMET_QE over-predicts errors on unseen domains; on PharmaChem its span precision is 0.23 and its recall is 0.80. Its embeddings cluster by domain rather than by quality.
  • The paper is internally inconsistent on some details. The Limitations section mentions only en-de, and the Conclusion says five MT systems instead of six.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.findings-acl.1145

@inproceedings{schmidt2026who,
  title = {{Who Watches the Watchmen? Humans Disagree With Translation Metrics on Unseen Domains}},
  author = {Schmidt, Finn and Wahle, Jan Philip and Ruas, Terry and Gipp, Bela},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2026},
  year = {2026},
  doi = {10.18653/v1/2026.findings-acl.1145},
  url = {https://doi.org/10.18653/v1/2026.findings-acl.1145}
}

References

46 references, 21 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sweta Agrawal, António Farinhas, Ricardo Rei, and André F. T. Martins. 2024. Can automatic metrics assess high-quality translations? Preprint, arXiv:2405.18348. ↗
  2. Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the cross-lingual transferability of monolingual representations. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4623–4637. find our card
  3. Iz Beltagy, Kyle Lo, and Arman Cohan. 2019. Scibert: A pretrained language model for scientific text. arXiv preprint arXiv:1903.10676. ↗ our card
  4. Anja Belz, Craig Thomson, Ehud Reiter, Gavin Abercrombie, Jose M Alonso-Moral, Mohammad Arvan, Anouck Braggaar, Mark Cieliebak, Elizabeth Clark, Kees van Deemter, and 1 others. 2023. Missing information, unresponsive authors, experimental flaws: The impossibility of assessing the reproducibility of previous human evaluations in nlp. In Proceedings of the Fourth Workshop on Insights from Negative Results in NLP, pages 1–10. find
  5. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  6. Erenay Dayanık and Sebastian Padó. 2021. Disentangling document topic and author gender in multiple languages: Lessons for adversarial debiasing. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis, pages 50–61. find
  7. DeepL SE. 2024. Deepl translator. https://www.deepl.com/translator. ↗
  8. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. arXiv preprint arXiv:2305.14324. ↗
  9. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. Preprint, arXiv:2308.07286. ↗ our card
  10. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. ↗ our card
  11. Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:811–825. ↗
  12. Markus Freitag, Nitika Mathur, Daniel Deutsch, Chi-Kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Frederic Blain, Tom Kocmi, Jiayi Wang, David Ifeoluwa Adelani, Marianna Buchicchio, Chrysoula Zerva, and Alon Lavie. 2024. Are LLMs breaking MT metrics? results of the WMT24 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 47–81, Miami, Florida, USA. Association for Computational Linguistics. find our card
  13. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, and 1 others. 2023. Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628. find our card
  14. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André FT Martins. 2022b. Results of wmt22 metrics shared task: Stop using bleu–neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68. find our card
  15. Yvette Graham, Timothy Baldwin, and Nitika Mathur. 2015. Accurate evaluation of segment-level machine translation metrics. In North American Chapter of the Association for Computational Linguistics. find our card
  16. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th linguistic annotation workshop and interoperability with discourse, pages 33–41. find
  17. Nuno M Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André FT Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  18. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics. find our card
  19. Tom Kocmi and Christian Federmann. 2023a. Gemba-mqm: Detecting translation quality error spans with gpt-4. arXiv preprint arXiv:2310.13988. ↗ our card
  20. Tom Kocmi and Christian Federmann. 2023b. Large language models are state-of-the-art evaluators of translation quality. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 193–203, Tampere, Finland. European Association for Machine Translation. ↗
  21. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. Preprint, arXiv:2107.10821. ↗
  22. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024. Error span annotation: A balanced approach for human evaluation of machine translation. arXiv preprint arXiv:2406.11580. ↗ our card
  23. Alon Lavie, Greg Hanneman, Sweta Agrawal, Diptesh Kanojia, Chi-Kiu Lo, Vilém Zouhar, Frederic Blain, Chrysoula Zerva, Eleftherios Avramidis, Sourabh Deoghare, and 1 others. 2025. Findings of the wmt25 shared task on automated translation evaluation systems: Linguistic diversity is challenging and references still help. In Proceedings of the Tenth Conference on Machine Translation, pages 436–483. find
  24. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. find our card
  25. Microsoft. 2024. Microsoft translator documentation. https://learn.microsoft.com/azure/ ai-services/translator/. ↗
  26. OpenAI. 2024. Gpt-4o system card. https://openai.com/research/gpt-4o-system-card. ↗ our card
  27. Stefano Perrella, Lorenzo Proietti, Alessandro Scirè, Edoardo Barba, and Roberto Navigli. 2024. Guardians of the machine translation meta-evaluation: Sentinel metrics fall in! arXiv preprint arXiv:2408.13831. ↗ our card
  28. Barbara Plank. 2022. The’problem’of human label variation: On ground truth in data, modeling and evaluation. arXiv preprint arXiv:2211.02570. ↗ our card
  29. José Pombal, Nuno M Guerreiro, Ricardo Rei, and André FT Martins. 2025. Adding chocolate to mint: Mitigating metric interference in machine translation. Transactions of the Association for Computational Linguistics, 13:1319–1339. ↗
  30. Lorenzo Proietti, Stefano Perrella, and Roberto Navigli. 2025. Has machine translation evaluation achieved human parity? the human reference and the limits of progress. arXiv preprint arXiv:2506.19571. ↗
  31. Ricardo Rei, Nuno M Guerreiro, Daan Van Stigt, Marcos Treviso, Luísa Coheur, José GC de Souza, André FT Martins, and 1 others. 2023. Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task. In Proceedings of the Eighth Conference on Machine Translation, pages 841–848. find our card
  32. Ricardo Rei, José Pombal, Nuno M Guerreiro, João Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, and 1 others. 2024. Tower v2: Unbabel-ist 2024 submission for the general mt shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 185–204. find
  33. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025. ↗ our card
  34. Gabriele Sarti, Vilém Zouhar, Malvina Nissim, and Arianna Bisazza. 2025. Unsupervised word-level quality estimation for machine translation through the lens of annotators (dis)agreement. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, page 18320–18337. Association for Computational Linguistics. find
  35. Amanpreet Singh, Mike D’Arcy, Arman Cohan, Doug Downey, and Sergey Feldman. 2023. Scirepeval: A multi-format benchmark for scientific document representations. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 5548–5566. find
  36. Lucia Specia, Nicola Cancedda, Marc Dymetman, Marco Turchi, and Nello Cristianini. 2009. Estimating the sentence-level quality of machine translation systems. In European Association for Machine Translation Conferences/Workshops. find our card
  37. Emma Steigerwald, Valeria Ramírez-Castañeda, Débora Y. C. Brandt, András Báldi, J. Shapiro, Lynne Bowker, and Rebecca D. Tarvin. 2022. Overcoming language barriers in academia: Machine translation tools and a vision for a multilingual future. Bioscience, 72:988 – 998. ↗
  38. Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. ArXiv, abs/1409.3215. ↗
  39. Shaomu Tan and Christof Monz. 2025. Remedy: Learning machine translation evaluation from human preferences with reward modeling. arXiv preprint arXiv:2504.13630. ↗
  40. Brian Thompson, Nitika Mathur, Daniel Deutsch, and Huda Khayrallah. 2024. Improving statistical significance in human evaluation of automatic metrics via soft pairwise accuracy. arXiv preprint arXiv:2409.09598. ↗
  41. Jörg Tiedemann. 2012. Parallel data, tools and interfaces in opus. In Lrec, volume 2012, pages 2214–2218. find
  42. Christian Tomani, David Vilar, Markus Freitag, Colin Cherry, Subhajit Naskar, Mara Finkelstein, Xavier Garcia, and Daniel Cremers. 2024. Quality-aware translation models: Efficient generation and quality estimation in a single model. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 15660–15679. ↗
  43. J. Wahle, T. Ruas, S. M. Mohammad, N. Meuschke, and B. Gipp. 2023. Ai usage cards: Responsibly reporting ai-generated content. In 2023 ACM/IEEE Joint Conference on Digital Libraries (JCDL), pages 282–284, Los Alamitos, CA, USA. IEEE Computer Society. ↗
  44. Haoran Xu, Kenton Murray, Philipp Koehn, Hieu Hoang, Akiko Eriguchi, and Huda Khayrallah. 2024. X-alma: Plug & play modules and adaptive rejection for quality translation at scale. arXiv preprint arXiv:2410.03115. ↗
  45. Ran Zhang, Wei Zhao, and Steffen Eger. 2025. How good are LLMs for literary translation, really? literary translation evaluation with humans and LLMs. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 10961– 10988, Albuquerque, New Mexico. Association for Computational Linguistics. find
  46. Vilém Zouhar, Shuoyang Ding, Anna Currey, Tatyana Badeka, Jenyuan Wang, and Brian Thompson. 2024. Fine-tuned machine translation metrics struggle in unseen domains. arXiv preprint arXiv:2402.18747. ↗ our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/schmidt-2026-who-watches-watchmen-humans-disagree/