Elamur Research⌕
☰
home / papers / 2026 / MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation

MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation

Summary

As machine translation systems improve, noise in human evaluation can hide the real quality differences between them. This paper tests a two-pass variant of MQM, called MQM re-annotation. An expert rater opens a document that already contains error spans and can edit, delete or add errors. The prior spans come from the rater's own earlier pass, from a different human, or from an automatic MQM system. The authors used WMT 2023 Metrics Shared Task News data in Chinese-English and English-German, the Anthea interface, and professional translators as raters. Because no true ground truth exists, they judged quality by agreement between disjoint rater groups.

Raters edited their own annotations the least, other humans' annotations more, and automatic output the most. Most of the gain came from catching errors missed the first time. Re-annotating human ratings raised cross-group agreement. Starting from LLM-produced spans also beat annotating from scratch, at no extra human cost. Human-on-human re-annotation was best overall, but it costs roughly twice the labor. The method is especially useful for building reference data to meta-evaluate span-level metrics.

Key points

  • Data: Chinese-English has 247 segments, 25 documents, 16 systems and 8 raters. English-German has 100 segments, 29 documents, 13 systems and 10 raters. Each segment received 3 initial human MQM annotations.
  • Automatic systems:
    • GEMBA-MQM, which prompts GPT-4.
    • An AutoMQM variant built by fine-tuning Gemini 1.0 Pro on WMT 2020–2022 MQM data.
    • Both work without a reference translation.
  • Agreement metrics: Pairwise Ranking Agreement (PRA) and character F1. On segment-level pairwise agreement:
    • Single vs. Single scored 0.511 (zh-en) and 0.574 (en-de).
    • Self vs. Other re-annotations reached 0.553 and 0.641.
    • Auto vs. Auto re-annotations reached 0.546 and 0.631.
  • Artificial spans: Fake Major-severity errors were injected to check whether raters over-trust prior spans. Most were removed, but a few raters often kept them unchanged.
  • Release and limits: The data is public in the google/wmt-mqm-human-evaluation repository (generalMT2023). The study covers only expert raters, two reasonably strong automatic systems, and the News domain.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2026.acl-long.1553

@inproceedings{riley2026mqm,
  title = {{MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation}},
  author = {Riley, Parker and Deutsch, Daniel and Finkelstein, Mara and DiIanni, Colten and Juraska, Juraj and Freitag, Markus},
  booktitle = {Proceedings of the 64th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.1553},
  url = {https://doi.org/10.18653/v1/2026.acl-long.1553}
}

References

23 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Frederic Blain, Chrysoula Zerva, Ricardo Rei, Nuno M. Guerreiro, Diptesh Kanojia, José G. C. de Souza, Beatriz Silva, Tânia Vaz, Yan Jingxuan, Fatemeh Azadi, Constantin Orasan, and André Martins. 2023. Findings of the WMT 2023 shared task on quality estimation. In Proceedings of the Eighth Conference on Machine Translation, pages 629–653, Singapore. Association for Computational Linguistics. find our card
  2. Daniel Deutsch, George Foster, and Markus Freitag. 2023. Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 12914– 12929, Singapore. Association for Computational Linguistics. find
  3. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André Martins, Graham Neubig, Ankush Garg, Jonathan Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. In Proceedings of the Eighth Conference on Machine Translation, pages 1066– 1083, Singapore. Association for Computational Linguistics. ↗ our card
  4. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. ↗ our card
  5. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  6. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pages 733–774, Online. Association for Computational Linguistics. find our card
  7. Gemini Team. 2024. Gemini: A family of highly capable multimodal models. Preprint, arXiv:2312.11805. ↗ our card
  8. Nuno M. Guerreiro, Ricardo Rei, Daan van Stigt, Luisa Coheur, Pierre Colombo, and André F. T. Martins. 2024. xcomet: Transparent machine translation evaluation through fine-grained error detection. Transactions of the Association for Computational Linguistics, 12:979–995. ↗ our card
  9. Juraj Juraska, Daniel Deutsch, Mara Finkelstein, and Markus Freitag. 2024. MetricX-24: The Google submission to the WMT 2024 metrics shared task. In Proceedings of the Ninth Conference on Machine Translation, pages 492–504, Miami, Florida, USA. Association for Computational Linguistics. find our card
  10. Rebecca Knowles. 2021. On the stability of system rankings at WMT. In Proceedings of the Sixth Conference on Machine Translation, pages 464–477, Online. Association for Computational Linguistics. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Koehn, Benjamin Marie, Christof Monz, Kenton Murray, Masaaki Nagata, Martin Popel, Maja Popović, and 3 others. 2024a. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In Proceedings of the Ninth Conference on Machine Translation, pages 1–46, Miami, Florida, USA. Association for Computational Linguistics. ↗ our card
  11. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, and 2 others. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, pages 1–42, Singapore. Association for Computational Linguistics. find
  12. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 1–45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find
  13. Tom Kocmi and Christian Federmann. 2023. GEMBA-MQM: Detecting translation quality error spans with GPT-4. In Proceedings of the Eighth Conference on Machine Translation, pages 768–775, Singapore. Association for Computational Linguistics. find our card
  14. Tom Kocmi, Vilém Zouhar, Eleftherios Avramidis, Roman Grundkiewicz, Marzena Karpinska, Maja Popović, Mrinmaya Sachan, and Mariya Shmatova. 2024b. Error span annotation: A balanced approach for human evaluation of machine translation. In Proceedings of the Ninth Conference on Machine Translation, pages 1440–1453, Miami, Florida, USA. Association for Computational Linguistics. ↗ our card
  15. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  16. OpenAI. 2024. Gpt-4 technical report. arXiv:2303.08774. ↗ our card
  17. Preprint, Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  18. Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. 2024. Finding replicable human evaluations via stable ranking probability. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4908–4919, Mexico City, Mexico. Association for Computational Linguistics. ↗
  19. Belén Saldías Fuentes, George Foster, Markus Freitag, and Qijun Tan. 2022. Toward more effective human evaluation for machine translation. In Proceedings of the 2nd Workshop on Human Evaluation of NLP Systems (HumEval), pages 76–89, Dublin, Ireland. Association for Computational Linguistics. ↗
  20. Yixiao Song, Parker Riley, Daniel Deutsch, and Markus Freitag. 2025. Enhancing human evaluation in machine translation with comparative judgement. In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 20536–20551, Vienna, Austria. Association for Computational Linguistics. find
  21. Vilém Zouhar, Tom Kocmi, and Mrinmaya Sachan. 2025. Ai-assisted human evaluation of machine translation. Preprint, arXiv:2406.12419. ↗
  22. Zhongtao Liu, Parker Riley, Daniel Deutsch, Alison Lui, Mengmeng Niu, Apurva Shah, and Markus Freitag. 2024. Beyond human-only: Evaluating human-machine collaboration for collecting high-quality translation data. In Proceedings of the Ninth Conference on Machine Translation, pages 1095–1106, Miami, Florida, USA. Association for Computational Linguistics. find
  23. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/riley-2026-mqm-re-annotation-technique-collaborative/