Elamur Research
☰
home / papers / 2024 / Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation

Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2024.wmt-1.131.pdf

Summary

Human evaluation remains the reference standard for judging machine translation, but current protocols involve a trade-off. MQM gives detailed error categories but requires trained experts and is slow and costly. This is a particular problem for low-resource languages, where such experts are scarce. Direct Assessment (DA+SQM) is quick and cheap, but its scores are noisy. The authors propose Error Span Annotation (ESA), which combines the two approaches. Annotators first highlight problematic parts of the translation and label each one only as minor or major. They then give the whole segment a 0–100 score. Marking errors first is intended to make the final score more grounded. The authors built ESA in the Appraise tool and tested it against MQM and DA+SQM on English→German WMT23 outputs from 12 MT systems plus one human reference. ESA ranked systems as usefully as MQM, showed higher inter- and intra-annotator agreement, and was noticeably faster. It also works without MQM-trained experts, so large-scale evaluation becomes cheaper and more accessible across languages.

Key points

  • Protocol design: ESA uses only two severity levels (minor and major) and no error types. Spans are marked on the target side, with a special tag for omissions. This choice keeps the protocol compatible with future audio and video translation.
  • Speed: Pooled across annotators, the median time per paragraph fell from 38 s (MQM) to 29 s (ESA), a 23% reduction. Using the average of each annotator's median, the drop was from 49 s to 34 s, or 32%.
  • Scoring method: The authors criticise the usual MQM-style formula (−5 per major error, −1 per minor error) because it does not scale with text length. They also show that scoring from spans alone works worse than combining spans with the annotator's direct score.
  • Cost: MQM experts in the study earned double the standard translator wage, while ESA needs no such specialists.
  • Scope and availability: Experiments covered only English→German. Code and collected data are released at github.com/wmt-conference/ErrorSpanAnnotation.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

75 papers of our library cite it (of the 140673 whose bibliographies we read); 3 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.wmt-1.131

@inproceedings{kocmi2024error,
  title = {{Error Span Annotation: A Balanced Approach for Human Evaluation of Machine Translation}},
  author = {Kocmi, Tom and Zouhar, Vilém and Avramidis, Eleftherios and Grundkiewicz, Roman and Karpinska, Marzena and Popović, Maja and Sachan, Mrinmaya and Shmatova, Mariya},
  booktitle = {Proceedings of the Ninth Conference on Machine Translation},
  year = {2024},
  doi = {10.18653/v1/2024.wmt-1.131},
  url = {https://doi.org/10.18653/v1/2024.wmt-1.131}
}

References

29 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. ALPAC. 1966. Language and machines. Computers in translation and linguistics. find
  2. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 conference on machine translation (WMT17). In Proceedings of the Second Conference on Machine Translation, 169–214. Association for Computational Linguistics. find our card
  3. Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lucia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. 2016. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers, 131–198. Association for Computational Linguistics. find our card
  4. Aljoscha Burchardt. 2013. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35. Aslib. find our card
  5. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2007. (meta-) evaluation of machine translation. In Proceedings of the Second Workshop on Statistical Machine Translation, 136–158. Association for Computational Linguistics. find
  6. Chris Callison-Burch, Cameron Fordyce, Philipp Koehn, Christof Monz, and Josh Schroeder. 2008. Further meta-evaluation of machine translation. In Proceedings of the Third Workshop on Statistical Machine Translation, 70–106. Association for Computational Linguistics. find
  7. Sheila Castilho. 2020. On the same page? comparing inter-annotator agreement in sentence and document level human machine translation evaluation. In Proceedings of the Fifth Conference on Machine Translation, 1150–1159. Association for Computational Linguistics. find
  8. Christian Federmann. 2018. Appraise evaluation framework for machine translation. In Proceedings of the 27th International Conference on Computational Linguistics: System Demonstrations, 86–88. Association for Computational Linguistics. find
  9. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021a. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. find our card
  10. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, 578–628. Association for Computational Linguistics. find our card
  11. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondřej Bojar. 2021b. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, 733–774. Association for Computational Linguistics. find our card
  12. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, 33–41. Association for Computational Linguistics. find
  13. Filip Klubička, Antonio Toral, and Víctor M. Sánchez-Cartagena. 2018. Quantitative Fine-grained Human Evaluation of Machine Translation Systems: A Case Study on English to Croatian. Machine Translation, 32(3):195–215. find
  14. Rebecca Knowles and Chi-kiu Lo. 2024. Calibration and context in human evaluation of machine translation. Natural Language Processing, 1–25. find
  15. Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thamme Gowda, Roman Grundkiewicz, Barry Haddow, Philipp Koehn, Benjamin Marie, Christof Monz, Makoto Morishita, Kenton Murray, Makoto Nagata, Toshiaki Nakazawa, Martin Popel, Maja Popović, and Mariya Shmatova. 2023. Findings of the 2023 conference on machine translation (WMT23): LLMs are here but not quite there yet. In Proceedings of the Eighth Conference on Machine Translation, 1–42. Association for Computational Linguistics. find
  16. Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popović. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT), 1–45. Association for Computational Linguistics. find
  17. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, 478–494. Association for Computational Linguistics. find
  18. Philipp Koehn and Christof Monz. 2006. Manual and automatic evaluation of machine translation between European languages. In Proceedings on the Workshop on Statistical Machine Translation, 102–121. Association for Computational Linguistics. find
  19. Julia Kreutzer, Nathaniel Berger, and Stefan Riezler. 2020. Correct me if you can: Learning from error corrections and markings. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, 135–144. European Association for Machine Translation. find
  20. Arle Lommel, Aljoscha Burchardt, Maja Popović, Kim Harris, Eleftherios Avramidis, and Hans Uszkoreit. 2014. Using a new analytic measure for the annotation and analysis of MT errors on real data. In Proceedings of the 17th Annual Conference of the European Association for Machine Translation, 165– 172. European Association for Machine Translation. find
  21. Marianna Martindale and Marine Carpuat. 2018. Fluency over adequacy: A pilot study in measuring user trust in imperfect MT. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), 13–25. Association for Machine Translation in the Americas. find
  22. Maja Popović. 2020. Informative manual evaluation of machine translation output. In Proceedings of the 28th International Conference on Computational Linguistics, 5059–5069. International Committee on Computational Linguistics. find
  23. Parker Riley, Daniel Deutsch, George Foster, Viresh Ratnakar, Ali Dabirmoghaddam, and Markus Freitag. 2024. Finding replicable human evaluations via stable ranking probability. arXiv preprint arXiv:2404.01474. ↗
  24. Paul Rottger, Bertie Vidgen, Dirk Hovy, and Janet Pierrehumbert. 2022. Two contrasting data annotation paradigms for subjective NLP tasks. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 175–190. Association for Computational Linguistics. find our card
  25. David Vilar, Gregor Leusch, Hermann Ney, and Rafael E. Banchs. 2007. Human evaluation of machine translation through binary system comparisons. In Proceedings of the Second Workshop on Statistical Machine Translation, 96–103. Association for Computational Linguistics. find
  26. David Vilar, Jia Xu, Luis Fernando D’Haro, and Hermann Ney. 2006. Error analysis of statistical machine translation output. In Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC’06). European Language Resources Association (ELRA). find our card
  27. Johnny Wei, Tom Kocmi, and Christian Federmann. 2022. Searching for a higher power in the human evaluation of MT. In Proceedings of the Seventh Conference on Machine Translation (WMT), 129– 139. Association for Computational Linguistics. find
  28. John S. White, Theresa A. O’Connell, and Francis E. O’Mara. 1994. The ARPA MT evaluation methodologies: Evolution, lessons, and future approaches. In Proceedings of the First Conference of the Association for Machine Translation in the Americas. find
  29. Vilém Zouhar, Tom Kocmi, and Mrinmaya Sachan. 2024. AI-assisted human evaluation of machine translation. find