Elamur Research⌕
☰
home / papers / 2022 / MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem

MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem

Summary

Learned regression metrics for machine translation, such as COMET, BLEURT and C-SPEC, correlate well with human judgements. However, they output a single number with no explanation. This paper, submitted to the WMT 2022 Metrics Shared Task, brings the MQM human-evaluation protocol into automatic evaluation. MaTESe treats quality estimation as sequence tagging: each token of a candidate translation is labelled as part of a Major error, a Minor error or no error. The sentence score is the sum of span penalties. The model reads the candidate together with its reference, while the MaTESe-QE variant reads the source sentence instead, so it needs no reference. Training uses the public MQM annotations of Freitag et al. (2021a), simplified to severity only, with the three raters' annotations merged per sentence. The authors also propose two span-level diagnostics. On WMT 2021 news data, the metrics match or beat the leading submissions for English→German and come close for Chinese→English. Accuracy falls on out-of-domain TED data. The approach matters because it produces interpretable error locations rather than an opaque score.

Key points

  • Scoring: a Major span costs −5 and a Minor span −1. Sentence scores are capped at a minimum of −25, and corpus scores are sentence averages.
  • Pre-processing:
    • Non-translation, Omission, Source-error and Neutral annotations are dropped.
    • Critical severity becomes Major, and error categories are discarded.
    • When merged spans overlap, the Major span is kept over the Minor one.
  • New span metrics: TSH (Target Span Hit) and HSH (Hypothesis Span Hit) score how far predicted error spans partially coincide with gold ones, using micro-averaging.
  • Backbones tested: XLM-RoBERTa, mBART and RoBERTa. XLM-RoBERTa Large clearly outperforms its Base version.
  • Data limitations: the training data has about 80K annotated sentences but only about 6K unique sources. The authors blame this for the TED drop. MaTESe-QE trails the reference-based MaTESe only slightly.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

29 papers of our library cite it (of the 140673 whose bibliographies we read); 11 of them have a page here, 18 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2022.wmt-1.51

@inproceedings{perrella2022matese,
  title = {{MaTESe: Machine Translation Evaluation as a Sequence Tagging Problem}},
  author = {Perrella, Stefano and Proietti, Lorenzo and Scirè, Alessandro and Campolungo, Niccolò and Navigli, Roberto},
  booktitle = {Proceedings of the Seventh Conference on Machine Translation (WMT)},
  year = {2022},
  doi = {10.18653/v1/2022.wmt-1.51},
  url = {https://doi.org/10.18653/v1/2022.wmt-1.51}
}

References

20 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics. find our card
  2. Hyung Won Chung, Thibault Fevry, Henry Tsai, Melvin Johnson, and Sebastian Ruder. 2021. Rethinking embedding coupling in pre-trained language models. In International Conference on Learning Representations. ↗ our card
  3. Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshop and Interoperability with Discourse, pages 33–41, Sofia, Bulgaria. Association for Computational Linguistics. find
  4. Lifeng Han, Irina Sorokina, Gleb Erofeev, and Serge Gladkoff. 2021. cushLEPOR: customising hLEPOR metric using optuna for higher agreement with human judgments or pre-trained language model LaBSE. In Proceedings of the Sixth Conference on Machine Translation, pages 1014–1023, Online. Association for Computational Linguistics. ↗
  5. Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. 2021a. Just ask! evaluating machine translation by asking and answering questions. In Proceedings of the Sixth Conference on Machine Translation, pages 495–506, Online. Association for Computational Linguistics. find
  6. Mateusz Krubiński, Erfan Ghadery, Marie-Francine Moens, and Pavel Pecina. 2021b. MTEQA at WMT21 metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1024–1029, Online. Association for Computational Linguistics. find
  7. Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726–742. ↗
  8. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. find
  9. Arle Lommel, Aljoscha Burchardt, and Hans Uszkoreit. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing translation quality metrics. Tradumàtica: tecnologies de la traducció, 0:455–463. ↗
  10. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. USA. Association for Machine Translation in the Americas. find our card
  12. Michal Stefanik, Vít Novotný, and Petr Sojka. 2021. Regressive ensemble for machine translation quality evaluation. In Proceedings of the Sixth Conference on Machine Translation, pages 1041–1048, Online. Association for Computational Linguistics. ↗
  13. Kosuke Takahashi, Yoichi Ishibashi, Katsuhito Sudoh, and Satoshi Nakamura. 2021. Multilingual machine translation evaluation metrics fine-tuned on pseudo-negative examples for WMT 2021 metrics task. In Proceedings of the Sixth Conference on Machine Translation, pages 1049–1052, Online. Association for Computational Linguistics. find
  14. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc. find
  15. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  16. Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In Third Workshop on Very Large Corpora. find
  17. Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-IST 2021 submission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030–1040, Online. Association for Computational Linguistics. find our card
  18. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  19. Thibault Sellam, Amy Pu, Hyung Won Chung, Sebastian Gehrmann, Qijun Tan, Markus Freitag, Dipanjan Das, and Ankur Parikh. 2020. Learning to evaluate translation beyond English: BLEURT submissions to the WMT metrics 2020 shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 921–927, Online. Association for Computational Linguistics. find our card
  20. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, find our card

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/perrella-2022-matese-machine-translation-evaluation-as/