Elamur Research⌕
☰
home / papers / 2024 / Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains

Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains

Summary

MT evaluation now leans heavily on metrics such as COMET, which start from a pre-trained model and are further trained on human quality ratings. Those ratings come mostly from WMT shared tasks, where domains tend to repeat from year to year. The authors test whether these metrics still work in a domain they never saw during training. They build a biomedical MQM corpus from the outputs of 21 systems submitted to the WMT21 biomedical translation task. It covers 11 translation directions and has roughly 25k segment-level judgments from linguists with medical expertise, along with new professional references that were themselves quality-checked. To compare domains fairly, they measure each metric's segment-level Kendall's τ gain over a Surface-Form baseline (the mean of BLEU, ChrF and TER). Fine-tuned metrics (COMET, BLEURT, UniTE) gain much less on bio than on WMT data, while untuned metrics like BERTScore and PRISM show no such drop. Follow-up experiments attribute the gap to the DA/MQM fine-tuning stages, not to the pre-trained model. This challenges claims of domain robustness and gives the community a public resource for studying the issue.

Key points

  • Dataset size and coverage: the bio MQM dataset has about 25k annotations over 11 directions (Pt→En, En↔De, En↔Es, En↔Ru, En↔Fr, Zh↔En) and 21 systems.
  • Annotation scheme: a Critical severity level was added, scored the same as Major.
  • Splits and release: about 25% of segments per pair serve as train/dev data. The dataset is released at github.com/amazon-science/bio-mqm-dataset.
  • Domain distinctness: bio has the lowest average vocabulary overlap with the WMT domains (0.436), compared with 0.486–0.582 for the e-commerce, news, social and conversation domains.
  • No forgetting effect: retraining COMET (En-De, Zh-En, Ru-En) for more DA/MQM epochs never produces a strong bio metric. This suggests the model is weak on bio from the start rather than losing ability it once had.
  • In-domain data helps a lot:
    • Adding 6k bio judgments raises COMET's bio correlation from 0.287 to 0.328.
    • Even 1k judgments reaches 0.313.
  • Effect of the pre-trained model:
    • Better pre-trained models (XLM-Roberta adapted with MLM/TLM, larger or fine-tuned NLLB) improve BERTScore and PRISM.
    • They do not improve COMET.
  • COMET is still the top scorer: despite the gap, COMET remains the best-performing metric on bio in absolute terms.
  • Stated limitations:
    • Only one unseen domain is studied.
    • Only segment-level evaluation is done.
    • The conclusions rely on two assumptions: that Surface-Form metrics are domain-invariant, and that τ is linear in a metric's true quality.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Machine translation · Natural language processing

Topics: Machine translation

Cited by

28 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 24 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2024.acl-short.45

@inproceedings{zouhar2024fine,
  title = {{Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains}},
  author = {Zouhar, Vilém and Ding, Shuoyang and Currey, Anna and Badeka, Tatyana and Wang, Jenyuan and Thompson, Brian},
  booktitle = {Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)},
  year = {2024},
  doi = {10.18653/v1/2024.acl-short.45},
  url = {https://doi.org/10.18653/v1/2024.acl-short.45}
}

References

33 references, 14 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Yujin Baek, Zae Myung Kim, Jihyung Moon, Hyun-joong Kim, and Eunjeong Park. 2020. PATQUEST: Papago translation quality estimation. In Proceedings of the Fifth Conference on Machine Translation, pages 991–998, Online. Association for Computational Linguistics. find
  2. José G. C. de Souza, Marco Turchi, and Matteo Negri. 2014. Machine translation quality estimation across domains. In Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers, pages 409–420, Dublin, Ireland. Dublin City University and Association for Computational Linguistics. find
  3. Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Unsupervised cross-lingual representation learning at scale. CoRR, abs/1911.02116. ↗
  4. Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Graham Neubig, Ankush Garg, Jonathan H. Clark, Markus Freitag, and Orhan Firat. 2023. The devil is in the errors: Leveraging large language models for fine-grained machine translation evaluation. ↗ our card
  5. Johann Frei and Frank Kramer. 2023. German medical named entity recognition model and data set creation using machine translation and word alignment: Algorithm development and validation. JMIR Form Res, 7:e39077. ↗
  6. Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9:1460–1474. ↗ our card
  7. Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation, pages 578–628, Singapore. Association for Computational Linguistics. find our card
  8. Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022. Results of WMT22 metrics shared task: Stop using BLEU – neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46–68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  9. Ian J Goodfellow, Mehdi Mirza, Da Xiao, Aaron Courville, and Yoshua Bengio. 2013. An empirical investigation of catastrophic forgetting in gradient-based neural networks. arXiv preprint arXiv:1312.6211. ↗
  10. Dam Heo, WonKee Lee, Baikjin Jung, and Jong-Hyeok Lee. 2021. Quality estimation using dual encoders with transfer learning. In Proceedings of the Sixth Conference on Machine Translation, pages 920–927, Online. Association for Computational Linguistics. find
  11. Tom Kocmi and Christian Federmann. 2023. Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520. ↗
  12. Samuel Läubli, Rico Sennrich, and Martin Volk. 2018. Has machine translation achieved human parity? a case for document-level evaluation. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4791–4796, Brussels, Belgium. Association for Computational Linguistics. find our card
  13. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. find our card
  14. Aurélie Névéol, Antonio Jimeno Yepes, and Mariana Neves. 2020. MEDLINE as a parallel corpus: a survey to gain insight on French-, Spanish- and Portuguese-speaking authors’ abstract writing practice. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3676– 3682, Marseille, France. European Language Resources Association. find
  15. NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation. ↗
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics. ↗ our card
  17. Maja Popović. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  18. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  19. Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7881–7892, Online. Association for Computational Linguistics. find
  20. Javad Pourmostafa Roshan Sharami, Dimitar Shterionov, Frédéric Blain, Eva Vanmassenhove, Mirella De Sisto, Chris Emmery, and Pieter Spronck. 2023. Tailoring domain adaptation for machine translation quality estimation. In Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pages 9–20, Tampere, Finland. European Association for Machine Translation. ↗
  21. Matthew Snover, Bonnie Dorr, Rich Schwartz, Linnea Micciulla, and John Makhoul. 2006. A study of translation edit rate with targeted human annotation. In Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, pages 223–231, Cambridge, Massachusetts, USA. Association for Machine Translation in the Americas. find our card
  22. Tianxiang Sun, Junliang He, Xipeng Qiu, and Xuanjing Huang. 2022. BERTScore is unfair: On social bias in language model-based metrics for text generation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 3726–3739, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics. find
  23. Brian Thompson, Mehak Preet Dhaliwal, Peter Frisch, Tobias Domhan, and Marcello Federico. 2024. A shocking amount of the web is machine translated: Insights from multi-way parallelism. arXiv preprint arXiv:2401.05749. ↗
  24. Brian Thompson, Jeremy Gwinnup, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019a. Overcoming catastrophic forgetting during domain adaptation of neural machine translation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2062–2068, Minneapolis, Minnesota. Association for Computational Linguistics. find
  25. Brian Thompson, Rebecca Knowles, Xuan Zhang, Huda Khayrallah, Kevin Duh, and Philipp Koehn. 2019b. HABLex: Human annotated bilingual lexicons for experiments in machine translation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1382–1387, Hong Kong, China. Association for Computational Linguistics. find
  26. Brian Thompson and Matt Post. 2020a. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 90–121, Online. Association for Computational Linguistics. find our card
  27. Brian Thompson and Matt Post. 2020b. Paraphrase generation as zero-shot multilingual translation: Disentangling semantic similarity from lexical and syntactic diversity. In Proceedings of the Fifth Conference on Machine Translation, pages 561–570, Online. Association for Computational Linguistics. find our card
  28. Antonio Toral. 2020. Reassessing claims of human parity and super-human performance in machine translation at WMT 2019. In Proceedings of the 22nd Annual Conference of the European Association for Machine Translation, pages 185–194, Lisboa, Portugal. European Association for Machine Translation. ↗
  29. Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly easy document-level MT metrics: How to convert any pretrained metric into a document-level metric. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 118–128, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics. find our card
  30. Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. 2022. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8117–8127, Dublin, Ireland. Association for Computational Linguistics. find our card
  31. Lucy Lu Wang, Kyle Lo, Yoganand Chandrasekhar, Russell Reas, Jiangjiang Yang, Doug Burdick, Darrin Eide, Kathryn Funk, Yannis Katsis, Rodney Michael Kinney, Yunyao Li, Ziyang Liu, William Merrill, Paul Mooney, Dewey A. Murdick, Devvret Rishi, Jerry Sheehan, Zhihong Shen, Brandon Stilson, Alex D. Wade, Kuansan Wang, Nancy Xin Ru Wang, Christopher Wilhelm, Boya Xie, Douglas M. Raymond, Daniel S. Weld, Oren Etzioni, and Sebastian Kohlmeier. 2020. CORD-19: The COVID-19 open research dataset. In Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020, Online. Association for Computational Linguistics. ↗
  32. Lana Yeganova, Dina Wiemann, Mariana Neves, Federica Vezzani, Amy Siu, Inigo Jauregi Unanue, Maite Oronoz, Nancy Mah, Aurélie Névéol, David Martinez, Rachel Bawden, Giorgio Maria Di Nunzio, Roland Roller, Philippe Thomas, Cristian Grozea, Olatz Perez-de Viñaspre, Maika Vicente Navarro, and Antonio Jimeno Yepes. 2021. Findings of the WMT 2021 biomedical translation shared task: Summaries of animal experiments as new test set. In Proceedings of the Sixth Conference on Machine Translation, pages 664–683, Online. Association for Computational Linguistics. find
  33. Vilém Zouhar, Shehzaad Dhuliawala, Wangchunshu Zhou, Nico Daheim, Tom Kocmi, Yuchen Eleanor Jiang, and Mrinmaya Sachan. 2023. Poor man’s quality estimation: Predicting reference-based MT metrics without the reference. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 1311– 1325, Dubrovnik, Croatia. Association for Computational Linguistics. Langs WMT Test Train Test Bio Dev Total De-En En-De Es-En En-Es Ru-En En-Ru Fr-En En-Fr Zh-En En-Zh Pt-En -18k -- -19k -- 23k -- -28k -- -16k -- 27k -- 2457 2695 1013 1112 1324 825 1108 1228 2838 3900 701 903 917 309 330 388 237 352 308 913 1200 222 3360 3612 1322 1442 1712 1062 1460 1536 3751 5100 924 All 60k 71k 19k 6k 25k ↗

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/papers/zouhar-2024-fine-tuned-machine-translation-metrics/