Elamur Research⌕
☰
home / papers / 2025 / Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German

Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German

Paper · 2025 · Paolo Di Natale, Elena Chiocchetti and Egon Waldemar Stemle

Summary

The paper tackles a practical gap: no one had checked which automatic MT evaluation metrics can be trusted for Italian and South Tyrolean German. South Tyrolean German is a low-resourced standard variety of German that is co-official in the province of Bolzano/Bozen. There, translated legal and administrative texts can become legally binding, and machine translation is increasingly used, often by non-specialists. Earlier South Tyrolean MT work relied on lexical-overlap scores simply because no localized validation existed.

The authors run a segment-level meta-evaluation on MT@BZ, a corpus of machine-translated bilingual decrees covering both translation directions. Human error tags are converted into quality scores with custom penalty weights. String-based, embedding-based and learned metrics are then compared on how well they agree with these human judgments, across different quality ranges.

Embedding-based metrics do best on high-quality output. Learned neural metrics track humans better when translations are weaker. The study also finds a persistent bias against the minor variety. These results matter for choosing evaluation tools, for planning annotated resources, and for safely using metrics in Minimum Bayes Risk (MBR) decoding.

Key points

  • Dataset. MT@BZ is a manually error-annotated legal corpus of decrees, evaluated in both directions, IT→DE and DE→IT. Segments are sentences.
  • Annotation and scoring.
    • Annotation follows a custom scheme based on the SCATE taxonomy, which separates accuracy errors from fluency errors.
    • Each error type carries a penalty weight. For example, Part of Speech and Untranslated errors cost 25, many other accuracy errors cost 15–20, and Punctuation and Capitalization cost 5.
  • Embedding-based subscores. Variants built on bert-base-multilingual, bart-large-mnli, deberta-xlarge-mnli and roberta-large-mnli were analysed.
    • On the whole dataset, IT→DE accuracy correlation is about 0.77–0.78.
    • On the mistake-only subset, it falls to about 0.52–0.55.
  • Motivation.
    • About 350,000 German speakers (69% of the population) have the right to public services in German.
    • Where the two versions diverge, the Italian text prevails.
    • MT systems tend to slip in terminology from other German legal systems, such as kommunale Steuer instead of South Tyrolean Gemeindesteuer.
  • Main finding. Which metric family performs best depends on translation quality: embedding-based metrics lead at the high end, learned metrics in lower ranges. Metrics also show bias against minor-variety variants.

Interesting ideas

  • I1. Stratify meta-evaluation by quality level. Report correlation on the full set and separately on an error-only subset, because metric rankings can flip between the two and a pooled number hides this.
  • I2. Turn categorical error tags into a graded score with a weighted taxonomy. Meaning-level accuracy errors cost far more than surface slips like punctuation, which gives a scalar target that reflects what actually matters.
  • I3. Keep the optimizer and the judge apart. Decoding (as in MBR) and final evaluation should not use the same metric, or even closely correlated metrics of the same type, because the system learns their quirks.
  • I4. Decompose a similarity score into precision and recall. This shows whether a metric mainly punishes additions or omissions, making an opaque number more diagnosable.
  • I5. Test whether a metric unfairly penalizes regional lexical variants that keep the meaning. An example is the South Tyrolean legal term versus the Germany-German one, an evaluation axis rarely checked.

For Mova

How these ideas could serve Mova, our small open language model — its files, stages and tests:

  • I1 (learn): Evaluate parser and reader metrics separately on the hard slice (sentences where witness parsers disagree or answers fail), so an aggregate score cannot mask weakness exactly where errors live.
  • I2 (back): Score text→graph→text round trips with weighted error classes, so that losing a semantic role or entity costs much more than a morphology or punctuation difference.
  • I3 (learn): When silver data comes from agreement among witness parsers, keep architecturally similar witnesses from dominating, and evaluate with an independent gold set so shared biases are not rewarded.
  • I4 (back): Report precision and recall of the meaning graph in compressor and generation tests, so additions (hallucinated facts) and omissions (dropped facts) are tracked as distinct failure modes.
  • I5 (world): Map regional or dialectal synonyms to the same ontology concept, and add controls checking that a variant word does not change the extracted entities or the answer.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Masaryk UniversityBrno · Czechia10 works · 437 citations🔬Eurac ResearchBolzano · Italy2 works · 30 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Machine translation · 1,087 worksEvaluation & benchmarks · 1,220 works

Cite

Cite as Paolo Di Natale, Elena Chiocchetti and Egon Waldemar Stemle. 2025. Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German. Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025).

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{natale2025meta,
  title = {{Meta-evaluation of Automatic Machine Translation Metrics between Italian and a Minor Language Variety of German}},
  author = {Paolo {Di Natale}, Elena Chiocchetti and Egon Waldemar Stemle},
  booktitle = {Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)},
  year = {2025},
  url = {https://aclanthology.org/2025.clicit-1.38/}
}

References

2 references, 2 of them with a card of ours. ↗ leads to the work.

  1. in: Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), ELRA and ICCL, Torino, Italia, 2024, pp. 3553–3565. URL: https://aclanthology.org/2024.lrec-main.315/. [5] A. Magueresse, V. Carles, E. Heetderks, Low-resource languages: A review of past work and future challenges, 2020. URL: https://arxiv.org/abs/ 2006.07264. arXiv:2006.07264 . [6] R. Knowles, S. Larkin, C.-K. Lo, MSLC24: Further challenges for metrics on a wide landscape of translation quality, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 475–491. URL: https://aclanthology.org/2024. wmt-1.34/. doi: 10.18653/v1/2024.wmt-1.34 . [7] V. Dewangan, B. R. S, G. Suri, R. Sonavane, When every token counts: Optimal segmentation for low-resource language models, in: Proceedings of the First Workshop on Language Models for Low-Resource Languages, Association for Computational Linguistics, Abu Dhabi, United Arab Emirates, 2025, pp. 294–308. URL: https://aclanthology. org/2025.loreslm-1.24/. [8] F. De Camillis, E. W. Stemle, E. Chiocchetti, F. Fernicola, The MT@BZ corpus: machine translation & legal language, in: Proceedings of the 24th Annual Conference of the European Association for Machine Translation, European Association for Machine Translation, Tampere, Finland, 2023, pp. 171– 180. URL: https://aclanthology.org/2023.eamt-1.17/. [9] A. Oliver, S. Alvarez-Vidal, E. Stemle, E. Chiocchetti, Training an NMT system for legal texts of a low-resource language variety south tyrolean German - Italian, in: Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), European Association for Machine Translation (EAMT), Sheffield, UK, 2024, pp. 573–579. URL: https://aclanthology. org/2024.eamt-1.47/. [10] S. Perrella, L. Proietti, A. Scirè, E. Barba, R. Navigli, Guardians of the machine translation meta-evaluation: Sentinel metrics fall in!, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 16216–16244. URL: https://aclanthology.org/2024.acl-long.856/. doi: 10.18653/v1/2024.acl-long.856 . [11] T. Kocmi, C. Federmann, R. Grundkiewicz, M. Junczys-Dowmunt, H. Matsushita, A. Menezes, To ship or not to ship: An extensive evaluation of automatic metrics for machine translation, in: Proceedings of the Sixth Conference on Machine Translation, Association for Computational Linguistics, Online, 2021, pp. 478–494. URL: https://aclanthology.org/2021.wmt-1.57/. [12] M. Freitag, G. Foster, D. Grangier, V. Ratnakar, Q. Tan, W. Macherey, Experts, errors, and context: A large-scale study of human evaluation for machine translation, Transactions of the Association for Computational Linguistics 9 (2021) 1460– 1474. URL: https://aclanthology.org/2021.tacl-1.87/. doi: 10.1162/tacl_a_00437 . [13] F. D. Camillis, La traduzione non professionale nelle istituzioni pubbliche dei territori di lingua minoritaria: il caso di studio dell’amministrazione della Provincia autonoma di Bolzano, Ph.D. thesis, alma, 2021. URL: https://amsdottorato.unibo.it/id/eprint/ 9695/. [14] F. De Camillis, E. Chiocchetti, Machine-translating legal language: error analysis on an italian-german corpus of decrees, Terminology science & research 27 (2024) 1–27. URL: https://journal-eaft-aet.net/ index.php/tsr/article/view/8304/7492. [15] J. O. Alabi, D. I. Adelani, M. Mosbach, D. Klakow, Adapting pre-trained language models to African languages via multilingual adaptive fine-tuning, in: Proceedings of the 29th International Conference on Computational Linguistics, International Committee on Computational Linguistics, Gyeongju, Republic of Korea, 2022, pp. 4336–4349. URL: https://aclanthology.org/2022.coling-1.382/. [16] J. Sun, T. Sellam, E. Clark, T. Vu, T. Dozat, D. Garrette, A. Siddhant, J. Eisenstein, S. Gehrmann, Dialect-robust evaluation of generated text, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 6010–6028. URL: https://aclanthology.org/2023.acl-long.331/. doi: 10.18653/v1/2023.acl-long.331 . [17] C. Amrhein, N. Moghe, L. Guillou, ACES: Translation accuracy challenge sets for evaluating machine translation metrics, in: Proceedings of the Seventh Conference on Machine Translation (WMT), Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 2022, pp. 479–513. URL: https://aclanthology.org/2022.wmt-1.44/. [18] Y. Yan, T. Wang, C. Zhao, S. Huang, J. Chen, M. Wang, BLEURT has universal translations: An analysis of automatic metrics by minimum risk training, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 5428–5443. URL: https://aclanthology. org/2023.acl-long.297/. doi: 10.18653/v1/2023. acl-long.297 . [19] P. Fernandes, A. Farinhas, R. Rei, J. G. C. de Souza, P. Ogayo, G. Neubig, A. Martins, Quality-aware decoding for neural machine translation, in: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Association for Computational Linguistics, Seattle, United States, 2022, pp. 1396–1412. URL: https://aclanthology.org/2022.naacl-main.100/. doi: 10. 18653/v1/2022.naacl-main.100 . [20] G. Kovacs, D. Deutsch, M. Freitag, Mitigating metric bias in minimum Bayes risk decoding, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 1063–1094. URL: https://aclanthology.org/2024.wmt-1.109/. doi: 10. 18653/v1/2024.wmt-1.109 . [21] J. Pombal, N. M. Guerreiro, R. Rei, A. F. T. Martins, Adding chocolate to mint: Mitigating metric interference in machine translation, 2025. URL: https://arxiv.org/abs/2503.08327. arXiv:2503.08327 . [22] M. Freitag, N. Mathur, D. Deutsch, C.-K. Lo, E. Avramidis, R. Rei, B. Thompson, F. Blain, T. Kocmi, J. Wang, D. I. Adelani, M. Buchicchio, C. Zerva, A. Lavie, Are LLMs breaking MT metrics? results of the WMT24 metrics shared task, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 47– 81. URL: https://aclanthology.org/2024.wmt-1.2/. doi: 10.18653/v1/2024.wmt-1.2 . [23] N. Moghe, T. Sherborne, M. Steedman, A. Birch, Extrinsic evaluation of machine translation metrics, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 13060–13078. URL: https://aclanthology.org/2023.acl-long.730/. doi: 10.18653/v1/2023.acl-long.730 . [24] S. Agrawal, A. Farajian, P. Fernandes, R. Rei, A. F. T. Martins, Assessing the role of context in chat translation evaluation: Is context helpful and under what conditions?, Transactions of the Association for Computational Linguistics 12 (2024) 1250– 1267. URL: https://aclanthology.org/2024.tacl-1.69/. doi: 10.1162/tacl_a_00700 . [25] R. Rei, N. M. Guerreiro, M. Treviso, L. Coheur, A. Lavie, A. Martins, The inside story: Towards better understanding of machine translation neural evaluation metrics, in: Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Toronto, Canada, 2023, pp. 1089–1105. URL: https://aclanthology.org/2023.acl-short.94/. doi: 10. 18653/v1/2023.acl-short.94 . ↗
  2. [26] M. Freitag, N. Mathur, C.-k. Lo, E. Avramidis, R. Rei, B. Thompson, T. Kocmi, F. Blain, D. Deutsch, C. Stewart, C. Zerva, S. Castilho, A. Lavie, G. Foster, Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent, in: Proceedings of the Eighth Conference on Machine Translation, Association for Computational Linguistics, Singapore, 2023, pp. 578– 628. URL: https://aclanthology.org/2023.wmt-1.51/. doi: 10.18653/v1/2023.wmt-1.51 . [27] N. Moghe, A. Fazla, C. Amrhein, T. Kocmi, M. Steedman, A. Birch, R. Sennrich, L. Guillou, Machine translation meta evaluation through translation accuracy challenge sets, Computational Linguistics 51 (2025) 73–137. URL: https://aclanthology.org/2025. cl-1.4/. doi: 10.1162/coli_a_00537 . [28] E. Avramidis, S. Manakhimova, V. Macketanz, S. Möller, Machine translation metrics are better in evaluating linguistic errors on LLMs than on encoder-decoder systems, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 517–528. URL: https://aclanthology.org/2024.wmt-1.37/. doi: 10.18653/ v1/2024.wmt-1.37 . [29] V. Zouhar, S. Ding, A. Currey, T. Badeka, J. Wang, B. Thompson, Fine-tuned machine translation metrics struggle in unseen domains, in: Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), Association for Computational Linguistics, Bangkok, Thailand, 2024, pp. 488–500. URL: https://aclanthology.org/2024.acl-short.45/. doi: 10. 18653/v1/2024.acl-short.45 . [30] A. Lommel, S. Gladkoff, A. Melby, S. E. Wright, I. Strandvik, K. Gasova, A. Vaasa, A. Benzo, R. Marazzato Sparano, M. Foresi, J. Innis, L. Han, G. Nenadic, The multi-range theory of translation quality measurement: MQM scoring models and statistical quality control, in: Proceedings of the 16th Conference of the Association for Machine Translation in the Americas (Volume 2: Presentations), Association for Machine Translation in the Americas, Chicago, USA, 2024, pp. 75–94. URL: https://aclanthology.org/2024. amta-presentations.6/. [31] K. Papineni, S. Roukos, T. Ward, W.-J. Zhu, Bleu: a method for automatic evaluation of machine translation, in: Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Philadelphia, Pennsylvania, USA, 2002, pp. 311–318. URL: https://aclanthology.org/P02-1040/. doi: 10.3115/1073083.1073135 . [32] T. Sellam, D. Das, A. Parikh, BLEURT: Learning robust metrics for text generation, in: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, Online, 2020, pp. 7881–7892. URL: https://aclanthology.org/2020.acl-main.704/. doi: 10.18653/v1/2020.acl-main.704 . [33] T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, Y. Artzi, Bertscore: Evaluating text generation with bert, 2020. URL: https://arxiv.org/abs/1904.09675. arXiv:1904.09675 . [34] M. Popović, chrF: character n-gram F-score for automatic MT evaluation, in: Proceedings of the Tenth Workshop on Statistical Machine Translation, Association for Computational Linguistics, Lisbon, Portugal, 2015, pp. 392–395. URL: https://aclanthology. org/W15-3049/. doi: 10.18653/v1/W15-3049 . [35] R. Rei, C. Stewart, A. C. Farinha, A. Lavie, COMET: A neural framework for MT evaluation, in: Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Association for Computational Linguistics, Online, 2020, pp. 2685–2702. URL: https://aclanthology.org/ 2020.emnlp-main.213/. doi: 10.18653/v1/2020. emnlp-main.213 . [36] R. Rei, M. Treviso, N. M. Guerreiro, C. Zerva, A. C. Farinha, C. Maroti, J. G. C. de Souza, T. Glushkova, D. Alves, L. Coheur, A. Lavie, A. F. T. Martins, CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task, in: Proceedings of the Seventh Conference on Machine Translation (WMT), Association for Computational Linguistics, Abu Dhabi, United Arab Emirates (Hybrid), 2022, pp. 634–645. URL: https://aclanthology.org/ 2022.wmt-1.60/. [37] R. Rei, N. M. Guerreiro, J. Pombal, D. van Stigt, M. Treviso, L. Coheur, J. G. C. de Souza, A. Martins, Scaling up CometKiwi: Unbabel-IST 2023 submission for the quality estimation shared task, in: Proceedings of the Eighth Conference on Machine Translation, Association for Computational Linguistics, Singapore, 2023, pp. 841–848. URL: https://aclanthology.org/2023.wmt-1.73/. doi: 10.18653/ v1/2023.wmt-1.73 . [38] J. Juraska, D. Deutsch, M. Finkelstein, M. Freitag, MetricX-24: The Google submission to the WMT 2024 metrics shared task, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 492–504. URL: https://aclanthology.org/2024.wmt-1.35/. doi: 10.18653/ v1/2024.wmt-1.35 . [39] M. Snover, B. Dorr, R. Schwartz, L. Micciulla, J. Makhoul, A study of translation edit rate with targeted human annotation, in: Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers, Association for Machine Translation in the Americas, Cambridge, Massachusetts, USA, 2006, pp. 223–231. URL: https://aclanthology.org/2006.amta-papers.25/. [40] Y. Wan, D. Liu, B. Yang, H. Zhang, B. Chen, D. Wong, L. Chao, UniTE: Unified translation evaluation, in: Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Association for Computational Linguistics, Dublin, Ireland, 2022, pp. 8117–8127. URL: https://aclanthology.org/2022.acl-long.558/. doi: 10.18653/v1/2022.acl-long.558 . [41] N. M. Guerreiro, R. Rei, D. v. Stigt, L. Coheur, P. Colombo, A. F. T. Martins, xcomet: Transparent machine translation evaluation through fine-grained error detection, Transactions of the Association for Computational Linguistics 12 (2024) 979– 995. URL: https://aclanthology.org/2024.tacl-1.54/. doi: 10.1162/tacl_a_00683 . [42] V. Zouhar, P. Chen, T. K. Lam, N. Moghe, B. Haddow, Pitfalls and outlooks in using COMET, in: Proceedings of the Ninth Conference on Machine Translation, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 1272–1288. URL: https://aclanthology.org/2024.wmt-1.121/. doi: 10. 18653/v1/2024.wmt-1.121 . [43] D. Deutsch, G. Foster, M. Freitag, Ties matter: Meta-evaluating modern metrics with pairwise accuracy and tie calibration, in: Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Singapore, 2023, pp. 12914–12929. URL: https://aclanthology.org/2023.emnlp-main.798/. doi: 10. 18653/v1/2023.emnlp-main.798 . [44] S. Agrawal, A. Farinhas, R. Rei, A. Martins, Can automatic metrics assess high-quality translations?, in: Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics, Miami, Florida, USA, 2024, pp. 14491–14502. URL: https://aclanthology.org/2024.emnlp-main.802/. doi: 10. 18653/v1/2024.emnlp-main.802 . ↗

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/natale-2025-meta-evaluation-automatic-machine-translation/