Elamur Research
☰
home / papers / 2019 / YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources

YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/W19-5358.pdf

Summary

The paper tackles a long-standing gap in machine translation evaluation. BLEU remains dominant because it works for any language, even though it agrees poorly with human judgments. Richer semantic metrics such as METEOR and MEANT need linguistic resources that many languages lack. YiSi is an open-source framework designed to work at whatever level of resources a language has. Every variant computes an idf-weighted F-score over aligned n-grams, and it can optionally add semantic frames from a shallow semantic parser.

The variants differ in the resources they need:

  • YiSi-0 needs no resources and compares words by their longest common character substring.
  • YiSi-1 compares MT output to a reference using embeddings.
  • YiSi-2 needs no reference. It compares the source input to the MT output through cross-lingual embeddings.

The main change in this version is replacing word2vec with contextual BERT embeddings. The author tested the variants on the WMT 2018 metrics evaluation data, which uses direct-assessment human scores. YiSi-1 outperformed every WMT 2018 participant at segment level for nearly all language pairs, and at system level for many of them. This shows that a single metric can scale from low-resource to high-resource conditions.

Key points

  • Embeddings come from three models:
    • layer 18 of English cased BERT-Large for English;
    • layer 9 of Chinese BERT-Base for Chinese;
    • layer 9 of multilingual cased BERT-Base for all other languages and for YiSi-2.
  • Semantic role labeling uses mateplus for English and German and mate-tools for Chinese. Roles are collapsed into 8 categories: the agent ("who"), the action ("did"), the patient ("what"), the recipient ("whom"), the time ("when"), the location ("where"), the reason ("why") and the manner ("how"). The structural weight β is 0.1.
  • The α parameter shifts the balance between precision and recall. It is set to 0.7 for evaluation, making the metric recall-oriented, and to 0.5 for system tuning.
  • At segment level, YiSi-1 reached a Kendall correlation of .544 on de-en and .734 on en-de, versus the best previous scores of .498 and .696. English-Turkish was the only pair where it did not win.
  • At system level, the resource-free YiSi-0 scored .857 Pearson on tr-en, far above chrF (.452) and BLEU (.657). The reference-free YiSi-2 trailed the reference-based variants clearly at segment level.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Topics: Machine translation

Cited by

112 papers of our library cite it (of the 140673 whose bibliographies we read); 16 of them have a page here, 96 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/w19-5358

@inproceedings{lo2019yisi,
  title = {{YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources}},
  author = {Lo, Chi-kiu},
  booktitle = {Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1)},
  year = {2019},
  doi = {10.18653/v1/w19-5358},
  url = {https://doi.org/10.18653/v1/w19-5358}
}

References

17 references, 8 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Anders Björkelund, Love Hafdell, and Pierre Nugues. 2009. Multilingual semantic role labeling. In Proceedings of the Thirteenth Conference on Computational Natural Language Learning (CoNLL 2009): Shared Task, pages 43–48, Boulder, Colorado. Association for Computational Linguistics. find
  2. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pages 489–513, Copenhagen, Denmark. Association for Computational Linguistics. Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computational Linguistics. find our card
  3. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the WMT16 Metrics Shared Task. In Proceedings of the First Conference on Machine Translation, pages 199– 231, Berlin, Germany. Association for Computational Linguistics. find our card
  4. Maja Popović. 2015. chrF: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics. ↗ our card
  5. Michael Denkowski and Alon Lavie. 2014. METEOR universal: Language specific translation evaluation for any target language. In 9th Workshop on Statistical Machine Translation (WMT 2014). find our card
  6. Michael Roth and Kristian Woodsend. 2014. Composition of word representations improves semantic role labelling. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 407–413, Doha, Qatar. Association for Computational Linguistics. find
  7. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805. ↗
  8. Miloš Stanojević, Amir Kamran, Philipp Koehn, and Ondřej Bojar. 2015. Results of the WMT15 Metrics Shared Task. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 256– 273, Lisbon, Portugal. Association for Computational Linguistics. find
  9. Chi-kiu Lo. 2017. MEANT 2.0: Accurate semantic MT evaluation for any output language. In Proceedings of the Second Conference on Machine Translation, pages 589–597, Copenhagen, Denmark. Association for Computational Linguistics. find our card
  10. Chi-kiu Lo, Michel Simard, Darlene Stewart, Samuel Larkin, Cyril Goutte, and Patrick Littell. 2018. Accurate semantic textual similarity for cleaning noisy parallel corpora using semantic machine translation evaluation metric: The NRC supervised submissions to the parallel corpus filtering task. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 908–916, Belgium, Brussels. Association for Computational Linguistics. find
  11. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998–6008. find
  12. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with BERT. CoRR, abs/1904.09675. ↗
  13. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pages 671–688, Belgium, Brussels. Association for Computational Linguistics. find our card
  14. Matous Machacek and Ondrej Bojar. 2014. Results of the WMT14 Metrics Shared Task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages 293–301, Baltimore, Maryland, USA. Association for Computational Linguistics. find our card
  15. Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Proceedings of the 26th International Conference on Neural Information Processing Systems, NIPS’13, pages 3111–3119, USA. Curran Associates Inc. find
  16. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In 40th Annual Meeting of the Association for Computational Linguistics (ACL-02), pages 311–318, Philadelphia, Pennsylvania. find our card
  17. Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language find