Elamur Research
☰
home / papers / 2021 / A Fine-Grained Analysis of BERTScore

A Fine-Grained Analysis of BERTScore

Written for the Research library (claude-opus-5-5) from the paper text. Original — aclanthology.org/2021.wmt-1.59.pdf

Summary

BERTScore judges a candidate translation by matching its BERT token embeddings against those of a reference. Earlier probing work showed that BERT handles negation, word order, numbers and named entities poorly. Hanna and Bojar ask whether these weaknesses carry over to the metric. They propose three requirements for a good reference-based MT metric:

  • two valid references should score close to 1 against each other;
  • erroneous candidates should score below an alternative reference;
  • flawed candidates should be ordered by how bad they are.

They test BERTScore against these requirements on three annotated resources: TQ-AutoTest, PE 2 rr, and a grammatical error correction (GEC) corpus. BERTScore reliably penalises candidates that get content words wrong, mistranslate ambiguous words or contain multi-word errors. It is much less reliable when an incorrect candidate differs from the reference only in function words or stays lexically and stylistically close to it. On the GEC data, it meets all three requirements. BERTScore also beats BLEU throughout. The study shows where the metric can be trusted, for example when comparing very different systems, and argues for more linguistically annotated evaluation data.

Key points

  • TQ-AutoTest: German sources covering 14 linguistic phenomena. Phenomena with fewer than 5 examples were removed. This dataset only allows tests of the first two requirements.
  • PE 2 rr: Only the German–English portion was used, with word-level error labels in 8 classes (e.g., "addition", "lexical error", "untranslated"). Fully correct MT outputs were excluded. The two references in this dataset differ strongly in wording, and the paper reports a BERTScore of 0.587 for these reference pairs.
  • Hardest cases: Errors in function words, such as wrong tag questions ("You're crazy, or?"), were the most difficult for BERTScore. Bad candidates with high lexical overlap with the reference also often escaped penalty.
  • Style sensitivity: Because of this, the authors suggest BERTScore suits comparing widely differing systems better than ranking closely related systems or several outputs of one system.
  • GEC results: On the GEC data, BERTScore ranked partially corrected sentences by quality. The authors caution that this corpus mostly contains grammatical rather than semantic errors, so it does not fully reflect real MT errors.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Linguistics & languages · Machine translation

Cited by

62 papers of our library cite it (of the 140673 whose bibliographies we read); 4 of them have a page here, 58 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work.

@inproceedings{hanna2021fine,
  title = {{A Fine-Grained Analysis of BERTScore}},
  author = {Hanna, Michael and Bojar, Ondřej},
  booktitle = {Proceedings of the Sixth Conference on Machine Translation},
  year = {2021},
  doi = {10.48448/dhpa-w533},
  url = {https://doi.org/10.48448/dhpa-w533}
}

References

25 references, 10 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Sriram Balasubramanian, Naman Jain, Gaurav Jindal, Abhijeet Awasthi, and Sunita Sarawagi. 2020. What’s in a name? are BERT named entity representations just as good for any other name? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 205–214, Online. Association for Computational Linguistics. find
  2. Christopher Bryant and Hwee Tou Ng. 2015. How far are we from fully automatic high quality grammatical error correction? In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 697–707, Beijing, China. Association for Computational Linguistics. find
  3. Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D. Manning. 2019. What does BERT look at? an analysis of BERT’s attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276–286, Florence, Italy. Association for Computational Linguistics. find
  4. BERTScore is a new metric for MT evaluation that uses BERT, and is as a result difficult to in-Jacob Devlin, Ming-Wei Chang, Kenton Lee, and terpret. We define desiderata for BERTScore and Kristina Toutanova. 2019. BERT: Pre-training of other such metrics’ performance, and use targeted deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference datasets to find when BERTScore fails. We find of the North American Chapter of the Association that BERTScore fails to assign low scores when for Computational Linguistics: Human Language a bad candidate sentence has high lexical overlap Technologies, Volume 1 (Long and Short Papers), with the reference in terms of content words. De-pages 4171–4186, Minneapolis, Minnesota. Associ-spite this, in less challenging scenarios, BERTscore ation for Computational Linguistics. 515 find
  5. Allyson Ettinger. 2020. What BERT Is Not: Lessons from a New Suite of Psycholinguistic Diagnostics for Language Models. Transactions of the Association for Computational Linguistics, 8:34–48. find our card
  6. Marina Fomicheva and Lucia Specia. 2019. Taking MT Evaluation Metrics to Extremes: Beyond Correlation with Human Judgments. Computational Linguistics, 45(3):515–558. find
  7. Goran Glavas and Ivan Vulić. 2021. Is supervised syntactic parsing beneficial for language understanding tasks? an empirical investigation. In EACL. find
  8. John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129–4138, Minneapolis, Minnesota. Association for Computational Linguistics. find our card
  9. Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020b. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pages 688–725, Online. Association for Computational Linguistics. find our card
  10. Hwee Tou Ng, Siew Mei Wu, Ted Briscoe, Christian Hadiwinoto, Raymond Hendy Susanto, and Christopher Bryant. 2014. The CoNLL-2014 shared task on grammatical error correction. In Proceedings of the Eighteenth Conference on Computational Natural Language Learning: Shared Task, pages 1–14, Baltimore, Maryland. Association for Computational Linguistics. find
  11. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318. ↗ our card
  12. Thang M. Pham, Trung Bui, Long Mai, and Anh Nguyen. 2020. Out of order: How important is the sequential order of words in a sentence in natural language understanding tasks? find
  13. Phu Mon Htut, Jason Phang, Shikha Bordia, and Samuel R. Bowman. 2019. Do attention heads in bert track syntactic dependencies? ArXiv, abs/1911.12246. ↗
  14. Tom Kocmi, Christian Federmann, Roman Grundkiewicz, Marcin Junczys-Dowmunt, Hitokazu Matsushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. find
  15. Alon Lavie and Abhaya Agarwal. 2007. Meteor: An automatic metric for mt evaluation with high levels of correlation with human judgments. In Proceedings of the Second Workshop on Statistical Machine Translation, StatMT ’07, page 228–231, USA. Association for Computational Linguistics. find our card
  16. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the WMT19 metrics shared task: Segment-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pages 62–90, Florence, Italy. Association for Computational Linguistics. find our card
  17. Vivien Macketanz, Renlong Ai, Aljoscha Burchardt, and Hans Uszkoreit. 2018. TQ-AutoTest – an automated test suite for (machine) translation quality. In Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources Association (ELRA). find
  18. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4984–4997, Online. Association for Computational Linguistics. find our card
  19. Maja Popović and Mihael Arčan. 2016. PE2rr corpus: Manual error annotation of automatically pre-annotated MT post-edits. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 27–32, Portorož, Slovenia. European Language Resources Association (ELRA). find
  20. Abhilasha Ravichander, Eduard Hovy, Kaheer Suleman, Adam Trischler, and Jackie Chi Kit Cheung. 2020. On the systematicity of probing contextualized word representations: The case of hypernymy in BERT. In Proceedings of the Ninth Joint Conference on Lexical and Computational Semantics, pages 88–102, Barcelona, Spain (Online). Association for Computational Linguistics. find
  21. Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics. find our card
  22. Ieva Staliunaite and Ignacio Iacobacci. 2020. Compositional and lexical semantics in roberta, bert and distilbert: A case study on coqa. In EMNLP. find
  23. Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang, Adam Poliak, R Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. 2019. What do you learn from context? probing for sentence structure in contextualized word representations. find our card
  24. Eric Wallace, Yizhong Wang, Sujian Li, Sameer Singh, and Matt Gardner. 2019. Do NLP models know numbers? probing numeracy in embeddings. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5307– 5315, Hong Kong, China. Association for Computational Linguistics. find our card
  25. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. find