Elamur Research⌕
☰
home / papers / 2020 / BLEURT: Learning Robust Metrics for Text Generation

BLEURT: Learning Robust Metrics for Text Generation

Paper · 2020 · Thibault Sellam, Dipanjan Das and Ankur Parikh

Summary

Classical metrics such as BLEU and ROUGE count only n-gram overlap. As a result, they correlate poorly with human judgments of generated text, especially when the systems being compared are of similar quality. The authors propose BLEURT, a learned metric for English based on BERT. It takes a reference sentence and a candidate sentence and predicts the score a human would assign. The main novelty is an additional pre-training stage on synthetic data. Sentences from Wikipedia are automatically perturbed, and each resulting pair is assigned several cheap signals: scores from other metrics, the backtranslation likelihood, and the type of logical relation between the sentences. Only after this is the model fine-tuned on a few thousand human ratings. BLEURT achieved the best results on WMT Metrics 2017–2019 and transferred well to WebNLG 2017. The synthetic stage helps especially when the training data are scarce, skewed, or out of domain. This matters because generation systems improve over time and move on to new tasks, so a metric must extrapolate rather than merely reproduce an old distribution of ratings.

Key points

  • The synthetic corpus was built from about 1.8 million Wikipedia segments, which yielded around 6.5 million perturbed variants. Three techniques were used for this: mask filling with BERT, backtranslation, and random word dropping.
  • During pre-training, the model learns 9 tasks simultaneously. The signals include BLEU, ROUGE, BERTscore, backtranslation likelihood via French and German, entailment/contradiction/neutral labels, and a backtranslation flag.
  • The model achieved state-of-the-art results on the WMT Metrics Shared Task 2017–2019 for to-English language pairs. Separately, its robustness to improvements in system quality was tested on a synthetic benchmark based on WMT 2017.
  • On WebNLG 2017 (evaluated for semantics, grammar, and fluency), the variant with synthetic pre-training needed only 836 records to compete with the baseline metrics. The version additionally fine-tuned on WMT, without any training on WebNLG, achieved results comparable to BERTscore.
  • Ablations showed that the BERTscore, entailment, and backtranslation signals improve the model, whereas BLEU and ROUGE hurt it on WMT 2017. The code and models are available at github.com/google-research/bleurt.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏢GoogleMountain View · United States430 works · 424,363 citations

 Fields, topics and concepts

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 16,667 worksArtificial Intelligence · 16,194 worksArts and Humanities · 16,039 worksLanguage and Linguistics · 15,891 works

🧭 Topics 2Research subjects and languages the work is about.Evaluation & benchmarks · 1,220 worksText generation · 202 works

💡 Concepts 1Key ideas and methods explained on our concept pages.Text summarization · 49 works

Cite

Cite as Thibault Sellam, Dipanjan Das and Ankur Parikh. 2020. BLEURT: Learning Robust Metrics for Text Generation. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.704 · arXiv 2004.04696

@inproceedings{sellam2020bleurt,
  title = {{BLEURT: Learning Robust Metrics for Text Generation}},
  author = {Thibault Sellam, Dipanjan Das and Ankur Parikh},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.704},
  eprint = {2004.04696},
  archivePrefix = {arXiv},
  url = {https://doi.org/10.18653/v1/2020.acl-main.704}
}

References

47 references, 29 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of ICLR. ↗
  2. Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of ACL.
  3. Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR. ↗
  4. Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017. Results of the wmt17 metrics shared task. In Proceedings of WMT.
  5. Ondřej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. 2016. Results of the wmt16 metrics shared task. In Proceedings of WMT.
  6. Samuel R Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. Proceedings of EMNLP. ↗
  7. Arun Tejasvi Chaganty, Stephen Mussman, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evaluation. Proceedings of ACL. ↗
  8. Qian Chen, Xiaodan Zhu, Zhenhua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017. Enhanced lstm for natural language inference. Proceedings of ACL. find
  9. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of CVPR. find
  10. Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2013. Ppdb: The paraphrase database. In Proceedings NAACL HLT.
  11. Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini. 2017. The webnlg challenge: Generating text from rdf data. In Proceedings of INLG.
  12. Ben Goodrich, Mohammad Ahmad Saleh, Peter Liu, and Vinay Rao. 2019. Assessing the factual accuracy of text generation. In Proceedings of ACM SIGKDD. find
  13. Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. Proceedings of NAACL HLT.
  14. Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. Proceedings of EMNLP.
  15. Philipp Koehn. 2009. Statistical machine translation. Cambridge University Press. ↗
  16. Karen Kukich. 1983. Design of a knowledge-based report generator. In Proceedings of ACL.
  17. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Workshop on Text Summarization Branches Out.
  18. Chia-Wei Liu, Ryan Lowe, Iulian V Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How not to evaluate your dialogue system: An empirical study of unsupervised evaluation metrics for dialogue response generation. Proceedings of EMNLP.
  19. Sumit Chopra, Michael Auli, and Alexander M Rush. 2016. Abstractive sentence summarization with attentive recurrent neural networks. In Proceedings of NAACL HLT.
  20. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv:1907.11692. ↗
  21. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL HLT.
  22. Qingsong Ma, Ondřej Bojar, and Yvette Graham. 2018. Results of the wmt18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of WMT.
  23. Ondřej Dušek, Karin Sevegnani, Ioannis Konstas, and Verena Rieser. 2019. Automatic quality estimation for natural language generation: Ranting (jointly rating and ranking). Proceedings of INLG. find
  24. Qingsong Ma, Yvette Graham, Shugen Wang, and Qun Liu. 2017. Blend: a novel combined mt metric based on direct assessment–casict-dcu submission to wmt17 metrics task. In Proceedings of WMT. find
  25. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of NAACL HLT. ↗
  26. Qingsong Ma, Johnny Wei, Ondřej Bojar, and Yvette Graham. 2019. Results of the wmt19 metrics shared task: Segment-level and strong mt systems pose big challenges. In Proceedings of WMT.
  27. Inderjeet Mani. 1999. Advances in automatic text summarization. MIT press. find
  28. Oriol Vinyals and Quoc Le. 2015. A neural conversational model. Proceedings of ICML. ↗
  29. Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2019. Putting evaluation in context: Contextual embeddings improve machine translation evaluation. In Proceedings of ACL. find
  30. Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of ICLR. ↗
  31. Kathleen McKeown. 1992. Text generation. Cambridge University Press. find
  32. Jekaterina Novikova, Ondřej Dušek, Amanda Cercas Curry, and Verena Rieser. 2017. Why we need new evaluation metrics for nlg. Proceedings of EMNLP.
  33. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL. ↗
  34. Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of ACL. find
  35. Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Improving neural machine translation models with monolingual data. Proceedings of ACL. find
  36. Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. 2018. Ruse: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of WMT.
  37. Anastasia Shimorina, Claire Gardent, Shashi Narayan, and Laura Perez-Beltrachini. 2019. Webnlg challenge: Human evaluation results. Technical report. find
  38. John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016. Towards universal paraphrastic sentence embeddings. Proceedings of ICLR. ↗
  39. Adina Williams, Nikita Nangia, and Samuel R Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. Proceedings of NAACL HLT.
  40. Sam Wiseman, Stuart M Shieber, and Alexander M Rush. 2017. Challenges in data-to-document generation. Proceedings of EMNLP.
  41. Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. Sum-qe: a bert-based summary quality estimation model supplementary material. In Proceedings of EMNLP. find
  42. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. Proceedings of ICLR. ↗
  43. Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. Proceedings of EMNLP. ↗
  44. Ronnie W Smith and D Richard Hipp. 1994. Spoken natural language dialog systems: A practical approach. Oxford University Press. A Implementation Details of the Pre-Training Phase Milos Stanojevic and Khalil Sima’an. 2014. Beer: Better evaluation as ranking. In Proceedings of WMT. This section provides implementation details for some of the pre-training techniques described in the main paper. find
  45. Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. In Proceedings of NIPS. A.1 Data Generation Ran Tian, Shashi Narayan, Thibault Sellam, and Ankur P Parikh. 2019. Sticking to the facts: Confident decoding for faithful data-to-text generation. arXiv:1910.08684. ↗
  46. Gaurav Singh Tomar, Thyago Duque, Oscar Täckström, Jakob Uszkoreit, and Dipanjan Das. 2017. Neural paraphrase identification of questions with noisy pretraining. Proceedings of the First Workshop on Subword and Character Level Models in NLP. ↗
  47. Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proceedings of NIPS.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/sellam-2020-bleurt-learning-robust-metrics-text/