Elamur Research⌕
☰
home / papers / 2020 / Fill in the BLANC: Human-free quality estimation of document summaries

Fill in the BLANC: Human-free quality estimation of document summaries

Paper · 2020 · Oleg Vasilyev, Vedant Dharnidharka and John Bohannon

Summary

Both standard ways of judging summaries have weaknesses. ROUGE needs human-written reference summaries and only counts token overlap, so it never looks at the source document. Human ratings are slow, costly and hard to reproduce.

The authors propose BLANC, a metric that needs no reference summary. It treats a summary as good if it helps a pre-trained language model fill in blanks (the Cloze task) in the document itself. Using BERT, they mask words in each document sentence on a fixed, reproducible schedule. They then compare how accurately the model recovers those words with and without access to the summary. BLANC-help places the summary in front of each sentence and compares it against a same-length string of periods. BLANC-tune first fine-tunes BERT on masked copies of the summary.

Experiments on CNN / Daily Mail summaries scored by human annotators show that BLANC tracks human judgments about as well as ROUGE. It can also compete with small panels of annotators. Because no human input is needed, the method allows fully automatic, reproducible evaluation of summaries.

Key points

  • BLANC equals the accuracy of unmasking with the summary minus the accuracy without it. It can range from -1 to 1, but typical values fall between 0 (no help) and about 0.3.
  • Default BLANC-help settings: every 6th word is masked (M = 6, mirroring BERT's 15% masking rate), and only words of at least 4 characters are eligible (L_min = 4). For extractive summaries, a "no-copy-pair guard" prevents a sentence copied into the summary from making its own reconstruction unfairly easy.
  • On 855 CNN / Daily Mail summaries, BLANC correlated with averaged human scores at 0.22, versus 0.28 for Jensen-Shannon divergence. On highly compressed summaries (summary/text length < 0.05), BLANC kept a 0.18 correlation, while Jensen-Shannon was not significant (p = 0.15).
  • In a study with 10 annotators, all 120 splits into groups of 3 and 7 were tested. BLANC's correlation with the group of 7 was competitive with the group of 3's correlation, especially for overall quality and fluency.
  • The authors announce a public Python package named blanc. In their preliminary tests, BLANC-tune and BLANC-help gave similar values.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

📚 Shelves 4 Computer Science · 9,086 works Artificial Intelligence · 8,889 works Arts and Humanities · 9,296 works Language and Linguistics · 9,206 works

Cite

Cite as Oleg Vasilyev, Vedant Dharnidharka and John Bohannon. 2020. Fill in the BLANC: Human-free quality estimation of document summaries. Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.eval4nlp-1.2

@inproceedings{vasilyev2020fill,
  title = {{Fill in the BLANC: Human-free quality estimation of document summaries}},
  author = {Oleg Vasilyev, Vedant Dharnidharka and John Bohannon},
  booktitle = {Proceedings of the First Workshop on Evaluation and Comparison of NLP Systems},
  year = {2020},
  doi = {10.18653/v1/2020.eval4nlp-1.2},
  url = {https://doi.org/10.18653/v1/2020.eval4nlp-1.2}
}

References

29 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Ayana, Shiqi Shen, Yu Zhao, Zhiyuan Liu, and Maosong Sun. 2016. Neural headline generation with sentence-wise optimization. arXiv, arXiv:1604.01904v2. Version 2. — [arXiv:1604.01904](https://arxiv.org/abs/1604.01904) find
  2. Hardy, Shashi Narayan, and Andreas Vlachos. 2019. Ehighres: Highlight-based reference-less evaluation of summarization. arXiv, arXiv:1906.01361. — [arXiv:1906.01361](https://arxiv.org/abs/1906.01361) find
  3. Ping Chen, Fei Wu, Tong Wang, and Wei Ding. 2018. A semantic qa-based approach for text summarization evaluation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, pages 4800–4807. find
  4. Karl Moritz Hermann, Tomáš Kočiský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems 28, pages 1693–1701. Curran Associates, Inc. our card
  5. Arman Cohan and Nazli Goharian. 2016. Revisiting summarization evaluation for scientific articles. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16), pages 806–813. find
  6. European Language Resources Association (ELRA, 2016). find
  7. Shun Kiyono, Sho Takase, Jun Suzuki, Naoaki Okazaki, Kentaro Inui, and Masaaki Nagata. 2017. Source-side prediction for neural headline generation. arXiv, arXiv:1712.08302. — [arXiv:1712.08302](https://arxiv.org/abs/1712.08302) find
  8. Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv, arXiv:1810.04805. our card
  9. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan… our card
  10. Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Ming Zhou Jianfeng Gao, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. arXiv, arXiv:1905.03197. our card
  11. Fatma Elghannam and Tarek El-Shishtawy. 2015. Keyphrase based evaluation of automatic text summarization. arXiv, arXiv:1505.06228. — [arXiv:1505.06228](https://arxiv.org/abs/1505.06228) find
  12. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the factual consistency of abstractive text summarization. arXiv, arXiv:1910.12840. find
  13. Güneş Erkan and Dragomir R. Radev. 2004. Lexrank: Graph-based centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22(1):457–479. find
  14. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Proceedings of Workshop on Text Summarization Branches Out, pages 74–81. our card
  15. Matan Eyal, Tal Baumel, and Michael Elhadad. 2019. Question answering as an automatic evaluation metric for news article summarization. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1, pages 3… our card
  16. Annie Louis and Ani Nenkova. 2009. Automatically evaluating content selection in summarization without human models. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 306–314. Association for Computational Linguistics. find
  17. Annie Louis and Ani Nenkova. 2013. Automatically assessing machine summary content without a gold standard. Computational Linguistics, 39(2):267– 300. our card
  18. Lisa Fan, Dong Yu, and Lu Wang. 2018. Robust neural abstractive summarization systems and Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M. Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized embeddings and earth mover distance. arXiv, arXiv:1909.02622. our card
  19. Yuning Mao, Liyuan Liu, Qi Zhu, Xiang Ren, and Jiawei Han. 2019. Facet-aware evaluation for extractive text summarization. arXiv, arXiv:1908.10383. — [arXiv:1908.10383](https://arxiv.org/abs/1908.10383) find
  20. Jun-Ping Ng and Viktoria Abrecht. 2015. Better summarization evaluation with word embeddings for rouge. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1925–1930. Association for Computational Linguistics. our card
  21. Daniel M. Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B. Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2020. Fine-tuning language models from human preferences. arXiv, arXiv:1909.08593v2. find
  22. Thomas Scialom, Sylvain Lamprier, Benjamin Piwowarski, and Jacopo Staiano. 2019. Answers unite! unsupervised metrics for reinforced summarization models. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural La… our card
  23. Liqun Shao, Hao Zhang, Ming Jia, and Jie Wang. 2017. Efficient and effective single-document summarizations and a word-embedding measurement of quality. arXiv, arXiv:1710.00284. — [arXiv:1710.00284](https://arxiv.org/abs/1710.00284) find
  24. Wilson L Taylor. 1953. Cloze procedure: A new tool for measuring readability. Journalism Bulletin, 30(4):415–433. find
  25. Oleg Vasilyev, Tom Grek, and John Bohannon. 2019. Headline generation: Learning from decomposable document titles. arXiv, arXiv:1904.08455v3. Version 3. — [arXiv:1904.08455](https://arxiv.org/abs/1904.08455) find
  26. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier-ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface’s transformers: State-of-the-art natural language processing. arXiv, arXiv:1910.03771. our card
  27. Stratos Xenouleas, Prodromos Malakasiotis, Marianna Apidianaki, and Ion Androutsopoulos. 2019. Sumqe: a bert-based summary quality estimation model. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Languag… find
  28. Peng Xu and Pascale Fung. 2019. A novel repetition normalized adversarial reward for headline generation. In ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7325–7329, Brighton, United Kingdom. IEEE. find
  29. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. arXiv, arXiv:1904.09675v3. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/vasilyev-2020-fill-blanc-human-free-quality/