Elamur Research⌕
☰
home / papers / 2020 / Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports

Summary

Abstractive summarizers trained to match human references can score well on overlap metrics like ROUGE and still state wrong facts. For example, a summary may reach a high ROUGE-L while reversing a finding such as the presence of pneumothorax. Zhang et al. address this for radiology, where the findings section of a report is condensed into an impression and errors can cause medical harm.

They build an automatic fact-checking setup. An information extraction module pulls clinical observations from both the generated and the reference summary, and comparing the two yields a factual accuracy score. This score serves as a reward in reinforcement learning, alongside a ROUGE-based reward and a language model objective, to train a background-augmented pointer-generator.

On two hospital datasets (Stanford and RIH), automatic metrics and blinded radiologist judgments both show clear gains in factual correctness and overall quality over the baseline, approaching human-written summaries. Optimizing ROUGE alone via RL already improves correctness compared with maximum likelihood training. The authors describe this as the first work to directly optimize a summarizer for factual correctness.

Key points

  • Backbone model: the background-augmented pointer-generator (Bi-LSTM encoder, LSTM decoder, attention, copy mechanism, separately encoded background section). The best variant, RL R+C, combines ROUGE and factual correctness rewards.
  • Motivation: earlier work found that about 30% of neural radiology summaries contained factual errors.
  • Human evaluation: two board-certified radiologists ranked 50 Stanford test examples on fluency, factual correctness/completeness, and overall quality. RL R+C beat the baseline on correctness and overall quality but was slightly less preferred on fluency, where 60% of comparisons were ties.
  • Style analysis: a Mixture of Softmaxes LM was trained on about 222,000 human summaries. RL R+C's perplexity was much closer to human references, while the baseline overused generic text: the sentence "no significant interval change" appeared in 34% of baseline outputs on the Stanford dev set, versus 24% for RL R+C and 17% for humans.
  • Limitations: the method depends on a rule-based IE module that suits only a limited fact space, so open-domain generalization remains unsolved. It also targets mainly binary errors, such as whether a disease is present.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

Fields: Natural language processing

Cited by

80 papers of our library cite it (of the 140884 whose bibliographies we read); 8 of them have a page here, 72 not yet.

Cite

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/2020.acl-main.458

@inproceedings{zhang2020optimizing,
  title = {{Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reports}},
  author = {Zhang, Yuhao and Merck, Derek and Tsai, Emily and Manning, Christopher D. and Langlotz, Curtis},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
  doi = {10.18653/v1/2020.acl-main.458},
  url = {https://doi.org/10.18653/v1/2020.acl-main.458}
}

References

43 references, 13 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. find our card
  2. The 2015 International Conference on Learning Representations. find
  3. Ziqiang Cao, Furu Wei, Wenjie Li, and Sujian Li. 2017. Faithful to the original: Fact aware neural abstractive summarization. The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-2017). find
  4. Arun Chaganty, Stephen Mussmann, and Percy Liang. 2018. The price of debiasing automatic metrics in natural language evalaution. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018). find
  5. Yen-Chun Chen and Mohit Bansal. 2018. Fast abstractive summarization with reinforce-selected sentence rewriting. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL 2018). find our card
  6. Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL 2016). find our card
  7. Wojciech Kryściński, Bryan McCann, Caiming Xiong, and Richard Socher. 2019b. Evaluating the factual consistency of abstractive text summarization. arXiv preprint arXiv:1910.12840. find
  8. Bhuwan Dhingra, Manaal Faruqui, Ankur Parikh, Ming-Wei Chang, Dipanjan Das, and William Cohen. 2019. Handling divergent reference texts when evaluating table-to-text generation. Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL 2019). find our card
  9. Christy Y. Li, Xiaodan Liang, Zhiting Hu, and Eric P. Xing. 2019. Hybrid retrieval-generation reinforced agent for medical image report generation. Advances in neural information processing systems. find
  10. Yue Dong, Yikang Shen, Eric Crawford, Herke van Hoof, and Jackie Chi Kit Cheung. 2018. Bandit-Sum: Extractive summarization as a contextual bandit. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP 2018). find
  11. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. Text Summarization Branches Out: ACL Workshop. find
  12. Guanxiong Liu, Tzu-Ming Harry Hsu, Matthew Mc-Dermott, Willie Boag, Wei-Hung Weng, Peter Szolovits, and Marzyeh Ghassemi. 2019. Clinically accurate chest X-ray report generation. arXiv preprint arXiv:1904.02633. — [arXiv:1904.02633](https://arxiv.org/abs/1904.02633) find
  13. Günes Erkan and Dragomir R Radev. 2004. LexRank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. find
  14. Peter J. Liu, Mohammad Saleh, Etienne Pot, Ben Goodrich, Ryan Sepassi, Lukasz Kaiser, and Noam Shazeer. 2018. Generating wikipedia by summarizing long sequences. find our card
  15. The 2018 International Conference for Learning Representations. find
  16. Tobias Falke, Leonardo F. R. Ribeiro, Prasetya Ajie Utama, Ido Dagan, and Iryna Gurevych. 2019. Ranking generated summaries by correctness: An interesting but challenging application for natural language inference. In Proceedings of the 57th Annual Meeting of the Association for Computational Lingui… find our card
  17. Sean MacAvaney, Sajad Sotudeh, Arman Cohan, Nazli Goharian, Ish Talati, and Ross W. Filice. 2019. Ontology-aware clinical abstractive summarization. Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 19). find
  18. Ben Goodrich, Vinay Rao, Peter J. Liu, and Mohammad Saleh. 2019. Assessing the factual accuracy of generated text. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD 19). find
  19. Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al. 2019. CheXpert: A large chest radiograph dataset with uncertainty labels and expert comparison. The Thirty-Third AAAI Conference on Artifici… find
  20. Charles E Kahn Jr, Curtis P Langlotz, Elizabeth S Burnside, John A Carrino, David S Channin, David M Hovsepian, and Daniel L Rubin. 2009. Toward best practices in radiology reporting. Radiology, 252(3):852–856. find
  21. Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In The 2015 International Conference for Learning Representations. find
  22. Wojciech Kryściński, Nitish Shirish Keskar, Bryan Mc-Cann, Caiming Xiong, and Richard Socher. 2019a. Neural text summarization: A critical evaluation. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing (EMNLP 2019). find our card
  23. Christopher D Manning, Prabhakar Raghavan, and Hinrich Schütze. 2008. Introduction to information retrieval. Cambridge University Press. find
  24. Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David Mc-Closky. 2014. The Stanford CoreNLP natural language processing toolkit. In Association for Computational Linguistics (ACL) System Demonstrations. find our card
  25. Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2016a. SummaRuNNer: A recurrent neural network based sequence model for extractive summarization of documents. The Thirty-First AAAI Conference on Artificial Intelligence (AAAI-2017). find our card
  26. Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Caglar Gulcehre, and Bing Xiang. 2016b. Abstractive text summarization using sequence-to-sequence RNNs and beyond. Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning. find
  27. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations. find our card
  28. Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. find
  29. Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. find our card
  30. The 2016 International Conference on Learning Representations. find
  31. Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7008–7024. find our card
  32. Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get to the point: Summarization with pointer-generator networks. In The 2017 Annual Meeting of the Association of Computational Linguistics (ACL 2017). find
  33. Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(1):1929–1958. find
  34. Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. 2015. Pointer networks. In Advances in Neural Information Processing Systems, pages 2692–2700. find
  35. Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning, 8(3-4):229–256. find
  36. Ronald J Williams and David Zipser. 1989. A learning algorithm for continually running fully recurrent neural networks. Neural Computation, 1(2):270– 280. find
  37. Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017. Challenges in data-to-document generation. Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP 2017). find our card
  38. Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. Breaking the softmax bottleneck: A high-rank RNN language model. find
  39. The 2018 International Conference for Learning Representations. find
  40. Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In The 2020 International Conference for Learning Representations. find
  41. Yuhao Zhang, Daisy Yi Ding, Tianpei Qian, Christopher D. Manning, and Curtis P. Langlotz. 2018. Learning to summarize radiology findings. In EMNLP 2018 Workshop on Health Text Mining and Information Analysis. find
  42. Split Train Dev Test Time Coverage Stanford RIH 2009/01 – 2014/04 2014/05 – 2014/08 2014/09 – 2014/12 find
  43. Variable 2017/11 – 2018/06 2018/07 – 2018/09 2018/10 – 2018/12 *Table 6: Time coverage of different splits in the Stanford and RIH datasets.* find

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhang-2020-optimizing-factual-correctness-summary-study/