Elamur Research⌕
☰
home / papers / 2018 / On the Automatic Generation of Medical Imaging Reports

On the Automatic Generation of Medical Imaging Reports

Paper · 2018 · Baoyu Jing, Pengtao Xie and Eric Xing

Summary

Writing reports for medical images takes skilled physicians a lot of time and is error-prone for less experienced ones. The authors therefore study how to generate such reports automatically. A full report mixes different kinds of content, such as keyword tags and a multi-sentence findings paragraph. Abnormal regions are also hard to localize.

To handle this, the paper builds a single multi-task model:

  • A CNN encodes image regions.
  • A multi-label classifier predicts tags, and the embeddings of those tags serve as semantic features.
  • A co-attention module weighs visual and semantic information together to form a context vector.
  • A two-level (hierarchical) LSTM decodes the report. A sentence-level LSTM produces one topic vector per sentence and decides when to stop, and a word-level LSTM turns each topic into a sentence.

The model was tested on two public datasets of radiology and pathology images. The full co-attention model beat the baselines and its own ablated variants on every reported caption metric. Its output was also closest to the real balance of normal versus abnormal findings. This makes it an early step toward generating free-form clinical narratives rather than templated or semi-structured ones.

Key points

  • Reports have three parts: impression, findings and tags. The tags are keywords extracted with the Medical Text Indexer (MTI) and are predicted as a multi-label classification task.
  • Visual features come from the last convolutional layer of VGG-19. Its last two fully connected layers act as the MLC tag predictor, and embeddings of the top-M tags feed the co-attention.
  • The sentence LSTM uses deep output layers for both topic generation and a STOP/CONTINUE decision. The word LSTM then generates each sentence from its topic.
  • Share of generated sentences describing abnormalities (Table 2):
    • Ours-CoAttention: 0.529
    • Soft Attention: 0.490
    • Ours-no-Attention: 0.247
    • Ground truth: 0.615
  • Motivation: typing the findings for a single image takes about 5–10 minutes. The authors present this as the first system to generate natural, physician-written-style reports rather than semi-structured ones.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🎓Carnegie Mellon UniversityPittsburgh · United States365 works · 158,048 citations

📚 Shelves 4The field of science this work belongs to (OpenAlex fields).Computer Science · 9,086 worksComputer Vision and Pattern Recognition · 1,282 worksMedicine · 80 worksHealth Informatics · 53 works

Cite

Cite as Baoyu Jing, Pengtao Xie and Eric Xing. 2018. On the Automatic Generation of Medical Imaging Reports. Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/p18-1240

@inproceedings{jing2018automatic,
  title = {{On the Automatic Generation of Medical Imaging Reports}},
  author = {Baoyu Jing, Pengtao Xie and Eric Xing},
  booktitle = {Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)},
  year = {2018},
  doi = {10.18653/v1/p18-1240},
  url = {https://doi.org/10.18653/v1/p18-1240}
}

References

26 references, 9 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Jimmy Ba, Volodymyr Mnih, and Koray Kavukcuoglu. 2015. Multiple object recognition with visual attention. ICLR. find
  2. Louke Delrue, Robert Gosselin, Bart Ilsen, An Van Landeghem, Johan de Mey, and Philippe Duyck. 2011. Difficulties in the interpretation of chest radiography. In Comparative Interpretation of CT and Standard Radiography of the Chest, pages 27–49. Springer. find
  3. Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Rodriguez, Sameer Antani, George R Thoma, and Clement J McDonald. 2015. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Association, 23(2):304–31… find
  4. Michael Denkowski and Alon Lavie. 2014. Meteor universal: Language specific translation evaluation for any target language. In Proceedings of the ninth workshop on statistical machine translation, pages 376–380.
  5. Jeffrey Donahue, Lisa Anne Hendricks, Sergio Guadarrama, Marcus Rohrbach, Subhashini Venugopalan, Kate Saenko, and Trevor Darrell. 2015. Long-term recurrent convolutional networks for visual recognition and description. In Proceedings of the IEEE conference on computer vision and pattern recognition…
  6. Hao Fang, Saurabh Gupta, Forrest Iandola, Rupesh K Srivastava, Li Deng, Piotr Dollár, Jianfeng Gao, Xiaodong He, Margaret Mitchell, John C Platt, et al. 2015. From captions to visual concepts and back. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1473–1482… find
  7. Yunchao Gong, Yangqing Jia, Thomas Leung, Alexander Toshev, and Sergey Ioffe. 2013. Deep convolutional ranking for multilabel image annotation. ICLR. find
  8. Matthieu Guillaumin, Thomas Mensink, Jakob Verbeek, and Cordelia Schmid. 2009. Tagprop: Discriminative metric learning in nearest neighbor models for image auto-annotation. In Computer Vision, 2009 IEEE 12th International Conference on, pages 309–316. IEEE. find
  9. Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735–1780. find
  10. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3128–3137.
  11. Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980. find
  12. Pavel Kisilev, Eugene Walach, Ella Barkan, Boaz Ophir, Sharon Alpert, and Sharbell Y Hashoul. 2015. From medical image to automatic medical report generation. IBM Journal of Research and Development, 59(2/3):2–1. find
  13. Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for generating descriptive image paragraphs. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). find
  14. Xiaodan Liang, Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P. Xing. 2017. Recurrent topic-transition gan for visual paragraph generation. In The IEEE International Conference on Computer Vision (ICCV). find
  15. Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image captioning with semantic attention. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4651–4659. find
  16. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop, volume 8. Barcelona, Spain.
  17. Zizhao Zhang, Yuanpu Xie, Fuyong Xing, Mason Mc-Gough, and Lin Yang. 2017. Mdnet: A semantically and visually interpretable medical image diagnosis network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6428–6436. find
  18. Pengfei Liu, Xipeng Qiu, Xinchi Chen, Shiyu Wu, and Xuanjing Huang. 2015. Multi-timescale long short-term memory neural network for modelling sentences and documents. In Proceedings of the 2015 conference on empirical methods in natural language processing, pages 2326–2335. find
  19. Eric Martin and Chris Cundy. 2018. Parallelizing linear recurrent neural nets over sequence length. ICLR. find
  20. Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311–318. Association for Computational Linguistics.
  21. Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. 2014. How to construct deep recurrent neural networks. ICLR. find
  22. Hoo-Chang Shin, Kirk Roberts, Le Lu, Dina Demner-Fushman, Jianhua Yao, and Ronald M Summers. 2016. Learning to read chest x-rays: recurrent neural cascade model for automated image annotation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2497– 2506. find
  23. K. Simonyan and A. Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. CoRR, abs/1409.1556.
  24. Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575.
  25. Oriol Vinyals, Alexander Toshev, Samy Bengio, and Dumitru Erhan. 2015. Show and tell: A neural image caption generator. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3156–3164.
  26. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning, pages 2048–2057.

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/jing-2018-automatic-generation-medical-imaging-reports/