Elamur Research⌕
☰
home / papers / 2018 / MSMO: Multimodal Summarization with Multimodal Output

MSMO: Multimodal Summarization with Multimodal Output

Paper · 2018 · Junnan Zhu, Haoran Li, Tianshang Liu et al.

Summary

The paper argues that summaries of multimedia news should be multimodal themselves. It introduces MSMO, a task where a system receives an article plus an unaligned set of images and must return a short text summary together with the single most relevant image. The authors built a large corpus from the Daily Mail website and annotated some pictorial summaries. Their model extends the pointer-generator network with a VGG19 image encoder, a visual attention layer, and a multimodal attention layer that weights textual against visual context during decoding. A visual coverage vector reduces repeated attention to the same images, and its final state determines which image is chosen. To evaluate outputs, they propose MMAE, an automatic metric that learns to combine text salience, image salience, and image-text relevance. MMAE agrees with human ratings far better than any single component metric and transfers to an unseen system. This gives future multimodal summarization work a dataset, a baseline, and a joint evaluation method.

Key points

  • In a user study, adding an image to the text summary raised user satisfaction with informativeness by 12.4% compared with text alone.
  • The architecture has four parts: a BiLSTM text encoder, a VGG19 image encoder, a multimodal attention layer, and an LSTM decoder. Three visual attention variants were tested: ATG (global fc7 features), ATL (local pool5 features), and HAN (hierarchical attention over local features).
  • The final MMAE is a linear regression over ROUGE-L, MAX sim and IP. On the test set it reaches Pearson r ≈ 0.665, while the best single metric (IP) reaches about 0.598 and ROUGE-L about 0.349.
  • On a new baseline system (LexRank sentence extraction plus a random image), MMAE kept a correlation of about 0.635 with human judgments, versus about 0.322 for ROUGE-L.
  • On 9,851 test articles, ATG achieved the best MMAE score (3.35). All attention models beat the extractive baseline GR, which scored 3.20.

Our summary of the paper, in our own words — not its text. The work belongs to its authors; read it at the links above.

 Institutions

🏛️Chinese Academy of SciencesBeijing · China68 works · 22,753 citations🔬Institute of AutomationBeijing · China20 works · 13,875 citations🎓University of Chinese Academy of SciencesBeijing · China32 works · 7,615 citations

Also on the byline: Center for Excellence in Brain Science and Intelligence Technology.

📚 Shelves 2The field of science this work belongs to (OpenAlex fields).Computer Science · 9,085 worksComputer Vision and Pattern Recognition · 1,282 works

🧭 Topics 2Research subjects and languages the work is about.Vision & language · 269 worksSummarization · 159 works

Cite

Cite as Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang and Chengqing Zong. 2018. MSMO: Multimodal Summarization with Multimodal Output. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.

The paper itself, not this page. BibTeX from the library's record of the work. DOI 10.18653/v1/d18-1448

@inproceedings{zhu2018msmo,
  title = {{MSMO: Multimodal Summarization with Multimodal Output}},
  author = {Junnan Zhu, Haoran Li, Tianshang Liu, Yu Zhou, Jiajun Zhang and Chengqing Zong},
  booktitle = {Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing},
  year = {2018},
  doi = {10.18653/v1/d18-1448},
  url = {https://doi.org/10.18653/v1/d18-1448}
}

References

39 references, 11 of them with a card of ours. ↗ leads to the work, «find» searches for one we have no id for yet.

  1. Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proceedings of the International Conference on Learning Representations (ICLR).
  2. Anil Bhattacharyya. 1943. On a measure of divergence between two statistical populations defined by their probability distributions. Bull. Calcutta Math. Soc., 35:99–109. find
  3. Jingwen Bian, Yang Yang, and Tat-Seng Chua. 2013. Multimedia summarization for trending topics in microblogs. In Proceedings of the 22nd ACM international conference on Conference on information & knowledge management, pages 1807–1812. find
  4. Jingwen Bian, Yang Yang, Hanwang Zhang, and Tat-Seng Chua. 2015. Multimedia summarization for social events in microblog stream. IEEE Transactions on multimedia, 17(2):216–228. find
  5. Kai Briechle and Uwe D Hanebeck. 2001. Template matching using fast normalized cross correlation. In Optical Pattern Recognition XII, volume 4387, pages 95–103. find
  6. Asli Celikyilmaz, Antoine Bosselut, Xiaodong He, and Yejin Choi. 2018. Deep communicating agents for abstractive summarization. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1662–16…
  7. Ronan Collobert, Jason Weston, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of Machine Learning Research (JMLR), 12(1):2493–2537.
  8. Günes Erkan and Dragomir R Radev. 2004. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457–479. find
  9. Georgios Evangelopoulos, Athanasia Zlatintsi, Alexandros Potamianos, Petros Maragos, Konstantinos Ra-pantzikos, Georgios Skoumas, and Yannis Avrithis. 2013. Multimodal saliency and fusion for movie summarization based on aural, visual, and textual attention. IEEE Transactions on Multimedia, 15(7):15… find
  10. Fartash Faghri, David J Fleet, Jamie Ryan Kiros, and Sanja Fidler. 2018. Vse++: Improving visual-semantic embeddings with hard negatives. In Proceedings of the British Machine Vision Conference (BMVC). find
  11. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of Neural Information Processing Systems (NIPS), pages 1693–1701.
  12. Andrej Karpathy and Li Fei-Fei. 2015. Deep visual-semantic alignments for generating image descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), pages 3128–3137.
  13. George Karypis. 2001. Evaluation of item-based top-n recommendation algorithms. In Proceedings of the tenth international conference on Information and knowledge management, pages 247–254. find
  14. Ryan Kiros, Ruslan Salakhutdinov, and Richard S Zemel. 2014. Unifying visual-semantic embeddings with multimodal neural language models. arXiv preprint arXiv:1411.2539.
  15. Haoran Li, Jiajun Zhang, Yu Zhou, and Chengqing Zong. 2016a. Guiderank: A guided ranking graph model for multilingual multi-document summarization. In The Fifth Conference on Natural Language Processing and Chinese Computing & The Twenty Fourth International Conference on Computer Processing of Orie… find
  16. Haoran Li, Junnan Zhu, Tianshang Liu, Jiajun Zhang, and Chengqing Zong. 2018a. Multi-modal sentence summarization with modality attention and image filtering. In Proceedings of the Twenty-Fifth International Joint Conference on Artificial Intelligence (IJCAI), pages 4152–4158. find
  17. Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. 2017. Multi-modal summarization for asynchronous collection of text, image, audio and video. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1092–1102. find
  18. Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. 2018b. Read, watch, listen and summarize: Multi-modal summarization for asynchronous text, image, audio and video. IEEE Transactions on Knowledge and Data Engineering (TKDE). find
  19. Zechao Li, Jinhui Tang, Xueming Wang, Jing Liu, and Hanqing Lu. 2016b. Multimedia news summarization in search. ACM Transactions on Intelligent Systems and Technology (TIST), 7(3):33. find
  20. Chin-Yew Lin. 2004a. Looking for a few good metrics: Automatic summarization evaluation-how many samples are enough? In Proceedings of NII Testbeds and Community for information access Research (NTCIR). find
  21. Paul Over and James Yen. 2004. An introduction to duc 2004 intrinsic evaluation of generic new text summarization systems. In Proceedings of the Document Understanding Conference (DUC). find
  22. Karolina Owczarzak, John M Conroy, Hoa Trang Dang, and Ani Nenkova. 2012. An assessment of the accuracy of automatic evaluation in summarization. In Proceedings of Workshop on Evaluation Metrics and System Comparison for Automatic Summarization, pages 1–9. find
  23. Romain Paulus, Caiming Xiong, and Richard Socher. 2018. A deep reinforced model for abstractive summarization. In Proceedings of the International Conference on Learning Representations (ICLR).
  24. Shengsheng Qian, Tianzhu Zhang, and Changsheng Xu. 2016. Multi-modal multi-view topic-opinion mining for social event analysis. In Proceedings of the 2016 ACM on Multimedia Conference, pages 2– 11. find
  25. Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015. A neural attention model for abstractive sentence summarization. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 379–389. find
  26. Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL), pages 1073–1083. find
  27. Rajiv Ratn Shah, Yi Yu, Akshay Verma, Suhua Tang, Anwar Dilawar Shaikh, and Roger Zimmermann. 2016. Leveraging multimodal information for event summarization and concept-level sentiment analysis. Knowledge-Based Systems, 108:102–109. find
  28. Chin-Yew Lin. 2004b. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out.
  29. Karen Simonyan and Andrew Zisserman. 2015. Very deep convolutional networks for large-scale image recognition. In Proceedings of the International Conference on Learning Representations (ICLR).
  30. Thang Luong, Hieu Pham, and Christopher D. Manning. 2015. Effective approaches to attention-based neural machine translation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1412–1421.
  31. Xiaojun Wan and Jianwu Yang. 2006. Improved affinity graph based multi-document summarization. In Proceedings of the Human Language Technology Conference of the NAACL, pages 181–184. find
  32. Ioannis Mademlis, Anastasios Tefas, Nikos Nikolaidis, and Ioannis Pitas. 2016. Multimodal stereoscopic movie summarization conforming to narrative characteristics. IEEE Transactions on Image Processing, 25(12):5828–5840. find
  33. Ramesh Nallapati, Bowen Zhou, Caglar Gulcehre, Bing Xiang, et al. 2016. Abstractive text summarization using sequence-to-sequence rnns and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL). find
  34. Liwei Wang, Yin Li, Jing Huang, and Svetlana Lazebnik. 2018. Learning two-branch neural networks for image-text matching tasks. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). find
  35. William Yang Wang, Yashar Mehdad, Dragomir R Radev, and Amanda Stent. 2016. A low-rank approximation approach to learning joint embeddings of news stories and images for timeline summarization. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational L… find
  36. Kelvin Xu, Jimmy Ba, Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. 2015. Show, attend and tell: Neural image caption generation with visual attention. In International Conference on Machine Learning (ICML), pages 2048–2057.
  37. Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. 2014. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics (TACL), 2:67–78. find
  38. Junnan Zhu, Long Zhou, Haoran Li, Jiajun Zhang, Yu Zhou, and Chengqing Zong. 2017. Augmenting neural sentence summarization through extractive summarization. In Proceedings of the 6th Conference on Natural Language Processing and Chinese Computing (NLPCC), pages 16–28. find
  39. Keneilwe Zuva and Tranos Zuva. 2012. Evaluation of information retrieval systems. International Journal of Computer Science & Information Technology, 4(3):35. find

Updated

↑ back to top

This page in Markdown → https://md.research.elamur.ai/free/en/papers/zhu-2018-msmo-multimodal-summarization-multimodal-output/